Proxyclick experienced a critical incident on October 13, 2021 affecting Dashboard and iPad app and 1 more component, lasting 1h 28m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 13, 2021, 07:28 AM UTC
We are currently investigating this issue
- identified Oct 13, 2021, 07:44 AM UTC
The issue has been identified, a global outage in our server provider is the root cause of the issue. We are working with them to address this issue.
- monitoring Oct 13, 2021, 08:22 AM UTC
The issue appears to be resolved, we are monitoring the situation.
- resolved Oct 13, 2021, 08:57 AM UTC
This incident has been resolved. A post mortem will be available as soon as posted by our servers provider.
- postmortem Oct 18, 2021, 01:22 PM UTC
This issue was identified as a general outage within our global server provider OVH, the below is the summary of the issue as communicated by them: _During the outage, the full Internet routing table was being announced in the OVHcloud IGP. The massive influx of routing information on the IGP led some routers to miss behave : OSPF table got full, overloading RAM and CPU. The impact was the IPv4 routing only and all IPv6 traffic were accessible._ _Our newer routers started to use D2 VIN as the default gateway for all the internet traffic, hence causing the traffic to flow to the US. This led to an unability to process the traffic properly for IPv4 on all our sites._ _We were able to take back control over the situation very quickly with the access to the physical faulty equipment and isolate it from the network. \(Once the D2 was put offline the network reconverged, emptying the OSPF tables on the devices and routing traffic to the nominal gateways\)._ _Our immediate actions is to re-assess our validation procedure on such type of devices \(which applies and commits the command line natively\) & reinforces accordingly the change process._ _As this incident impacted our customers using IPv4 protocol, our teams across the globe have been following the situation as closely as can be, to help them recover and keep them up to date._ From Proxyclick, during the outage we kept in consistent communication with our representatives at OVH to ensure our SLA metrics were met. We periodically evaluate the performance of our vendors to ensure delivery and quality of the service, the same applies in this instance, we will continue to monitor their performance to ensure it meets the standards and expectations of our customers.