SchemeServe experienced a minor incident on April 6, 2021 affecting 🎩 SchemeServe, lasting 14h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 06, 2021, 05:19 PM UTC
SchemeServe is hosted through Microsoft Azure which has been having ongoing outage issues across Europe since approximately 12pm today. We have critical tickets raised with their support team and hope to have full functionality restored shortly. Please accept our apologies for any inconvenience during this time.
- investigating Apr 07, 2021, 08:09 AM UTC
This issue is now resolved.
- resolved Apr 07, 2021, 08:09 AM UTC
This incident has been resolved.
- postmortem Apr 07, 2021, 08:30 AM UTC
Feedback from Microsoft Azure: Summary of Impact: Between 10:48 UTC and 16:56 UTC on 06 Apr 2021, customers using Azure Front Door and Azure CDN Standard from Microsoft may have encountered intermittent connection failures and service degradation. Impacted customers were primarily located in European geographies, however a subset of customers in other geographies may have encountered intermittent impact at a lower rate. Root Cause: Azure Front Door and Azure CDN Standard from Microsoft experienced significant traffic spike at our London edge site. This was detected by our monitoring and resulted in alerts sent to service team. The traffic increase initiated traffic management processes which started distributing traffic to all nearby sites in Europe, as expected. Mitigation: The service team came on board to mitigate the incident by further adjusting the traffic distribution to nearby sites to help contain the issue. By that time, the traffic volume had dropped back to normal levels. As part of our continuous efforts to better the Azure platform we have undertaken the following repair items. These will be released to production at a future release date. Improve automated load management to improve responsiveness to increased traffic events.