Scrutinizer incident
External Network Route Leak causing Inspection Failures
Scrutinizer experienced a major incident on June 24, 2019 affecting Workers, lasting 33m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 24, 2019, 10:41 AM UTC
We are currently investigating this issue.
- identified Jun 24, 2019, 11:00 AM UTC
We have identified an issue in network connectivity which leads to communication failures between different components. We are continuing to investigate the root cause.
- monitoring Jun 24, 2019, 11:07 AM UTC
We have temporarily mitigated and are continuing to investigate the root cause. Retried inspections should run through normally now.
- monitoring Jun 24, 2019, 11:22 AM UTC
We have tracked the root cause down to an upstream issue with Cloudflare which we use https://www.cloudflarestatus.com/incidents/46z55mdhg0t5 For the moment, this issue is mitigated on our end. However, we are seeing errors for other domains (such as node.js) and believe this might be a wider Internet connectivity issue. As a result, inspections that install a node.js version are currently failing. We are further investigating how to minimize the impact this has on your inspections and we will keep you updated here.
- monitoring Jun 24, 2019, 12:40 PM UTC
We are continuing to work on ways to mitigate this issue as best as possible on our end. We will keep you updated.
- monitoring Jun 24, 2019, 12:46 PM UTC
The issue has been resolved upstream and we are seeing reduced error rates for inspections. We are continuing to monitor the situation and will update as necessary. If you have a failed inspection related to node.js, a retry should now run through normally.
- resolved Jun 24, 2019, 01:13 PM UTC
This incident has been resolved. We are continuing to monitor and re-open if further updates are needed.