Blacksmith Outage History

Type: Incident Duration: 3 hours and 34 minutes Affected Components: us-central MacOS, eu-central x86, us-west x86, eu-west x86, us-west ARM, eu-central ARM Apr 20, 16:35:28 GMT+0 - Investigating - We are currently investigating this incident. Apr 20, 16:44:34 GMT+0 - Identified - We have now isolated this incident to a set of control plane ECS tasks that we are working on mitigating. Apr 20, 17:02:49 GMT+0 - Monitoring - We're seeing signs of recovery and are monitoring the result. Apr 20, 17:28:00 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 16 minutes Affected Components: us-central MacOS, eu-central x86, us-west x86, eu-west x86, us-west ARM, eu-central ARM Apr 20, 15:12:09 GMT+0 - Investigating - We are currently investigating this incident. Apr 20, 15:18:33 GMT+0 - Monitoring - We're seeing recovery and are monitoring the result. Apr 20, 15:27:57 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 18 minutes Affected Components: eu-central x86, us-west x86, eu-west x86, us-west ARM, eu-central ARM Apr 16, 17:00:07 GMT+0 - Identified - We're currently working through a backlog of queued jobs that accumulated due to a blip in our backend. The queue is actively draining and we expect it to fully clear in \~5 minutes. We will provide updates here. Apr 16, 17:12:39 GMT+0 - Monitoring - We're continuing to process queued jobs and will provide another update shortly. Apr 16, 17:18:28 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 30 minutes Affected Components: us-west x86 Apr 13, 18:41:57 GMT+0 - Investigating - Queue times for US x86 runners are currently elevated. We are currently investigating this incident. Apr 13, 19:03:37 GMT+0 - Monitoring - Queue times are at expected levels. We are continuing to monitor. Apr 13, 19:12:15 GMT+0 - Resolved - This incident has been resolved.

Type: Maintenance Duration: 1 hour Affected Components: , Incremental Docker Builders → Apr 12, 04:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Apr 12, 04:00:01 GMT+0 - Identified - Maintenance is now in progress Apr 12, 05:00:00 GMT+0 - Completed - Maintenance has completed successfully

Type: Incident Duration: 30 minutes Affected Components: Github → Actions, Github → Webhooks Mar 24, 20:28:39 GMT+0 - Investigating - We're seeing issues with Webhook ingestion and Actions due to this GitHub incident: We are currently investigating this incident. Mar 24, 20:58:30 GMT+0 - Resolved - This incident has been resolved.

Type: Maintenance Duration: 15 minutes Affected Components: Actions Cache Mar 20, 01:00:00 GMT+0 - Identified - We are upgrading the Actions Cache capacity, cache performance may be degraded during this maintenance window. Mar 20, 01:00:01 GMT+0 - Identified - Maintenance is now in progress Mar 20, 01:15:00 GMT+0 - Completed - Maintenance has completed successfully