GitHub webhooks degraded causing job queueing

Update timeline

1. resolved May 19, 2026, 02:50 PM UTC

   Type: Incident Duration: 8 hours and 49 minutes Affected Components: eu-central x86, us-central MacOS, us-west x86, eu-west x86, us-west ARM, eu-central ARM May 19, 23:39:51 GMT+0 - Resolved - This incident has been resolved. May 19, 14:50:48 GMT+0 - Investigating - We're seeing evidence of webhooks delivery being degraded from GitHub. We're investigating. May 19, 15:53:19 GMT+0 - Monitoring - We're seeing upstream recovery for GitHub webhook deliveries. Jobs may queue as we process GitHub's backlog of webhook events. May 19, 17:10:34 GMT+0 - Monitoring - We're still seeing a large backlog of queued jobs due to the incident that the system is working through, we're exploring mitigations. May 19, 20:02:52 GMT+0 - Monitoring - We're still working through a substantial backlog of queued tasks that has accumulated over this period of delayed webhook arrivals. May 19, 21:22:37 GMT+0 - Monitoring - We are close to the end of the backlog of the queued tasks and are seeing full recovery in certain runner pools.