EU - Incident with API Requests, Actions and Issues

Update timeline

1. investigating Nov 26, 2025, 04:27 PM UTC

   We are investigating reports of degraded performance for API Requests, Actions and Issues

2. investigating Nov 26, 2025, 04:33 PM UTC

   Git Operations is experiencing degraded performance. We are continuing to investigate.

3. resolved Nov 26, 2025, 04:49 PM UTC

   On November 26, 2025 between 16:07:00 UTC and 16:49:00 UTC multiple services were degraded in the Sweden-based GHEC-DR, with ~10% of requests receiving a 5xx error response. This was due to an increased usage from a single customer who was performing a migration on the Sweden GHEC-DR, which increased the amount of internal data processed by Billing. Attempts to mitigate the problem by increasing the rate limit caused a capacity exhaustion on the internal load balancer which caused a wide spread impact across other APIs and services. The incident was finally resolved by increasing the capacity of the internal load balancer. Due to this incident, ~25% of customers in Sweden GHEC-DR saw lower usage in their Billing pages and invoices from November and December, and will be notified individually. We are focused on improving our capacity to manage future usage spikes, including isolating the impact from a single customer's disproportionate resource consumption. Our goal is to identify and implement earlier mitigation strategies to prevent recurrence of such issues.