Resolved after 38 minutes, on May 12, 2026.
## Impact Customers experienced intermittent failures on authentication operations between 06:47 and 07:25 UTC. Affected operations included: - OAuth token refresh (`/oauth/token`) - OAuth authorization (`/oauth/authorize`) At peak degradation, approximately 16% of authentication requests were failing. Management API calls and AgentKit tool execution were not affected. --- ### Timeline **06:47 UTC — Investigating** We began observing elevated error rates on authentication endpoints. Token refresh requests were returning `INTERNAL_SERVER_ERROR`. **07:02 UTC — Identified** Root cause identified: elevated internal write load on our authentication database caused query timeouts on token refresh and auth session operations. The source of the write load was contained and the database began recovering. **07:02–07:25 UTC — Monitoring** Authentication requests returned to normal. Some customers experienced `invalid_grant` errors during this window. These occurred because refresh tokens that were in-flight during the degradation period were invalidated during recovery. Customers whose SDKs perform automatic token refresh recovered automatically as invalidated tokens were cycled out. **07:25 UTC — Resolved** All authentication operations returned to normal error rates. Full recovery confirmed. --- ### Root Cause A surge of internal database writes caused query latency to spike on our shared authentication database, resulting in timeouts on token refresh and related auth operations. The write surge originated from an internal process that was not rate-limited at the infrastructure layer. --- ### Remediation We have identified two fixes to prevent recurrence: 1. **Immediate fix:** Patching the internal process responsible for the database write surge to ensure rate-limited and rejected requests do not generate disproportionate downstream write load. 2. **Infrastructure hardening:** Implementing edge-layer rate limiting at our load balancer to stop traffic surges before they reach the application layer. This will prevent a similar write surge from impacting the authentication database in future. We will publish a follow-up once both fixes are deployed. --- We apologise for the disruption. If you have questions, please contact support.
Update timeline
- resolved May 13, 2026, 05:57 AM UTC
## Impact Customers experienced intermittent failures on authentication operations between 06:47 and 07:25 UTC. Affected operations included: - OAuth token refresh (`/oauth/token`) - OAuth authorization (`/oauth/authorize`) At peak degradation, approximately 16% of authentication requests were failing. Management API calls and AgentKit tool execution were not affected. --- ### Timeline **06:47 UTC — Investigating** We began observing elevated error rates on authentication endpoints. Token refresh requests were returning `INTERNAL_SERVER_ERROR`. **07:02 UTC — Identified** Root cause identified: elevated internal write load on our authentication database caused query timeouts on token refresh and auth session operations. The source of the write load was contained and the database began recovering. **07:02–07:25 UTC — Monitoring** Authentication requests returned to normal. Some customers experienced `invalid_grant` errors during this window. These occurred because refresh tokens that were in-flight during the degradation period were invalidated during recovery. Customers whose SDKs perform automatic token refresh recovered automatically as invalidated tokens were cycled out. **07:25 UTC — Resolved** All authentication operations returned to normal error rates. Full recovery confirmed. --- ### Root Cause A surge of internal database writes caused query latency to spike on our shared authentication database, resulting in timeouts on token refresh and related auth operations. The write surge originated from an internal process that was not rate-limited at the infrastructure layer. --- ### Remediation We have identified two fixes to prevent recurrence: 1. **Immediate fix:** Patching the internal process responsible for the database write surge to ensure rate-limited and rejected requests do not generate disproportionate downstream write load. 2. **Infrastructure hardening:** Implementing edge-layer rate limiting at our load balancer to stop traffic surges before they reach the application layer. This will prevent a similar write surge from impacting the authentication database in future. We will publish a follow-up once both fixes are deployed. --- We apologise for the disruption. If you have questions, please contact support.
Your product runs on more than Scalekit
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services