Clerk.io incident

Database Zone 2 Down

Major Resolved View vendor source →

Clerk.io experienced a major incident on December 9, 2024 affecting API (api.clerk.io) and Services (Search) and 1 more component, lasting 35m. The incident has been resolved; the full update timeline is below.

Started
Dec 09, 2024, 02:10 PM UTC
Resolved
Dec 09, 2024, 02:45 PM UTC
Duration
35m
Detected by Pingoru
Dec 09, 2024, 02:10 PM UTC

Affected components

API (api.clerk.io)Services (Search)Services (Recommendations)Services (Chat)Services (Email)Services (Audience)Data (Data Sync)Data (Data Analyzation)

Update timeline

  1. investigating Dec 09, 2024, 02:10 PM UTC

    We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2.

  2. resolved Dec 09, 2024, 02:45 PM UTC

    Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production.