Clerk.io Outage History

Started Sep 02, 2026, 05:05 AM UTC· resolved Sep 02, 2026, 09:46 AM UTC

1. investigating Sep 02, 2026, 05:05 AM UTC

   Today between 07:05 CEST and 07:13 CEST the main API (search and recommendations) was partially unavailable because of a database crash. We are currently investigating the crash and will be working to mitigate this in the future. We are very sorry for any inconvenience this has caused for you and your customers.

2. resolved Sep 02, 2026, 09:46 AM UTC

   Incident report: API availability Date: 2026-09-02 T 07:05 CEST Duration: Approximately 10 minutes Impact: Some API requests and background operations were delayed or failed for a subset of stores. This also had a knock-on effect on the API availability for other stores as workers got tied up, until automatic scaling added new servers. What happened One of our database servers stopped processing queries after an internal MariaDB/InnoDB synchronization operation became stuck. The database remained online at the network level but could not complete normal work. After its internal safety timeout was reached, MariaDB terminated the stalled process and recovered automatically during restart. Service returned to normal once database recovery completed. Cause The incident was caused by an internal database metadata lock that failed to make progress. The likely trigger was an unusual interaction between a high-volume background table rebuild and database checkpoint processing. This was not caused by increased customer traffic or an individual customer's actions. The same background processing normally completes without affecting availability. The precise database thread responsible could not be identified because detailed crash diagnostics were not enabled at the time. Data integrity MariaDB completed its normal crash-recovery process successfully. Interrupted transactions were rolled back automatically, and we found no evidence of database corruption. No customer action is required. Improvements We have increased the database log capacity to provide more processing headroom during large background operations. We are also preparing the following changes: Remove unnecessary database metadata operations from bulk imports. Upgrade MariaDB to the latest maintenance release. Collect detailed diagnostics if a similar database failure occurs again. Reduce the time MariaDB waits before recovering from a confirmed internal stall. Improve monitoring and recovery automation for database operations that appear online but cannot process real queries. Extend failure handling for background processing as well as customer-facing API requests. These changes are intended both to reduce the likelihood of recurrence and to restore service more quickly if another internal database stall occurs.

Read the full incident report →

Started Aug 27, 2026, 02:30 PM UTC· resolved Aug 27, 2026, 02:30 PM UTC

1. resolved Aug 27, 2026, 02:30 PM UTC

   The issue has been resolved, and the API is fully operational again. From 16:11 to 16:22 CEST, v2/search, v2/recommendations, and the dashboard were unavailable due to a database disruption. We are currently reviewing the incident and implementing measures to prevent such incidents from happening in the future. We apologize for the inconvenience.

Read the full incident report →

Started Jul 19, 2026, 04:48 AM UTC· resolved Jul 19, 2026, 04:48 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 05, 2026, 10:15 AM UTC· resolved Jun 05, 2026, 10:15 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started May 28, 2026, 02:26 PM UTC· resolved May 28, 2026, 02:26 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Feb 24, 2026, 06:19 AM UTC· resolved Feb 24, 2026, 06:25 AM UTC

1. investigating Feb 24, 2026, 06:19 AM UTC

   During a brief window from 07:19 (CET) to 07:24 (CET) today a faulty deployment let to instability in the API that handles search and recommendations for the /v2/ API routes. The cause was a feature update behaving differently in development vs production. The error was highlighted within minutes and the deployment was rolled back. We will perform a root cause analysis, finding out both why the bug doesn't trigger during development and how to automate detection of this case to prevent it from happening again. We sincerely apologize for any inconvenience this has caused and will continue striving to deliver an even better service.

2. resolved Feb 24, 2026, 06:25 AM UTC

   Services have recovered. Post-mortem analysis underway.

Read the full incident report →

Started Jan 20, 2026, 09:08 AM UTC· resolved Jan 20, 2026, 09:25 AM UTC

1. investigating Jan 20, 2026, 09:08 AM UTC

   We are currently experiencing an API slowdown caused by issues with one of our databases. The root cause has been identified, and our team is actively working on resolving the problem. We will provide updates as progress is made.

2. resolved Jan 20, 2026, 09:25 AM UTC

   The issue has been resolved, and the API is fully operational again. We are currently reviewing the incident and implementing measures to prevent such incidents from happening in the future.

Read the full incident report →

Started Oct 20, 2025, 07:58 AM UTC· resolved Oct 21, 2025, 05:14 AM UTC

1. investigating Oct 20, 2025, 07:58 AM UTC

   My.clerk.io and support are unavailable The production API is currently not affected - you will still see search and recommendation results on your store. Several of our service providers are currently experiencing issues, leading to timeouts and errors that prevent us from loading your account information into my.clerk.io or accessing our support tools. As this appears to be a wider internet issue, there is not much you or we can actively do at this time. We are monitoring the situation closely and will provide updates here as soon as we have more information.

2. resolved Oct 21, 2025, 05:14 AM UTC

   Service is now fully restored. At 00:53 CEST, AWS reported their services as fully recovered. At 03:33 CEST, our downstream provider reported their service restored as well, automatically restoring full and reliable access to my.clerk.io. We would like to reiterate that the production API and CDNs, which are used to provide the search, recommendations, and chat services for your webshop, as well as our email-related products, were unaffected during this time. Only my.clerk.io was sporadically unavailable. Nevertheless, we will be undertaking a project to ensure that the availability of my.clerk.io will not be relying on third parties to this degree but will instead gracefully degrade and let you access all your data using usual clerk.io tools, minus whatever component is currently having issues, whether that is payments or support chat.

Read the full incident report →

Started Sep 11, 2025, 07:03 AM UTC· resolved Sep 11, 2025, 02:07 PM UTC

1. investigating Sep 11, 2025, 07:03 AM UTC

   We are currently experiencing intermittent failures to requests for search and recommendations. Engineering is working to restore regular service. More information and RCA will follow.

2. resolved Sep 11, 2025, 02:07 PM UTC

   Post Mortem - 2025-09-11 - Intermittent API failures All times are CEST. Overview Earlier today logging, search, and recommendation API endpoints began exhibiting intermittent failures, not returning any results. We’re sorry for the disruption this caused to you and your customers. Impact Affected services: logging, search (v2), and recommendation (v2) APIs Symptoms: intermittent errors responses and missing results Windows: 08:52–09:25 CEST and ~12:20 (brief) Timeline 08:19 - A slow increase in incoming live updates (and CRUD requests) begin. The increase is masked by the rise of requests that happen every morning. 08:52 - Our inter-service messaging and event system reaches its memory limit, causing the first failed requests. As messages are consumed, capacity is freed up, leading to the intermittent nature of the issue. 09:02 - The issue is confirmed in monitoring and SRE begins investigating. 09:10 - Engineering confirms the issue in the messaging and event system and begins raising resources to increase event processing. 09:25 - Issue is resolved and intermittent failures cease. 12:20 - Another severe spike in incoming live updates cause another few minutes of intermittent request failures. Root Cause Analysis The root cause was the lack of available resources in the messaging and event system. A transient increase in load from live-updates led to it quite simply hitting a hardware limit and stopped accepting new events. Parts of the API rely on this system to perform session-, message-, user activity-, and usage-logging. With the system refusing new messages and events these parts of the API failed. Remediation Several avenues of remediation are being pursued, most have already been implemented: 1. Completed: The parts of the API that rely on being able to hand over events to the messaging and event system have been modified to 'degrade gracefully'. That means that in the case our infrastructure has issues, you will still get search results and recommendations back from us, instead we disable the tracking of usage and other logging events. 2. Completed: Increased resources for the messaging and event system. We have increased the amount of resources in the system eight-fold (8x), allowing for the absorption of much larger transient loads. 3. Planned for tonight: Separation of specific event types. We will be separating some of the event types out to another more resilient system, isolating it from large spikes in live updates and increasing its durability from hardware errors. We sincerely apologize for the inconvenience this has caused not just to you, but your customers and want to reassure you that we are not done looking into ways we can further strengthen our systems.

Read the full incident report →

Started Aug 29, 2025, 08:01 AM UTC· resolved Aug 29, 2025, 11:14 AM UTC

1. investigating Aug 29, 2025, 08:01 AM UTC

   Doing internal updates of dependencies, we encountered a problem that affected some parts of our search and recommendations. We are very sorry for the inconvenience

2. resolved Aug 29, 2025, 11:14 AM UTC

   Postmortem - 2025-08-29 Overview Earlier today, the search, recommendation, and chat services at api.clerk.io began malfunctioning, returning an InternalError with the message: "An internal error occurred. If you need assistance, please contact our support team." Timeline: All times CEST. 09:29 - A dependency update from our APM provider (Application Performance Monitoring) was staged for test and deployment. It contained no behavior changes on our part; dependency updates like these are routinely performed without issue. 09:31 - The update entered the testing pipeline and underwent unit and integration tests. 09:43 - The full testing pipeline was completed with no issues, and a blue/green deployment was started in production. A full set of servers was spun up with the new APM dependency. 09:44 - All new instances reported ready and began taking fractional traffic. 09:49 - No errors were detected in the new instances, and the old instances began spinning down, increasing traffic to the patched instances. 09:50 - An internal Redis server handling data queues showed a rapid—but not critical—rise in memory usage. 09:51 - To avoid crashing, the Redis instance began refusing new write commands. This led to several services that rely on this queue system malfunctioning in unexpected ways. Due to the structure of our internal and external monitoring, these issues occurred in a "gap" between automatic monitors and were not reported. External monitors use an endpoint that does not rely on visitor or message logging, so they did not report the error. Internal monitoring observed Redis closing connections but not out-of-memory conditions, and did not alert. ~10:20 - The issue was reported, and internal escalation began. 10:34 - The issue was identified as stemming from either of two updates, and both were staged for rollback. 10:47 - All code changes were rolled back, and normal operation was restored. Remediation: Several measures have been and will be taken to prevent similar issues in the future: The endpoint used for external monitoring will be modified to be more thorough and to "fail" earlier. External monitoring will be strengthened with additional requests that actively exercise different features, allowing us to detect issues in specific subsystems. Internal policy and training for error reporting and escalation will be updated to ensure faster responses to errors reported outside our established monitoring. The issue cannot yet be reproduced in development; we will work with the APM vendor to reproduce and isolate the behavior in a controlled environment and will update this document with a confirmed root cause.

Read the full incident report →

Started Jul 08, 2025, 07:05 AM UTC· resolved Jul 08, 2025, 07:30 AM UTC

1. investigating Jul 08, 2025, 07:05 AM UTC

   We have experienced a slowdown in our API performance. It's solved, and we are actively working to investigate what caused the issue. We appreciate your patience.

2. resolved Jul 08, 2025, 07:30 AM UTC

   The issue is resolved and we keep investigating further.

Read the full incident report →

Started Dec 09, 2024, 02:10 PM UTC· resolved Dec 09, 2024, 02:45 PM UTC

1. investigating Dec 09, 2024, 02:10 PM UTC

   We are currently experiencing issues with our datastore zone 2. This is now only effecting customers in datastore zone 2.

2. resolved Dec 09, 2024, 02:45 PM UTC

   Recap: At around 15:03 CET one of our database servers began timing out on new connections, leading to API requests for stores on that particular database to experience long wait times which led to system instability. The devops team was notified by automatic monitoring and by 15:10 CET all affected stores had been put into maintenance mode, stabilizing the API operations for all other customers. The issue was diagnosed as a bug in our database software, and the affected machine was patched and re-validated before being put back into production. By 15:21 CET the first batch of stores was pulled from maintenance mode and put back into full production under monitoring. By 15:32 CET the final store was back to full production.

Read the full incident report →