Search and Recommendations ...

Update timeline

1. investigating Aug 29, 2025, 08:01 AM UTC

   Doing internal updates of dependencies, we encountered a problem that affected some parts of our search and recommendations. We are very sorry for the inconvenience

2. resolved Aug 29, 2025, 11:14 AM UTC

   Postmortem - 2025-08-29 Overview Earlier today, the search, recommendation, and chat services at api.clerk.io began malfunctioning, returning an InternalError with the message: "An internal error occurred. If you need assistance, please contact our support team." Timeline: All times CEST. 09:29 - A dependency update from our APM provider (Application Performance Monitoring) was staged for test and deployment. It contained no behavior changes on our part; dependency updates like these are routinely performed without issue. 09:31 - The update entered the testing pipeline and underwent unit and integration tests. 09:43 - The full testing pipeline was completed with no issues, and a blue/green deployment was started in production. A full set of servers was spun up with the new APM dependency. 09:44 - All new instances reported ready and began taking fractional traffic. 09:49 - No errors were detected in the new instances, and the old instances began spinning down, increasing traffic to the patched instances. 09:50 - An internal Redis server handling data queues showed a rapid—but not critical—rise in memory usage. 09:51 - To avoid crashing, the Redis instance began refusing new write commands. This led to several services that rely on this queue system malfunctioning in unexpected ways. Due to the structure of our internal and external monitoring, these issues occurred in a "gap" between automatic monitors and were not reported. External monitors use an endpoint that does not rely on visitor or message logging, so they did not report the error. Internal monitoring observed Redis closing connections but not out-of-memory conditions, and did not alert. ~10:20 - The issue was reported, and internal escalation began. 10:34 - The issue was identified as stemming from either of two updates, and both were staged for rollback. 10:47 - All code changes were rolled back, and normal operation was restored. Remediation: Several measures have been and will be taken to prevent similar issues in the future: The endpoint used for external monitoring will be modified to be more thorough and to "fail" earlier. External monitoring will be strengthened with additional requests that actively exercise different features, allowing us to detect issues in specific subsystems. Internal policy and training for error reporting and escalation will be updated to ensure faster responses to errors reported outside our established monitoring. The issue cannot yet be reproduced in development; we will work with the APM vendor to reproduce and isolate the behavior in a controlled environment and will update this document with a confirmed root cause.