SheerID experienced a minor incident on December 15, 2023 affecting API, lasting 2h 56m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 15, 2023, 05:33 PM UTC
We are currently investigating this issue.
- identified Dec 15, 2023, 06:22 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Dec 15, 2023, 06:24 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Dec 15, 2023, 08:30 PM UTC
This incident has been resolved.
- postmortem Dec 15, 2023, 10:16 PM UTC
At 07:42 UTC on December 15th, SheerID engineering updated a Database index to improve query performance. Initially, the change looked good until 10:30 UTC, when a particular type of API request used the new index in a way SheerID engineering did not intend. The database CPU spiked but recovered quickly, and the alert cleared. Then, at 14:34 UTC, SheerID saw a new batch of requests that caused another Database CPU spike, but this time, the system started to show a growing impact on our API latency. SheerID Engineering was alerted again and worked to identify the problematic query and fix the problematic index. At 18:30 UTC, the Database returned to typical CPU levels of ~20%, and API latency was restored to typical.