Kami incident

Kami: API Outage

Critical Resolved View vendor source →

Kami experienced a critical incident on March 6, 2024 affecting API and Uploads and 1 more component, lasting 17m. The incident has been resolved; the full update timeline is below.

Started
Mar 06, 2024, 09:05 PM UTC
Resolved
Mar 06, 2024, 09:22 PM UTC
Duration
17m
Detected by Pingoru
Mar 06, 2024, 09:05 PM UTC

Affected components

APIUploadsKami Frontend

Update timeline

  1. investigating Mar 06, 2024, 09:01 PM UTC

    We're investigating an outage that started at 12:41 PM Pacific time, our engineers were alerted to the issue about 5 minutes later, and have been investigating the issue since. This is affecting our backend, so logging into Kami and loading documents are affected. Kami's offline mode should automatically kick in to ensure all work is saved. We will update you once we have more information.

  2. monitoring Mar 06, 2024, 09:05 PM UTC

    We've identified the cause of the outage now, and at 01:03 PM Pacific Time, we've seen the API has recovered, and our error rates have decreased. We're still monitoring the situation, but the outage appears to be resolved. We're seeing the success rate of document opens, login rates and annotations being saved are now improving to normal levels.

  3. resolved Mar 06, 2024, 09:22 PM UTC

    Our engineers have been monitoring Kami's health metrics, and as of 01:01 PM Pacific Time, we saw a full recovery of the Kami app to normal levels. This outage was caused by an issue that occurred on our side while running a database query. If you continue to experience issues, please email us at [email protected] so that we can track down the root cause. We're sorry you experienced this outage, and we want to thank you for your patience as we worked to resolve it. Correction