Kami experienced a critical incident on March 5, 2024 affecting API and Kami Frontend, lasting 26m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 05, 2024, 04:18 PM UTC
We're investigating an outage that started at 7:35am Pacific time, our engineers were alerted to the issue about 5 minutes later, and have been investigating the issue since. This is affecting our backend, so logging into Kami and loading documents are affected. Kami's offline mode should automatically kick in to ensure all work is saved.
- monitoring Mar 05, 2024, 04:31 PM UTC
We've identified our outage is being caused by Google SSO backend API was down. At 8:21am Pacific Time, we've seen that API recover, and our error rates have decreased. We're still monitoring the situation, but the outage appears to be resolved. We're seeing success rate of document opens, login rates and annotations is being saved improve to normal levels. Correction to previous post, this incident started at 7:35am Pacific Time.
- resolved Mar 05, 2024, 04:45 PM UTC
Engineers have been monitoring Kami's health metrics, by 8:21am Pacific Time / 11:21am Eastern Time, we saw a full recovery of the Kami app to normal levels. This outage did appear to be caused by an outage by Google SSO APIs. If you continue to experience issues, please email us at [email protected] so that we can track down the root cause. We're sorry you experienced this outage, and want to thank you for your patience as we worked to resolve it.