Oso incident

Degradation in write performance leading to delays in data ingestion

Oso experienced a minor incident on May 9, 2024 affecting Authorization APIs, lasting 3h 51m. The incident has been resolved; the full update timeline is below.

Started: May 09, 2024, 05:57 PM UTC
Resolved: May 09, 2024, 09:49 PM UTC
Duration: 3h 51m
Detected by Pingoru: May 09, 2024, 05:57 PM UTC

Affected components

Authorization APIs

Update timeline

investigating May 09, 2024, 05:57 PM UTC

We are observing degraded performance in our data ingestion pipelines leading to delays in writes propagating to all nodes.
monitoring May 09, 2024, 06:28 PM UTC

Between 17:30 and 17:54 UTC we observed a slower processing time of writes which caused delays in writes propagating and persisting to all nodes. During this time, authorization requests would be using stale data. We are continuing to monitor the situation while performing an analysis of the underlying issue.
resolved May 09, 2024, 09:49 PM UTC

We've seen no further degradation since 17:54 UTC and are marking this as resolved. Additionally, we've identified and deployed performance improvements to our data processing to alleviate future issues.