Oso incident

Stale reads in us-east-1

Notice Resolved View vendor source →

Oso experienced a notice incident on July 8, 2025, lasting 1h 55m. The incident has been resolved; the full update timeline is below.

Started
Jul 08, 2025, 01:48 PM UTC
Resolved
Jul 08, 2025, 03:44 PM UTC
Duration
1h 55m
Detected by Pingoru
Jul 08, 2025, 01:48 PM UTC

Update timeline

  1. identified Jul 08, 2025, 01:48 PM UTC

    We have identified that certain nodes stopped processing writes, which result in their returning stale results. We have replaced the affected nodes and are preemptively replacing all of the nodes in the region to remediate the issue.

  2. monitoring Jul 08, 2025, 02:35 PM UTC

    We have confirmed that all affected nodes are no longer in use and reads are no longer stale. We are continuing with the regional restart.

  3. resolved Jul 08, 2025, 03:01 PM UTC

    The regional restart is complete. We continue to observe that reads are accurate.

  4. postmortem Jul 11, 2025, 02:43 PM UTC

    Link to postmortem: [https://osohq.notion.site/Stale-reads-in-us-east-1-22a9f1471f2b80bd8808f8b6cb76a44d](https://osohq.notion.site/Stale-reads-in-us-east-1-22a9f1471f2b80bd8808f8b6cb76a44d)