Oso had 12 outages in the last 2 years totaling 298h 45m of downtime — averaging 0.5 incidents per month.
There were 12 Oso outages since June 3, 2025 totaling 298h 45m of downtime. Each is summarised below — incident details, duration, and resolution information.
Users in our us-east-1 region may observe HTTP 5xx errors on write requests or when using the Oso UI. We are investigating.
We have a deployed a fix and are monitoring the results.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
The issue has resolved. We are keeping in a Monitoring status while we work on a longer term fix to resolve the underlying root cause.
We have implemented a fix to mitigate future occurrence of this issue.
We have temporarily paused Oso Sync processing while we resolve an infrastructure issue. Syncs submitted during this window will not be processed. Authorization requests via Oso Cloud are unaffected. We will provide updates as the situation progresses.
This incident has been resolved.
We are investigating an issue affecting fallback snapshot freshness. Our team has identified the cause and is actively working on remediation. We will provide updates as the situation progresses.
This incident has been resolved.
We have identified an issue with our integration with S3 that is causing Oso Sync jobs to intermittently fail or time out. We have escalated the issue to AWS and are exploring other mitigations we can make.
We've deployed a fix and are monitoring the results.
Since deploying the fix, we have not observed additional failures or timeouts related to our integration with S3.
We are investigating occurrences of failed writes in the eu-west-1 region.
We have identified the affected cloud API resources and are applying remediation.
We are seeing a recovery in write traffic following the mitigation. We are continuing to monitor this region.
This incident has been resolved.
Between Oct 20 07:11 AM UTC and Oct 20 09:27 AM UTC, our system alarms for AWS started paging the On-Call team. We confirmed from the AWS status page that AWS was having an incident: https://health.aws.amazon.com/health/status AWS has reported that they have found the root cause and customers should be seeing recovery. Our internal monitoring dashboards have been available throughout this time and report that the Oso Cloud platform has been handling authorization decisions and write traffic without disruption. Given this, we currently believe there is no visible impact for Oso customers. We will continue to monitor and share any relevant status updates.
Between Oct 20 07:11 AM UTC and Oct 20 09:27 AM UTC, our system alarms for AWS started paging the On-Call team. We confirmed from the AWS status page that AWS was having an incident: https://health.aws.amazon.com/health/status AWS has reported that they have found the root cause and customers should be seeing recovery. Our internal monitoring dashboards have been available throughout this time and report that the Oso Cloud platform has been handling authorization decisions and write traffic without disruption. Customers using Oso Fallback nodes may have observed stale authorization decisions from Fallback instances during this time, but traffic should not have been routed to Fallback instances because Oso Cloud was responsive. Fallback instances should now be up to date. We currently believe this was the only visible impact for Oso customers. We will continue to monitor and share any relevant status updates.
AWS has reported that they are seeing network issues, which has coincided with customer reports of gateway timeouts attempting to reach the Oso Service in us-east-1. We are investigating the impact and exploring failover options.
AWS continues to experience networking issues in us-east-1, resulting in increased latency for some requests. We have applied an update to Oso Cloud to temporarily route traffic to alternate regions.
We have removed `us-east-1` from our DNS traffic routing policy for `api.osohq.com` and `cloud.osohq.com`. We have confirmed that all API traffic is now being routed to other nearby regions. We did not observe errors from our service during this incident, but we proceeded with the failover out of caution and due to inconsistent visibility. We have some shared services in `us-east-1` that continue to report healthy status, and we have not seen any other impacts. We will continue to monitor and update on any changes.
AWS is continuing to stabilize us-east-1. Out of an abundance of caution, we will continue to avoid routing traffic to this region until their incident is fully resolved.
The AWS incident has been resolved and their us-east-1 region appears to be stable. We have reverted our temporary mitigations to allow Oso Cloud traffic to flow to us-east-1 once more.
From 12:32-12:52 UTC, customers connecting to Oso from the us-west-1 region may have received HTTP errors or experienced increased latency. Postmortem to follow.
Link to postmortem: [https://osohq.notion.site/2025-10-05-unavailability-in-us-west-1](https://osohq.notion.site/2025-10-05-unavailability-in-us-west-1)
We have identified an increase in delay in write processing for some customers. We have identified the source to be a backlog of writes in our processing queue, and a sequential bottleneck for some environments. The backlog of writes has completed processing and we are monitoring the replication lag as it returns to normal levels.
We are continuing to monitor the replication recovery. Some affected customers should be seeing their workloads returning to normal, others should continue to see a reduction in write delay.
Over the next hour environments should see gradual recovery in write delays, with full recovery estimated in 2 hours.
Write latencies to all affected Oso Cloud environments have recovered.
Link to postmortem: [https://osohq.notion.site/2025-09-22-Write-delays-for-certain-customers-2769f1471f2b80e3b36fd94ada7f34ce](https://osohq.notion.site/2025-09-22-Write-delays-for-certain-customers-2769f1471f2b80e3b36fd94ada7f34ce)
We have identified that certain nodes stopped processing writes, which result in their returning stale results. We have replaced the affected nodes and are preemptively replacing all of the nodes in the region to remediate the issue.
We have confirmed that all affected nodes are no longer in use and reads are no longer stale. We are continuing with the regional restart.
The regional restart is complete. We continue to observe that reads are accurate.
Link to postmortem: [https://osohq.notion.site/Stale-reads-in-us-east-1-22a9f1471f2b80bd8808f8b6cb76a44d](https://osohq.notion.site/Stale-reads-in-us-east-1-22a9f1471f2b80bd8808f8b6cb76a44d)
Our website hosting provider is currently experiencing an ongoing incident. We are actively monitoring. Oso Cloud services are unaffected. If you're looking for our docs, you can access them here: https://www.osohq.com/docs.
We are continuing to monitor for any further issues.
Our website hosting provider is seeing improvements and continuing to monitor.
Our website hosting provider has resolved the incident. Oso Cloud services remain unaffected.