Omnivore experienced a major incident on October 20, 2025 affecting API and AWS ec2-us-east-1 and 1 more component, lasting 9h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Oct 20, 2025, 04:41 PM UTC
Around 07:11 UTC this morning, Amazon Web Services began experiencing an outage (https://health.aws.amazon.com/health/status). The Omnivore team was alerted at 07:24 UTC and has been monitoring ever since. So far, we haven't seen any clear signal of disruption in Omnivore services other than the following: * ~2.4k agents have entered a degraded state. They are not going offline, so API calls to these locations are succeeding, however customers may be experiencing slower than normal response times. * The Brink API experienced elevated error rates from 15:21-15:41 UTC. Omnivore customers would have also seen an elevated error rate for Brink locations and possibly delayed webhooks. We will continue to monitor for new impacts and will send out updates as needed.
- monitoring Oct 20, 2025, 05:50 PM UTC
Around 17:07 UTC the number of degraded agents returned to baseline levels. We have not observed any other spikes in the error level for Brink API. We are currently not seeing any other areas of concern, but since AWS's incident is still open, we will continue monitoring all of our services for new issues.
- monitoring Oct 20, 2025, 07:46 PM UTC
Around 19:27 UTC, we observed a spike in the error rate for our calls to the Toast API, approaching a 100% failure rate. At this time there are no actions we can take and we will continue to monitor the situation.
- monitoring Oct 20, 2025, 08:55 PM UTC
We are still seeing a near 100% error rate for the Toast integration. We will continue to monitor for improvement.
- monitoring Oct 20, 2025, 10:16 PM UTC
We are still seeing a near 100% error rate for the Toast integration. For further updates, check the Toast status page (https://status.toasttab.com).
- resolved Oct 21, 2025, 01:48 AM UTC
At 00:03 UTC on 10/21 error levels for the Toast integration returned to baseline. As this was the last remaining impacted Omnivore service, we are marking this incident as resolved.