Omnivore Outage History

Around 16:30 UTC, we applied a change and observed that the number of locations flapping between 'online' and 'degraded' dropped back to baseline levels. We will continue monitoring this issue to make sure our systems remain stable.

Brink has addressed the issue and we are seeing an influx of new tickets as we move to recover tickets missed during the window. We will continue to monitor and post another update once levels get back to normal.

Connectivity to all services have recovered to normal levels. We are continuing to monitor.

Around 07:11 UTC this morning, Amazon Web Services began experiencing an outage (https://health.aws.amazon.com/health/status). The Omnivore team was alerted at 07:24 UTC and has been monitoring ever since. So far, we haven't seen any clear signal of disruption in Omnivore services other than the following: * ~2.4k agents have entered a degraded state. They are not going offline, so API calls to these locations are succeeding, however customers may be experiencing slower than normal response times. * The Brink API experienced elevated error rates from 15:21-15:41 UTC. Omnivore customers would have also seen an elevated error rate for Brink locations and possibly delayed webhooks. We will continue to monitor for new impacts and will send out updates as needed.

Around 17:07 UTC the number of degraded agents returned to baseline levels. We have not observed any other spikes in the error level for Brink API. We are currently not seeing any other areas of concern, but since AWS's incident is still open, we will continue monitoring all of our services for new issues.

Around 19:27 UTC, we observed a spike in the error rate for our calls to the Toast API, approaching a 100% failure rate. At this time there are no actions we can take and we will continue to monitor the situation.

We are still seeing a near 100% error rate for the Toast integration. We will continue to monitor for improvement.

We are still seeing a near 100% error rate for the Toast integration. For further updates, check the Toast status page (https://status.toasttab.com).

A fix has been implemented, with all edge nodes now back in rotation and reporting healthy. We will continue to monitor for any further impacts.

Beginning around 19:20 UTC, we began seeing an increased number of timeouts when calling the NCR CloudConnect API impacting ticket and clock entry reads. All API calls to fetch ticket and clock entry data are failing and webhooks may be delayed until service is restored. We are reaching out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.

At this time, we have no further updates from NCR regarding an estimated resolution. We will continue to monitor the situation and update as soon as the situation changes.

NCR posted an update to their statuspage at 22:13 UTC indicating they have found a possible solution and are working on stabilizing services. Omnivore calls to the NCR CloudConnect API began seeing a return to more normal error rates at 22:10 UTC and have been maintaining those rates since. We will continue to monitor error rates until NCR has confirmed they have resolved the issue.

We have implemented a fix and are observing successful API writes. We will continue to monitor.

Starting at 18:25 UTC, we began seeing an increased number of errors when calling the Brink API, impacting ticket reads. API calls to ticket reads will likely fail at an increased rate and webhooks may be delayed until service returns to normal operation. So far, this seems isolated to a single Brink host. We have reached out to our Brink contacts and will continue to monitor until the situation is resolved. There are no other technical actions we can take to resolve the issue at this time.