Omnivore incident
NCR CloudConnect API - Increased error rate
Omnivore experienced a critical incident on March 5, 2025 affecting Aloha Cloud Connect, lasting 3h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Mar 05, 2025, 07:39 PM UTC
Beginning around 19:20 UTC, we began seeing an increased number of timeouts when calling the NCR CloudConnect API impacting ticket and clock entry reads. All API calls to fetch ticket and clock entry data are failing and webhooks may be delayed until service is restored. We are reaching out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.
- monitoring Mar 05, 2025, 08:43 PM UTC
At this time, we have no further updates from NCR regarding an estimated resolution. We will continue to monitor the situation and update as soon as the situation changes.
- monitoring Mar 05, 2025, 10:29 PM UTC
NCR posted an update to their statuspage at 22:13 UTC indicating they have found a possible solution and are working on stabilizing services. Omnivore calls to the NCR CloudConnect API began seeing a return to more normal error rates at 22:10 UTC and have been maintaining those rates since. We will continue to monitor error rates until NCR has confirmed they have resolved the issue.
- resolved Mar 05, 2025, 11:09 PM UTC
NCR has marked their ordering components as operational and Omnivore has continued to see normal error rates since recovery.