Omnivore experienced a major incident on April 12, 2023 affecting Aloha Cloud Connect, lasting 29d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 12, 2023, 05:44 PM UTC
Beginning around 10:00 UTC, we began seeing an increased number of timeouts when calling the NCR CloudConnect API impacting employee and job reads. API calls to fetch employees may present stale data. We are reaching out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.
- identified Apr 12, 2023, 07:24 PM UTC
We are continuing to see an increased error rate when calling the NCR CloudConnect API for employee and job data. API calls to fetch employees may present stale data. We have reached out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.
- monitoring Apr 12, 2023, 08:28 PM UTC
We are continuing to see an increased error rate when calling the NCR CloudConnect API for employee and job data. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time. Please refer to this NCR Incident for details: https://status.aloha.ncr.com/incidents/cnl38krr6n6b
- monitoring Apr 13, 2023, 01:37 PM UTC
Beginning around 8:03 UTC, we began seeing an increased number of timeouts when calling the Ticket routes of the NCR CloudConnect API. Based on this, we are upgrading the scope of the outage. API calls to fetch Tickets will likely fail and Ticket webhooks will be delayed until the NCR outage resolves. NCR is continuing to update their status page (at https://status.aloha.ncr.com/incidents/cnl38krr6n6b). We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.
- monitoring Apr 13, 2023, 08:32 PM UTC
Around 20:00 UTC, the number of timeouts when calling the Ticket routes of the NCR CloudConnect API decreased to normal levels. We are continuing to see an increased error rate when calling the NCR CloudConnect API for employee and job data. API calls to fetch employees may present stale data. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.
- monitoring Apr 25, 2023, 07:12 PM UTC
Beginning around 13:35 UTC, we began seeing an increased number of timeouts when calling the Ticket routes of the NCR CloudConnect API. API calls to fetch Tickets will likely fail and Ticket webhooks will be delayed until the NCR outage resolves. NCR is continuing to update their status page (at https://status.aloha.ncr.com/incidents/cnl38krr6n6b). We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.
- resolved May 11, 2023, 07:26 PM UTC
Starting at 17:45 UTC, we observed calls to the NCR CloudConnect API begin succeeding. Cached data is successfully being updated over time in batches. We will continue to closely monitor the success of these background jobs and for the success of calls to the NCR CloudConnect API.