Omnivore experienced a major incident on October 3, 2022 affecting Toast, lasting 1h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 03, 2022, 08:34 AM UTC
Beginning around 8:15 UTC, we began seeing an increased number of errors when calling the Toast API, impacting ticket and clock entry reads. API calls to fetch ticket and clock entry data will likely fail at an increased rate and webhooks may be delayed until service is restored. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.
- resolved Oct 03, 2022, 09:37 AM UTC
At 9:30 UTC, we found that a service responsible for authenticating against the Toas API had previously lost its database connection and failed to reconnect. We restarted the service and see that Toast API connections have now returned to normal.