Avochato experienced a minor incident on July 27, 2021 affecting avochato.com and API and 1 more component, lasting 1h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 27, 2021, 08:03 PM UTC
We are currently investigating this issue.
- identified Jul 27, 2021, 08:09 PM UTC
Certain features of the Avochato app are being impacted by a partial cloud outage. We are working to restore full coverage.
- monitoring Jul 27, 2021, 08:26 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Jul 27, 2021, 08:39 PM UTC
We are continuing to monitor. Specific inboxes may continue to be impacted.
- monitoring Jul 27, 2021, 08:50 PM UTC
We are continuing to monitor while we partner with our cloud provider to resolve an issue with cloud hardware. Thank you for your patience while we resolve this issue for the subset of impacted inboxes.
- monitoring Jul 27, 2021, 09:18 PM UTC
We are seeing improvements for the impacted inboxes and are continuing to monitor.
- resolved Jul 27, 2021, 09:53 PM UTC
This incident has been resolved.
- postmortem Jul 27, 2021, 09:53 PM UTC
## What Happened A subset of our cloud infrastructure suffered a hardware failure, which caused specific customers associated with that hardware to be unable to view broadcasts and contacts, as well as intermittent issues polling for live inbox updates. Once the hardware issue was resolved, traffic for the specific subset of impacted inboxes returned to normal. No data was lost. ## Resolution Avochato cloud engineers escalated the issue to our cloud partner, who escalated the issue internally. The hardware failure was resolved and our team will continue working with the vendor to ensure that similar incidents properly rotate the faulty hardware out of the cluster.