Avochato experienced a minor incident on November 30, 2020 affecting avochato.com and API and 1 more component, lasting 8h 3m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 30, 2020, 05:28 PM UTC
We are currently investigating this issue.
- monitoring Nov 30, 2020, 05:36 PM UTC
A fix has been implemented and we are monitoring the results.
- investigating Nov 30, 2020, 06:03 PM UTC
We are currently investigating this issue.
- investigating Nov 30, 2020, 06:25 PM UTC
We are continuing to investigate this issue.
- investigating Nov 30, 2020, 07:50 PM UTC
We are continuing to investigate this issue.
- monitoring Nov 30, 2020, 08:06 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Nov 30, 2020, 09:14 PM UTC
We are continuing to monitor for any further issues.
- resolved Dec 01, 2020, 12:04 AM UTC
This incident has been resolved.
- postmortem Dec 01, 2020, 03:53 PM UTC
## What Happened A sudden escalation in CPU in our production database cloud database provider led to a period of longer than normal load times. While our platform was under high load for Cyber Monday, our investigation leads us to believe this was a hardware-related failure and our operations team responded accordingly. Additionally, the period of slow response times led to our Slack application automatically turning off as per Slack’s platform policy. Engineers manually turned the app back on numerous times, but unfortunately some messages sent via Slack threads were unable to be delivered during the windows when the Slack app was disabled. ## Resolution Data was not lost as we were able to fail over to a read replica database. We have upgraded database hardware to move away from potentially degraded hardware and performed routine system maintenance.