Avochato experienced a notice incident on December 15, 2020 affecting avochato.com and API and 1 more component, lasting 2h 20m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 15, 2020, 03:39 PM UTC
We are investigating an issue that is preventing scheduled broadcasts from moving beyond the "sending" phase.
- identified Dec 15, 2020, 04:49 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Dec 15, 2020, 06:00 PM UTC
This incident has been resolved. Broadcasts that were scheduled to begin during the affected period will be restarted.
- postmortem Dec 15, 2020, 08:18 PM UTC
A software update relating to the broadcast scheduling mechanism introduced a regression in our system, which caused broadcasts to stay stuck in an initial “sending” state. This has been resolved with a subsequent software update. This issue was not caught by automated or QA testing, and we have updated our automated testing and quality assurance controls to prevent a similar issue from happening in the future. Affected broadcasts eventually retried and delivered their message to their audience after a delay.