Avochato experienced a critical incident on May 19, 2021 affecting avochato.com and API and 1 more component, lasting 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 19, 2021, 08:38 PM UTC
We are investigating an issue preventing users from accessing Avochato.
- resolved May 19, 2021, 08:40 PM UTC
This incident has been resolved.
- postmortem May 19, 2021, 09:13 PM UTC
## What happened During routine development cycle, database provisioning for a new service within our cloud services provider inadvertently applied an authorization change to our production database cluster. This caused requests to the Avochato app and homepage to fail temporarily until the changes were reverted. ## Resolution We immediately reverted the change when operations identified that it impacted production and the change rolled back out. ## Impact While Avochato was unavailable, queued automations and uploads entered our retry queues and successfully continued once we recovered. Incoming messages received during the period were not lost. Moving forward, our team has adjusted our processes for provisioning services regardless of their proximity to production services.