Avochato experienced a minor incident on June 8, 2021 affecting avochato.com and API and 1 more component, lasting 4h 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 08, 2021, 07:12 PM UTC
Our team is investigating slower than average response times when using the app.
- identified Jun 08, 2021, 08:02 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jun 08, 2021, 09:25 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 08, 2021, 10:20 PM UTC
This incident has been resolved.
- postmortem Jun 10, 2021, 09:04 PM UTC
_Note This incident was unrelated to an outage relating to CDNs that may have impacted customers sending MMS images with public-facing URLs powered by Fastly \(_[_read more here_](https://www.fastly.com/blog/summary-of-june-8-outage)_\)._ ## What Happened Avochato cloud services were temporarily unable to split traffic across our secure cloud databases, causing only one Avochato database to manage all load on the platform at once time. This resulted in slower than average speeds when processing updates to records and when serving pages throughout the app on mobile and desktop, as well as requests to our API. This slowdown compounded during the middle of the day, causing delays in fetching and writing data as well as delays syncing data to our search nodes. It is also possible that the delays would have impacted any time-sensitive operations such as adding contacts to a broadcast before the broadcast’s scheduled date. ## Resolution The engineering team patched the issue once it was determined safe to do so. Traffic patterns returned to normal and our throughput returned to expected levels. We have also made proactive performance improvements to double the normal throughput for certain data pipelines and implemented monitoring to immediately detect this regression in traffic. We apologize for the inconvenience to you and your team, Christopher, CTO