Boostlingo experienced a critical incident on April 13, 2023 affecting Boostlingo SMS and Boostllingo Voice IVR and 1 more component, lasting 5h 8m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 13, 2023, 08:07 PM UTC
We are currently investigating this issue.
- investigating Apr 13, 2023, 08:35 PM UTC
We are continuing to investigate this issue.
- investigating Apr 13, 2023, 08:40 PM UTC
We are continuing to investigate this issue.
- monitoring Apr 13, 2023, 09:01 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Apr 13, 2023, 09:05 PM UTC
We are continuing to monitor for any further issues.
- monitoring Apr 13, 2023, 09:45 PM UTC
As we are continuing to monitor we have seen similar behaviors to what previously lead to the disruptions. We may see degraded performance as our team works to ensure consistent stability.
- resolved Apr 14, 2023, 01:15 AM UTC
This incident has been resolved.
- postmortem Apr 14, 2023, 01:16 AM UTC
On April 13th 19:43 UTC, we experienced a series of database deadlocks that caused several services to lose access to the database and required a restart. The following is a summary of the incident and the steps taken to resolve the issue. The occurrence of deadlock situations in database tables resulted in numerous queries timing out, ultimately leading to several services losing database access and requiring a restart. The identified queries responsible for locking the tables were rectified, and additional indexes were incorporated to enhance performance concerning the updated queries. A hotfix \(version 6.0.3\) is scheduled for tonight April 14th to resolve an outstanding query that could not be instantly patched. The said query contributed to the deadlocks by retrieving substantial amounts of data.