Boostlingo experienced a major incident on June 19, 2024 affecting Boostlingo Group Rooms and Boostlingo Communication REST API, lasting 1h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 19, 2024, 02:18 PM UTC
We are currently investigating issues of calls not connecting.
- monitoring Jun 19, 2024, 02:33 PM UTC
A fix has been applied and we are monitoring the results.
- resolved Jun 19, 2024, 02:48 PM UTC
This incident has been resolved. We will continue our full investigation into the root cause and follow-up with a debrief in the next 48 hours.
- postmortem Jun 21, 2024, 05:15 PM UTC
At 5:59 a.m. PT on 6/19/2024, our internal monitoring alerted technical teams of errors that could cause system degradation. Our team immediately jumped in to identify the issue and work towards a resolution. At 6:05 a.m. PT, our support and account management teams began to receive notifications of long hold times and call failures. These were surfaced to our technical team for help with investigation and mitigation. Our investigation focused on both identifying the issue and identifying if the delivery assurance plan \(DAP\) would be effective in mitigating the issue. After thorough investigation and review, our development team initiated our delivery assurance plan at 7:17 a.m. PT. The delivery assurance plan was in place until 7:29 a.m. PT when our development team implemented a fix that resolved the underlying root cause of the issue, and calls were able to return to normal. The root cause was an issue with a database table responsible for call routing. There was a counter within the table that reached its maximum value, which prevented records from being added. Our team took action to ensure data retention and performed a reset on the counter, which allowed new records to be added. This fix will allow calls to flow indefinitely. The Development Team will also add additional development work to strengthen our monitors and technical infrastructure. If you would like additional information, please get in touch with your account manager directly or our support team at [[email protected]](mailto:[email protected]). We will have a more detailed root cause analysis available in the next two weeks, which we can share directly