SimpleVoIP experienced a major incident on October 28, 2024 affecting Core Voice Services, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 28, 2024, 05:36 PM UTC
We are investigating reports of significant delay before calls complete, including parked calls. These appear to be linked to last night's platform upgrade. Our Engineering team is working to alleviate this delay as quickly as possible.
- identified Oct 28, 2024, 06:22 PM UTC
Engineers are currently implementing a workaround fix. We should know in just a few minutes whether any issues persist.
- monitoring Oct 28, 2024, 06:43 PM UTC
The fix has been applied, and traffic across all zones has returned to normal. We will continue to keep a close eye out for any spikes in call errors or delay, but the applied fix will prevent this specific issue from reoccurring.
- resolved Oct 29, 2024, 05:48 PM UTC
All services have been stable since the fix was implemented yesterday afternoon. A more permanent fix will be applied to prevent this from reoccurring on future platform upgrades. An RFO with more detail will be posted here within the next couple of days.
- postmortem Nov 01, 2024, 09:37 PM UTC
At around 10:15am EDT on Mon, Oct 28, our monitoring system alerted us to a higher-than-normal call failures. Our Engineering team started the investigation to determine whether these calls were failing over via redundant routes. It was unknown whether there was any impact to end users at this time. We saw increased load on our application servers at around the same time, resulting in high response times for API and web activities. We determined that high database load was likely responsible for both the call failures and the slow response times. Our database servers were running a schema migration script as a follow-up to the voice platform upgrade that we had performed overnight, which we stopped at 1:15pm EDT with the intent to restart it after hours. That didn’t bring the load down, which led us to an automatic reindexing process that had started on the database servers after the upgrade. We stopped that process, bringing the database load back to normal and restoring both API response times and call traffic to normal as well. Our Engineering team has made a configuration change to our database servers to prevent this automatic reindexing from occurring again. This will prevent any future disruptions following upgrades or database migrations, including the DB capacity upgrade scheduled for 2024/11/03-04. We apologize to everyone who was affected by this service disruption. We do our best to mitigate the impacts of any maintenance by testing thoroughly during the maintenance window and making any necessary adjustments before business hours. Unfortunately, due to the load-based nature of this incident, we didn’t see any issues until traffic started ramping up in the morning. We thank you all for your patience as we approach the end of our capacity expansion project with next Monday morning’s maintenance.