SimpleVoIP experienced a major incident on November 16, 2020 affecting Core Voice Services and SimpleVoIP Admin Portal, lasting 4h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Nov 16, 2020, 05:14 PM UTC
We have identified an issue impacting the SimpleVoIP web portal that is also disrupting a portion of our phone traffic. Our engineering team is working to resolve this now.
- identified Nov 16, 2020, 05:28 PM UTC
We are still working to get these services up and running again as quickly as possible.
- monitoring Nov 16, 2020, 05:33 PM UTC
Services have been restored. We are keeping a close eye on this to ensure it remains stable.
- resolved Nov 16, 2020, 09:22 PM UTC
All services are now stable and have remained so for several hours. A formal RFO will be available through this status page within the next couple of days following some internal investigation. We apologize for any disruption this caused you, your users, or your customers.
- postmortem Nov 19, 2020, 05:51 PM UTC
SimpleVoIP encountered a database indexing issue on Monday morning at around 10:30am CST that caused some frequently-running queries to slow down, resulting in a traffic jam-like climb in waiting queries. After hitting the database connection limit, our other services were no longer able to connect in a timely fashion, which caused significant failures to our web services, including some services involved in routing calls for a subset of our customers. We resolved this by fixing the underlying indexing issue and clearing the list of backlogged queries, which brought the connection count and our response times back to normal levels at around 11:15am CST. This specific issue was resolved for good thanks to the change applied as part of the fix. Longer term, our engineers are continuing work on an ongoing project to migrate as many of our database operations as possible to redundant replicas to reduce load on the primary database and prevent similar issues from occurring in the future, both by preventing the lockup of an individual database and by providing redundancy for our critical services in the event that a database does experience issues. We apologize for the disruption that this caused to you and to your customers. Keeping your call services stable is our highest priority, and we thank you for your patience as we continue to improve the stability of our platform.