SpectrumVoIP experienced a notice incident on December 1, 2022 affecting Stratus, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 01, 2022, 05:13 PM UTC
We are aware of an incident that may be affecting subscriber call processing and working to resolve it as quickly as possible. Further update to be posted as they become available.
- identified Dec 01, 2022, 06:22 PM UTC
We have isolated the root cause to an upstream transit carrier that services our DFW data center. We have engaged their engineering team and they are working towards a resolution on their end. To expedite mitigating the issue, we will be diverting all call traffic from the affected location to a redundant data center.
- monitoring Dec 01, 2022, 11:28 PM UTC
We are continuing to monitor the situation.
- monitoring Dec 01, 2022, 11:28 PM UTC
We are continuing to monitor for any further issues.
- monitoring Dec 02, 2022, 03:08 AM UTC
We are continuing to monitor for any further issues.
- monitoring Dec 02, 2022, 02:35 PM UTC
We're investigating new reports stemming from this incident. Our engineering teams remain fully engaged and updates to follow.
- resolved Dec 02, 2022, 05:28 PM UTC
SpectrumVoIP can confirm all services are running as expected. This incident is being resolved.
- postmortem Dec 08, 2022, 01:00 AM UTC
The Stratus platform experienced a race condition that caused systemwide call setup failures. It was ultimately determined that the race condition was caused by a previously undiscovered software bug which was triggered by a corrupt systemwide hold audio file. This software bug presented the most challenging diagnostic event that we have ever experienced because of cascading effects, which in turn caused our performance monitoring software to indicate a network related issue. Because the file corruption occurred two days prior to the event, it was buried over 2,200 lines deep in our change logs. Over the weekend, a software patch was put in place by the switch vendor so that a file corruption of this nature cannot impact normal call setup in the future. We understand the importance of your business communications and the trust that you place upon us to safeguard the reliability of our service. We sincerely apologize for the difficulties caused by this event, and we continuously strive to ensure that issues of this nature will not reoccur.