SpectrumVoIP experienced a major incident on September 28, 2021 affecting Stratus and Inbound Calling and 1 more component, lasting 5d 4h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 28, 2021, 02:15 PM UTC
Our preliminary data indicates the DDoS attack responsible for the previous days issues has likely returned to the upstream provider's infrastructure. Their status portal also confirms partial outages across their services. Outbound calling and emergency calling remains routed around the provider and is not currently impacted.
- identified Sep 28, 2021, 02:53 PM UTC
The upstream carrier's teams are actively investigating the incident on their end impacting voice calls with intermittent service disruptions. We remain in constant contact with them and continue to monitor the situation. We routed all outbound calling around the major upstream provider of SpectrumVoIP's which is experiencing this issue on 9/25/21 and will continue to exclude them from our outbound calling for the duration of this incident. You may still experience intermittent issues with inbound calling. Issues include: No audio on inbound calls, DTMF (key presses) not being passed to menus on inbound calls, and intermittent inbound call failures. If you receive a call without audio, there will be audio if you call them back as the outbound call routes to an unaffected carrier. For inbound DTMF (button presses) not working on caller menus, a temporary work around is to configure the time-out option on your IVR to route the caller to a ring group or other destination of your choice. Please see: https://support.spectrumvoip.com/hc/en-us/articles/4407660020627-Update-Auto-Attendant-IVR-Time-Out-Destination We will continue to provide updates as additional details become available.
- identified Sep 28, 2021, 04:12 PM UTC
The upstream provider has acknowledged the DDoS attack has returned to their network. At this time we are not observing a large impact to our inbound calling services. Outbound calling remains routed around the provider and is not currently impacted. We continue to monitor the situation and will update the status as appropriate.
- identified Sep 28, 2021, 06:46 PM UTC
The upstream provider continues to acknowledge the DDoS attack has returned to their network. At this time we are observing a very limited impact to our inbound calling services. Outbound calling remains routed around the provider and is not currently impacted. We continue to monitor the situation and will update the status as appropriate.
- identified Sep 28, 2021, 09:19 PM UTC
The upstream provider continues to acknowledge the DDoS attack is impacting their network and they are actively working to mitigate its effects. At this time we are observing minimal impact to our inbound calling services. Outbound calling remains routed around the provider and is not currently impacted. We continue to monitor the situation and will update the status as appropriate.
- monitoring Sep 29, 2021, 03:31 AM UTC
We have upgrade the status of this incident to Monitoring at this time. We will leave this incident open until there has been no reoccurrence for 72 hours. The upstream provider is currently observing ongoing network stability and reporting all call types processing as expected at this time. SpectrumVoIP is observing the same and not seeing an impact to inbound calling for our services. Outbound calling remains routed around the provider and has not been affected. As has occurred the past two days, there is a possibility the attack may resume with greater force in the morning. However, the upstream provider has demonstrated increasing ability to mitigate the attack as the incident has progressed. We continue to monitor the situation and will update the status as appropriate.
- investigating Sep 29, 2021, 01:49 PM UTC
The upstream provider has not yet opened an incident for today however our monitoring indicates the DDoS attack has returned and may impact inbound calling. We are seeing intermittent incidents of audio distortion and audio dropout on inbound calls.
- investigating Sep 29, 2021, 02:09 PM UTC
The upstream provider has acknowledged calls from Verizon Wireless subscribers to their network may experience intermittent failures. Our outbound calling remains routed around provider and is not impacted. We continue to monitor the situation and will update the status as additional details become available.
- monitoring Sep 29, 2021, 04:52 PM UTC
We are updating the status of this incident to Monitoring. The upstream carrier has informed us they are seeing normal call setup from Verizon Wireless callers at this time. We are observing the same and not currently seeing the issue affecting our inbound calling. If you experience this issue please report it our support team so we may identify and chronicle any disruptions. You can reach our technical support team via email at [email protected] We continue to monitor the situation and will update the status as appropriate.
- monitoring Sep 29, 2021, 05:51 PM UTC
The upstream provider continues to acknowledge the DDoS attack is impacting their network and they are actively working to mitigate its effects. At this time we are observing minimal impact to our inbound calling services. Outbound calling remains routed around the provider and is not currently impacted. We continue to monitor the situation and will update the status as appropriate.
- monitoring Sep 30, 2021, 02:17 AM UTC
We can confirm the upstream provider has deployed substantial DDoS mitigation technology and the federal authorities are actively engaged in stopping this attack. The upstream carrier continues to mitigate service disruptions as the attack continues, inbound calling disruptions may potentially occur as this situation continues. At this time we are observing normal traffic patterns and no impact to our inbound calling services. Outbound calling remains routed around the provider and is not currently impacted by this incident. If you experience any service issues please contact our support team via [email protected] so that we may identify and log the issues. We continue to monitor the situation will update the status as appropriate.
- monitoring Sep 30, 2021, 04:05 AM UTC
The upstream provider continues to observe expected network behavior with successful call completions. SpectrumVoIP is observing the same and there is no impact to our inbound calling services at this time. Outbound calling remains routed around the provider and is not currently impacted by this incident. If you experience any service issues please contact our support team via [email protected] so that we may identify and log the issues. We continue to monitor the situation will update the status as appropriate.
- monitoring Sep 30, 2021, 01:42 PM UTC
The upstream carrier is currently observing expected network behavior with successful call completion. We will continue to monitor and provide updates accordingly. If you experience any service issues please contact our support team via [email protected] so that we may identify and log any disruptions.
- monitoring Sep 30, 2021, 07:18 PM UTC
The upstream carrier continues observing expected network behavior with successful call completion. We will continue to monitor and provide updates accordingly. If you experience any service issues please contact our support team via [email protected] so that we may identify and log any disruptions.
- monitoring Sep 30, 2021, 09:47 PM UTC
The upstream carrier continues observing expected network behavior with successful call completion. We will continue to monitor and provide updates accordingly. If you experience any service issues please contact our support team via [email protected] so that we may identify and log any disruptions.
- monitoring Oct 01, 2021, 03:25 PM UTC
The upstream carrier continues observing expected network behavior with successful call completion. We will continue to monitor and provide updates accordingly. If you experience any service issues please contact our support team via [email protected] so that we may identify and log any disruptions.
- resolved Oct 03, 2021, 06:32 PM UTC
This incident has been resolved.