2600Hz experienced a notice incident on September 25, 2021 affecting Telephony Services, lasting 1h 14m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Sep 25, 2021, 07:45 PM UTC
Please note that 2600Hz systems are fully functional and operating normally at this time. That said, Bandwidth.com has reported an outage. Bandwidth.com is used by a LOT of customers and powers a fraction of numbers belonging to 2600Hz DID services as well. They also power a lot of customers OFF our system, so while we may be up, calls TO those users who use Bandwidth.com may fail also. Bandwidth.com has reported that they are aware of the outage and working on it. From their status site: "As of 3:31pm EDT Bandwidth has acknowledged the issue: Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged. Sep 25, 15:31 EDT" We're posting this as mostly informational. We don't control the bandwidth.com network. Outages with Bandwidth.com are rare. We suspect they have their best people on it and will resolve it asap. But since they are one of the largest wholesale term/orig providers in the US, this will impact a lot of people, so you may see call failures related to their network issues. As we learn more we'll post here.
- monitoring Sep 25, 2021, 08:38 PM UTC
Hi folks, Since I know it has been a rough week, I wanted to provide some details on this issue. ALL providers who do not own physical wires in the ground have to interconnect with the old telecom network providers in order to provide services, and then transit those old phone lines and PSTN networks via their newer datacenters that are IP-based. There are only a few companies left who really do that - Bandwidth, Level3, Inteliquent, Peerless to name a few. The list is probably less than 10. Most companies big or small (ours, presumably Twilio, RingCentral, etc.) don't publicize this but they outsource this work to the companies listed above. With that in mind, this outage is out of our hands but it will have a big big impact, probably hundreds of thousands of lines if I had to guess (we haven't verified that), impacted. Outages at bandwidth.com are rare because they have become so large and trusted and their network is pretty good. We have already shifted outbound traffic away from bandwidth.com but your customers may be calling other people who use bandwidth.com . Those calls will fail until this is resolved. Additionally, inbound numbers, industry-wide, are single-homed to one provider - so if an inbound number is currently being managed by bandwidth.com, it will be impacted. We can not failover the inbound number routing (as far as I know, nobody can), so we have to wait this out. 2600Hz does not control any of this, we pick providers for things we can't control. We pick providers that we believe are good quality. Bandwidth.com always has been good quality. They're having a bad day today, though, and it's impacting us, and you, too. We trust their time is all over this and we'll be waiting it out, along with you, for resolution. Please note that we can NOT move your numbers to another provider for inbound - that's not a thing that can be done in VoIP or telecom in the US (phone numbers belong to one carrier only and the only way to change that is a port out, which takes days). Hope that helps understand the current situation.
- monitoring Sep 25, 2021, 08:53 PM UTC
We've heard that bandwidth.com has resolved this issue. We are monitoring to verify but wanted to pass that along.
- resolved Sep 25, 2021, 08:59 PM UTC
We've confirmed reports that all services are operational. We'll keep an eye out in case but we're going to resolve this issue for now.