2600Hz experienced a notice incident on June 29, 2021 affecting Telephony Services, lasting 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jun 29, 2021, 07:03 PM UTC
We are receiving customer reports of bad call quality in some areas. Please note that 2600Hz systems are running properly at this time - our new network setup is working well and all servers are functioning normally. We have double-checked and are unable to find any errors or degraded performance at present. That said, phone calls traverse more than just 2600Hz links and carriers when traversing the country from one point to another. While we have redundant uplinks for our own connections we are unable to control links that are beyond our direct path. We are seeing degradation of some upstream carriers whom we do not directly connect to which we suspect is resulting in poor call quality in some areas. We suspect the reports we are receiving related to call quality / call completion are related to other carriers having issues outside of our own (for example, https://downdetector.com/status/8x8/ reports a complete outage for some customers at 8x8, and we're seeing similar complaints for other providers who utilize Lumen/Level3). We are not able to control or repair other providers circuit issues, these are outside of our control. At this time we will need to wait for those upstream providers to resolve their own issues to resolve what is being reported. To be clear, these issues are not related to 2600Hz, our servers, or our direct carriers. We will update this notice when we see the reports of call quality issues decrease so that you are at least aware of the situation upstream.
- monitoring Jun 29, 2021, 07:04 PM UTC
We are monitoring the situation.
- monitoring Jun 29, 2021, 07:05 PM UTC
Sorry, that last update was erroneous! There is no fix implemented as this issue is outside our control. Here is a recap of the issue we are seeing from earlier: "We are receiving customer reports of bad call quality in some areas. Please note that 2600Hz systems are running properly at this time - our new network setup is working well and all servers are functioning normally. We have double-checked and are unable to find any errors or degraded performance at present. That said, phone calls traverse more than just 2600Hz links and carriers when traversing the country from one point to another. While we have redundant uplinks for our own connections we are unable to control links that are beyond our direct path. We are seeing degradation of some upstream carriers whom we do not directly connect to which we suspect is resulting in poor call quality in some areas. We suspect the reports we are receiving related to call quality / call completion are related to other carriers having issues outside of our own (for example, https://downdetector.com/status/8x8/ reports a complete outage for some customers at 8x8, and we're seeing similar complaints for other providers who utilize Lumen/Level3). We are not able to control or repair other providers circuit issues, these are outside of our control. At this time we will need to wait for those upstream providers to resolve their own issues to resolve what is being reported. To be clear, these issues are not related to 2600Hz, our servers, or our direct carriers. We will update this notice when we see the reports of call quality issues decrease so that you are at least aware of the situation upstream."
- resolved Jun 29, 2021, 07:15 PM UTC
The alerts we were receiving regarding external providers connectivity issues seem to have subsided. We will consider this resolved for now.