2600Hz experienced a notice incident on October 25, 2021 affecting Telephony Services and Management Portal, lasting 1h 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 25, 2021, 04:02 PM UTC
The 2600Hz Operations team received alerts that some of our servers in the US-Central were unreachable by ping at 8:05 AM PT. We began investigating at that time, and the 2600Hz support team reported escalations in the number of reports of UI, and call issues. Operations found that the servers experiencing issues were the subset hosted with a 3rd party provider. No status messages have been posted for that provider, and our Operations team has opened a ticket with them to identify the cause of the issue.The issue does appear to be intermittent, depending on which path is being taken to, or from the servers hosted with this provider. To attempt to relieve the impact of this issue, we have move the preferred API server which looks to have cleared up the UI issues based on our testing. To attempt to improve call completion in this zone, we have pushed call handling to media servers in our US-East, and US-West datacenters. If you have devices that are having trouble registering to US-central, based on the behavior we have seen this should be due to getting no response from the server. If you are using proper SRV records, then this should cause those devices to attempt to register to the next group of servers in that SRV record. If you are directly registered to US-Central or are not experiencing the failover for SRV, then please update your configs/DNS to prefer a proxy in the US-East, or US-West datacenters. Thank you, 2600Hz
- resolved Oct 25, 2021, 05:13 PM UTC
Our provider for the subset of servers in the US-Central zone has acknowledged that there was an issue on their network in their ORD datacenter where our servers live. They have advised us that their issues have been resolved, and our monitoring and manual tests agree. We have rolled back all changes made to alleviate any issues in the US-Central zone, and have resumed normal operations. This should be all clear. Please reach out to our support team if you are still experiencing any issues after this incident has been marked as resolved. Thank you, 2600Hz