Regional Connectivity - US-East and US-Central

Update timeline

1. investigating Sep 23, 2021, 02:45 PM UTC

   We are investigating reports of a potential issue in our US-East and US-Central data centers. We will provide updates as soon as we have identified the cause and impact of this event.

2. identified Sep 23, 2021, 02:54 PM UTC

   A network issue is impacting connectivity to servers in our US-East and US-Central regions. We are seeing large numbers of phones failing to register to both zones. We will attempt to implement a workaround to get these phones back online while we work the underlying issue.

3. identified Sep 23, 2021, 03:10 PM UTC

   The network connectivity to US-East and US-Central is stabilizing, and we are seeing phone registration counts return to normal levels as a result. Most devices are currently back online. We expect that all remaining devices should be restored within the next few minutes.

4. monitoring Sep 23, 2021, 03:25 PM UTC

   All phones should be up and online at this point. We have moved a significant number of phones to US-West to reduce the load on US-Central and US-East just in case, though the network flaps that we were seeing at the height of the issue have been stable for about half an hour. We will continue to monitor things closely as we wait for the official all-clear from the network team. If your users are experiencing any phone registration or inbound/outbound calling issues, please reach out to our support team to let us know.

5. identified Sep 23, 2021, 05:04 PM UTC

   We are seeing increased registration and API failures again, primarily in US-East. Our network team has determined that there is a DDOS attack in progress against our US-East and US-Central datacenters, for which they have been actively deploying countermeasures to mitigate the impact. We may continue to see some ups and downs as the attack patterns are responded to.

6. monitoring Sep 23, 2021, 06:26 PM UTC

   While we are still receiving occasional alerts for connectivity drops across all 3 zones (largely concentrated on US-East), these have been resolving quickly thanks to our network team's active mitigation efforts, and we have not seen any significant registration drops for the past hour. We've also done some route adjustment of our carrier traffic to avoid the hardest hit components on our system. This attack may continue for a while with resulting sporadic interruptions. Updates will be provided if there are any major changes or increased impact.

7. identified Sep 23, 2021, 08:09 PM UTC

   Our network team has given us the all-clear with respect to the DDOS attack. We are seeing some residual inbound call-handling issues, which seem to affect auto-attendants primarily, as well as some sporadic delays or failures on our user portal, and we are working to identify and resolve those as quickly as possible. If you or your users are experiencing any other issues, please contact our Support team.

8. identified Sep 23, 2021, 10:34 PM UTC

   There are two remaining issues that the team is actively working to resolve. First, some auto-attendants were not properly playing audio when called. We've implemented a fix for this, and early test calls are promising, but it's too early to call the all-clear on it so far. Second, some inbound calls to numbers that route to auto-attendants are being met with a fast busy signal. Our engineers are working to resolve this as quickly as possible.

9. monitoring Sep 23, 2021, 10:49 PM UTC

   We've implemented a fix for the remaining auto-attendant issues, and all test calls are succeeding at this point. We will continue to monitor this closely overnight. Please let our support team know if your sites are still experiencing any issues.

10. resolved Sep 24, 2021, 01:18 PM UTC

    At this point all services have been stable for over 12 hours. We will provide a more detailed RFO once all of the relevant information has been gathered. Please reach out to our support team with any issues or concerns. We apologize for the disruptions in service yesterday, and we wish everyone a safe and relaxing weekend.

11. postmortem Oct 11, 2021, 08:35 PM UTC

    **Event Summary** On September 20, 2021, SimpleVoIP became aware of a massive DDoS \(Distributed Denial of Service\) attack being carried out against Canadian carrier Voip.ms. As they were a fellow VoIP provider, we began preparation for this attack to spread to our IP space. The SimpleVoIP Engineering team immediately began to search for and implement options to bolster our defense against a potential attack of this nature. Unfortunately, this attack did eventually happen, confirming our suspicions. Due to our preparatory work, we were able to react quickly to apply DDoS protections to restore services, but not before several hours of serious global impact to our customers. **Timeline of Events** SimpleVoIP Engineering staff has been working for the past year on upgrading the entire SimpleVoIP network, which included updates to our network security, which included DDoS enhancements. However, after witnessing the attack on Voip.ms, we determined that our current DDoS measures would not withstand an attack of the magnitude that Voip.ms was experiencing. So, we immediately began looking for alternative solutions that we could implement in a short amount of time. SimpleVoIP entered a contract with Cloudflare, a provider boasting 100 Tbps of network capacity and DDoS protection. The SimpleVoIP Infrastructure teams began working with Cloudflare to do everything necessary to announce the SimpleVoIP IP space under their protection. All action items were completed by September 23, 2021, at 3:14 AM PT. SimpleVoIP Infrastructure moved the SimpleVoIP IP space over to Cloudflare and began testing. Some oddities were found, and by 5 AM PT, they were not resolved. As a result, we decided to hold off on the cutover of the rented IP space used by our production network until either required by an attack, or the following evening when we could perform this during a maintenance window, as it would cause an interruption in service. On September 23, 2021, at 7:46 AM PT, SimpleVoIP Engineering received the first indication of an outage, with our early warning systems detecting large numbers of phones deregistering. The initial attack was short, lasting for about 30 minutes. However, it had a longer impact, as it left our US-East cluster in an unusable state. On September 23, 2021, at 8:54 AM PT, the SimpleVoIP Engineering team began the work to move our most-used rented IP space under Cloudflare to mitigate the attack. This process was completed by 9:43 AM PT. Upon completing this work, the attackers moved their attack to a different, less published, rented IP space. As a result, it disrupted the SimpleVoIP network. So, at 10:06 AM PT, SimpleVoIP decided it would be best to stop advertising this network entirely. Upon dropping the advertisements for this last network, SimpleVoIP found that some services were not fully restored to a healthy state. This was due to the existence of a legacy network configuration that hadn't been fully removed as part of our network upgrade project. In this configuration, some services were still trying to talk to one another over the network we had just removed the advertisement for. SimpleVoIP immediately began work to reconfigure the affected devices to prefer a different IP space and recover any affected services. This took several hours, during which phones were up and online, but many call features were impacted \(most notably auto-attendants\). This was resolved by 15:35 PT, at which point all services were fully stabilized for our customers. On September 24, 2021, the SimpleVoIP infrastructure team completed the work necessary to migrate the de-advertised IP space under the Cloudflare network and began advertising that network again. To date, the SimpleVoIP platform has been unaffected by any further attacks. **Root Cause Analysis** Several VoIP providers experienced large-scale DDoS attacks before this attack on the SimpleVoIP network, and several impactful attacks against some larger providers have been perpetrated since then. During this attack, the SimpleVoIP network saw over 3000% increased traffic volume and over 1000% increase in packets per second. Before announcing our owned and rented IP spaces under Cloudflare's network, the SimpleVoIP edge network had been built with redundancy in mind. All our networks are announced over three separate tier 1 ISPs, each announced over the best path route via BGP. This ensures carrier-level redundancy to all our subnets. The SimpleVoIP network also uses a backbone network that is responsible for routing all traffic between each of our data centers to keep all inter-zone federation and replication alive in the event of a loss of connectivity over the public internet. The design of our network and the implementation of this design during network maintenance over the last year have allowed SimpleVoIP to make changes to our public routing very quickly. This allowed us to achieve a faster-than-normal turnaround on connecting to and announcing our subnets via Cloudflare. **Preventative Measures** By adding the DDoS mitigation services from Cloudflare, SimpleVoIP's network is currently well protected from large-scale DDoS attacks. SimpleVoIP Engineering is also beginning to explore other providers as a backup DDoS mitigation service if our primary provider experiences issues and we need to fall back to our previous network advertisements. SimpleVoIP learned several lessons from this event. Some of them were already known and will be addressed by action items that were already planned as part of our major network upgrade project that is still ongoing in all data centers. Other lessons were not previously known, and the SimpleVoIP Engineering team has split those into buckets to be solved in the short, medium, and long term. If you are subscribed to the SimpleVoIP status page, you will continue to see network maintenance notifications that are the result of the SimpleVoIP Engineering team continuing the work we had planned to improve the quality, and security of our network. We apologize for the inconveniences this outage has caused you, your users, and your customers. We strive to provide the best service and security to our customers, and rest assured we are doing our best to precisely do that. We hope this explanation helps you understand the events that transpired. Again, we thank you for your understanding!