Arcustech incident
US - Central - (Minneapolis) - Upstream Routing Issues
Arcustech experienced a minor incident on August 30, 2020, lasting 4h 42m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Aug 30, 2020, 10:20 AM UTC
Staff are currently investigating reported routing issues in our US Central data center location. We will update this incident report with more information as we have it.
- investigating Aug 30, 2020, 10:40 AM UTC
Our staff are still working with our data center provider in Minneapolis to narrow down the impact issue, and correct it as soon as possible.
- investigating Aug 30, 2020, 10:53 AM UTC
Based on reports from multiple outside sources, it is not just our upstream internet connections having issues. We are working with our upstream providers to gain more details as to the wide spread internet issues, even outside our network at the moment.
- investigating Aug 30, 2020, 11:30 AM UTC
Unfortunately this is not a localized issue, and is rather widespread and reported across the Internet at this point. You can see down detection report spikes for major Internet brands here: https://downdetector.com/ by scrolling down to see current report volumes.
- investigating Aug 30, 2020, 12:55 PM UTC
We are still seeing the same issues that are plaguing large portions of the Internet today. At this point it appears that Level 3 and Century Link are having a serious outage this morning which is wreaking havoc on the internet across the globe. If you are experiencing issues, you are not alone. This is a global event with a lot of service providers, web sites, and services being impacted. Keep in mind the Internet does not work by connecting you directly from your computer to the remote server you are asking data from. Like a complex highway system, your requests and the data sent back to you get routed through many different networks, and providers in both directions. An ongoing thread on Web Hosting Talk about the issues: https://twitter.com/webhostingtalk/status/1300053577684135938 "Centurylink (previously Level3) have "broken large chunks of the Internet today" by continuing to announce routes they don't have. Many major sites reporting issues. https://www.webhostingtalk.com/showthread.php?t=1820262 #CenturyLink #CenturyLinkdown #level3 #outage #networks #webhosting"
- identified Aug 30, 2020, 01:19 PM UTC
As someone explains quite well on the previously posted thread on Web Hosting Talk that we linked to in an earlier update: "Unfortunately with AS3356 announcing routes they no longer have, there's no real "re-routing" around this issue (IE: the status page "re-routing" is somewhat meaningless). We shut our Level3/Centurylink session down ~90 minutes ago, they're still announcing the ghost prefixes. The only fix at this point, is for every provider (worldwide), that uses Level3 to shut them down. The biggest network in the world.. think of the implications here (on every front), this is absolutely nuts!" Our US - Central data center is the only one of our 3 data centers that peers with Level3, so even having removed their route from our announced routes, they are still holding onto those routes impacting any ISP that also peers with Level3 from finding the right route to us over our other upstream providers.
- monitoring Aug 30, 2020, 02:26 PM UTC
We are starting to see strong signs that CenturyLink/Level3 finally is rolling out fixes to their routing meltdown today that impacted huge sections of the Internet, including unfortunately our US Central data center. Our team is still monitoring the situation, but as we stated before, there simply was no direct way for us to fix this major issue that CenturyLink graced the world with this morning. As always if you have any questions please feel free to ask our Support Team, but we feel we covered much of the important parts of today's issues in earlier incident updates.
- resolved Aug 30, 2020, 03:02 PM UTC
This incident has been resolved.