Layer27 experienced a critical incident on August 30, 2020 affecting Pod01 and Cloud Services and 1 more component, lasting 2h 24m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 30, 2020, 01:33 PM UTC
CenturyLink is experiencing a national outage which is affecting connectivity to all data center services. Services that can be migrated away from CenturyLink connections have been moved. However, as CenturyLink is a internet backbone provider, users may still experience connectivity issues as last-mile connections, or routes can still traverse CenturyLink infrastructure. We will continue to work to provide a more complete solution and will update as the overall situation changes.
- identified Aug 30, 2020, 02:09 PM UTC
CenturyLink is experiencing a national outage which is affecting connectivity to data center services. Services that can be migrated away from CenturyLink connections have been moved. However, as CenturyLink is a internet backbone provider, users may still experience connectivity issues as last-mile connections, or routes can still traverse CenturyLink infrastructure. We will continue to work to provide a more complete solution and will update as the overall situation changes.
- resolved Aug 30, 2020, 03:57 PM UTC
CenturyLink has posted that they were able to resolve the connectivity issue and through monitoring over the last half hour no issues have been reported. We will continue to monitor closely and notify if there are any further issues.
- postmortem Aug 30, 2020, 03:58 PM UTC
Between the hours of 6:00AM EDT and 11:30AM EDT on 08/30/2020, users encountered connectivity disruption while accessing data center services. The event was triggered by a nation-wide service outage at CenturyLink. As CenturyLink is a part of the internet backbone, this outage was systemic in nature and affected users locally and outside the region, regardless of what carrier they may have been using. Steps were taken to move services away from CenturyLink routes where possible. This would allow for users on non-CenturyLink internet services to access data center services with minimal interruptions. Not all services were able to be migrated due to connectivity constraints. We will continue to evaluate carrier solutions to ensure the best connectivity options are provided. Outages that encompass large sections of the internet are often difficult to mitigate.