GearHost experienced a major incident on January 16, 2024 affecting DEN1 and CloudSites, lasting 13h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 16, 2024, 04:55 PM UTC
Our networking team has been alerted to a large-scale network outage currently impacting the Denver network area. This outage is affecting a significant number of people and data centers in the Denver region. Our networking team is actively collaborating with our on-site data center partner to address this issue and ensure the stability of our services in the affected area. Please be assured that this incident is entirely unrelated to the outage experienced in the past few days. We understand the frustration and inconvenience this situation may cause, and we are working diligently to resolve it as quickly as possible. If you have any immediate concerns or require assistance, please do not hesitate to reach out to our support team at [email protected]
- investigating Jan 16, 2024, 05:24 PM UTC
The Denver OPS team has identified a routing issue in the Denver market. IP Cores were impacted, causing a reset of the routing process on some routers. Routing and connectivity has returned but we are still experiencing brief interruptions.
- investigating Jan 16, 2024, 05:36 PM UTC
The Denver OPS team continues to investigate the issue. We have removed bouncing peers which could be contributing to the issue. Additionally, we continue to isolate the issue by power cycling one core router at a time.
- investigating Jan 16, 2024, 05:59 PM UTC
The Denver OPS team is continuing to investigate this issue.
- investigating Jan 16, 2024, 06:44 PM UTC
The Denver OPS team is continuing to investigate this issue.
- identified Jan 16, 2024, 08:49 PM UTC
The Denver OPS team has identified the issue, and a fix has been pushed out. We ask that all customers verify their connectivity and let us know if you're still experiencing any issues.
- identified Jan 16, 2024, 09:36 PM UTC
The Denver OPS team continues to identify wedged tables and process throughout the network. We are continuing to make progress through each access layer device. We ask that customers report back their current status to assist in the effort.
- monitoring Jan 16, 2024, 10:01 PM UTC
Our Denver OPS team has successfully restored network connectivity to all regions. We are actively monitoring the situation and will keep the incident open until we are comfortable with the results. If you encounter any further issues, please open a support ticket at [email protected].
- resolved Jan 17, 2024, 08:38 PM UTC
We have seen no issues since our last update, so we are closing this incident.