US Central (MSP) - Upstream Networking Issues

Update timeline

1. identified Nov 05, 2021, 05:21 AM UTC

   Looks like our US Central (Minneapolis) data center provider is dealing with issues related to their Lumen/Level3 upstream provider connection. We are awaiting further details and hoping the issue can be routed around quickly. This is issue is currently only impacting visitor traffic originating from Lumen/Level3's network, and other upstream peering is all working as expected.

2. identified Nov 05, 2021, 05:53 AM UTC

   We are still awaiting a fix from our data center provider on this issue. Will update this incident as soon as we know more.

3. monitoring Nov 05, 2021, 06:12 AM UTC

   Traffic levels have been returning to normal over the past 5 minutes and things are looking stable. We will keep this incident open while we closely monitor things for a bit and will set this to resolved once we get full confirmation from our data center provider things are fully stable on their end. This incident looks to have initially been caused by a drop on Lumen/Level3, and during the work to route around the issue a primary router at the data center suffered an issue causing a secondary issue leading to a longer than expected impact.

4. resolved Nov 05, 2021, 06:19 AM UTC

   This incident has been resolved.