BitLaunch incident

Chicago networking outage

Critical Resolved View vendor source →

BitLaunch experienced a critical incident on March 16, 2025 affecting CHI1, lasting 8h 2m. The incident has been resolved; the full update timeline is below.

Started
Mar 16, 2025, 09:39 AM UTC
Resolved
Mar 16, 2025, 05:41 PM UTC
Duration
8h 2m
Detected by Pingoru
Mar 16, 2025, 09:39 AM UTC

Affected components

CHI1

Update timeline

  1. investigating Mar 16, 2025, 09:39 AM UTC

    We are currently investigating this issue.

  2. identified Mar 16, 2025, 09:43 AM UTC

    We have confirmed that this is an issue with our upstream and we have contacted them for a resolution.

  3. identified Mar 16, 2025, 10:21 AM UTC

    We are continuing to push our upstream for a resolution.

  4. identified Mar 16, 2025, 11:16 AM UTC

    We are continuing to push our upstream for a resolution.

  5. identified Mar 16, 2025, 12:00 PM UTC

    Our upstream are reporting an issue with one of their switches which they are working to resolve.

  6. identified Mar 16, 2025, 02:09 PM UTC

    We are continuing to push our upstream for updates. Unfortunately the issue is largely out of our control at this point - we'll provide updates as soon as we have them.

  7. identified Mar 16, 2025, 02:33 PM UTC

    A fix is being deployed and we are working with our upstream to restore the connection.

  8. identified Mar 16, 2025, 04:01 PM UTC

    We are continuing to work on a fix for this issue.

  9. identified Mar 16, 2025, 05:27 PM UTC

    We believe the switch is replaced at this point and we're monitoring the situation.

  10. identified Mar 16, 2025, 05:31 PM UTC

    We are continuing to work on a fix for this issue.

  11. monitoring Mar 16, 2025, 05:31 PM UTC

    A fix has been implemented and we are monitoring the results.

  12. resolved Mar 16, 2025, 05:41 PM UTC

    We have received confirmation from our upstream that this event is resolved: > One of the TOR switches experienced a failure, impacting the virtual chassis and causing downtime. Our network engineers determined the switch is beyond recovery and requires replacement. To restore connectivity, we have moved your uplink to a functioning switch. Connectivity has been successfully restored.