Exascale incident

Packet loss on Vodafone Ethernet Bearer - Affecting a number of Leased Lines terminating in Telehouse North London

Major Resolved View vendor source →

Exascale experienced a major incident on February 28, 2022, lasting 22h 22m. The incident has been resolved; the full update timeline is below.

Started
Feb 28, 2022, 11:15 AM UTC
Resolved
Mar 01, 2022, 09:37 AM UTC
Duration
22h 22m
Detected by Pingoru
Feb 28, 2022, 11:15 AM UTC

Update timeline

  1. investigating Feb 28, 2022, 11:15 AM UTC

    We're currently observing a number of Ethernet Leased Line circuits hard down. We're working with Vodafone to resolve. Updates to follow as received or every hour, whichever is sooner.

  2. investigating Feb 28, 2022, 12:38 PM UTC

    We are observing packet loss rarther than a hard down. We are continuing our testing with our partner Vodafone to find the root cause so that we can resolve the issue.

  3. identified Feb 28, 2022, 02:15 PM UTC

    An engineer has been dispached to Telehouse North to Power Cycle a Vodafone Adva CPE as advised by Vodafone NOC.

  4. monitoring Feb 28, 2022, 04:06 PM UTC

    We're now starting to see circuits recover.

  5. resolved Mar 01, 2022, 09:37 AM UTC

    We've now observed 8 hours of stablity, we're closing this issue.