Endless Group incident

Networking outages due to partner error

Major Resolved View vendor source →

Endless Group experienced a major incident on October 26, 2020 affecting Networking, lasting 4h 2m. The incident has been resolved; the full update timeline is below.

Started
Oct 26, 2020, 07:33 PM UTC
Resolved
Oct 26, 2020, 11:35 PM UTC
Duration
4h 2m
Detected by Pingoru
Oct 26, 2020, 07:33 PM UTC

Affected components

Networking

Update timeline

  1. investigating Oct 26, 2020, 07:33 PM UTC

    We are currently experiencing an incoming network traffic issue caused by one of our partners, @RATELIMITEDME . A rogue VM owned by them is currently consuming all incoming traffic to our host. If you receive the message {"success":true,"message":"The File Handler is working normally.","node_info":{"version":2}} when attempting to visit your site, you are affected by this problem. We are currently working with them to resolve this issue and expect to have service restored soon.

  2. monitoring Oct 26, 2020, 08:07 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Oct 26, 2020, 11:35 PM UTC

    This incident has been resolved.