Mammoth Cloud incident

Host nodes vps06/08/11/15/16/24 have high packet loss

Minor Resolved View vendor source →

Mammoth Cloud experienced a minor incident on August 23, 2016, lasting 2h 34m. The incident has been resolved; the full update timeline is below.

Started
Aug 23, 2016, 03:18 AM UTC
Resolved
Aug 23, 2016, 05:52 AM UTC
Duration
2h 34m
Detected by Pingoru
Aug 23, 2016, 03:18 AM UTC

Update timeline

  1. investigating Aug 23, 2016, 03:18 AM UTC

    Our monitoring system is showing high packet loss to the three Xen host nodes "vps08", "vps11", and "vps15". We are currently investigating the issue.

  2. investigating Aug 23, 2016, 03:50 AM UTC

    We are currently shutting down services hosted on node vps15 pending a system restart.

  3. investigating Aug 23, 2016, 04:15 AM UTC

    vps15 has been restarted and appears to be functioning correctly. We will now restart vps11

  4. investigating Aug 23, 2016, 04:28 AM UTC

    As the issue is affecting other servers we are currently restarting all affected hosts. Restart of vps11 and vps15 was completed earlier and appears stable.

  5. investigating Aug 23, 2016, 04:34 AM UTC

    Servers located on nodes vps06, vps08, vps16, and vps24 are currently being booted.

  6. monitoring Aug 23, 2016, 04:45 AM UTC

    All affected servers have now been restarted, please contact support if your server is still inaccessible.

  7. resolved Aug 23, 2016, 05:52 AM UTC

    We have identified the root cause of this outage and will provide a post-mortem later this week.