Binary Lane incident

sydcompute30 outage

Major Resolved View vendor source →

Binary Lane experienced a major incident on August 14, 2018 affecting Compute, lasting 13h 41m. The incident has been resolved; the full update timeline is below.

Started
Aug 14, 2018, 02:37 PM UTC
Resolved
Aug 15, 2018, 04:19 AM UTC
Duration
13h 41m
Detected by Pingoru
Aug 14, 2018, 02:37 PM UTC

Affected components

Compute

Update timeline

  1. investigating Aug 14, 2018, 02:37 PM UTC

    We are currently investigating this issue.

  2. identified Aug 14, 2018, 03:52 PM UTC

    The RAID array on this host node is reporting five of the eight disks as not connected; while our monitoring shows all 8 were available immediately up until the outage started. We are currently performing some physical tests to determine if the fault is with the disks or the controller.

  3. identified Aug 14, 2018, 05:46 PM UTC

    We are unable to access the faulty disks and have concluded that the array contents are lost. We will begin moving affected VPS to a new host node shortly. For customers who have purchased backups, we will restore the latest backup available. For customers who are not using our backup service we will perform a fresh install ready for restoring from offsite backup.

  4. identified Aug 14, 2018, 06:34 PM UTC

    We have begun the process of restoring backup / reinstalling operating system for affected VPS and distributed an email directly to affected customers. We expect all affected VPS to be available again by 6:00AM AEST.

  5. resolved Aug 15, 2018, 04:19 AM UTC

    This incident has been resolved.