Arcustech incident

US Central - v.2 Platform - Single Physical Server Issue

Major Resolved View vendor source →

Arcustech experienced a major incident on August 23, 2020, lasting 1h 47m. The incident has been resolved; the full update timeline is below.

Started
Aug 23, 2020, 02:48 AM UTC
Resolved
Aug 23, 2020, 04:36 AM UTC
Duration
1h 47m
Detected by Pingoru
Aug 23, 2020, 02:48 AM UTC

Update timeline

  1. investigating Aug 23, 2020, 02:48 AM UTC

    Staff are investigating errors coming from the storage system on a single physical server. This is only impacting client VPS servers housed on that single physical server. We will update this as soon as we know more.

  2. identified Aug 23, 2020, 03:20 AM UTC

    Currently it simply looks like the zfs file system put itself into a safe/suspended mode after reaching an error threshold, but the drives are checking out fine after a full restart. We are currently bringing affected client systems back online and will be initiating an added off-server data snapshot as we monitor the restarts. We are not seeing any data loss from this event.

  3. resolved Aug 23, 2020, 04:36 AM UTC

    All impacted client systems have been online for sometime, and the storage sub-system is showing no errors or stability issues. We are going to continue to monitor it closely but are going to make this issue as resolved now.