M.D.G. IT incident

Shared server am.mdgit.com.au is down for a disk check

Major Resolved View vendor source →

M.D.G. IT experienced a major incident on May 1, 2020 affecting Shared Hosting, Sydney, lasting 11h 31m. The incident has been resolved; the full update timeline is below.

Started
May 01, 2020, 03:03 AM UTC
Resolved
May 01, 2020, 02:34 PM UTC
Duration
11h 31m
Detected by Pingoru
May 01, 2020, 03:03 AM UTC

Affected components

Shared Hosting, Sydney

Update timeline

  1. identified May 01, 2020, 01:38 AM UTC

    The issue has been identified and a fix is being implemented.

  2. identified May 01, 2020, 02:00 AM UTC

    Disk check is still continuing

  3. identified May 01, 2020, 03:03 AM UTC

    This disk check has failed and needs to be rerun

  4. identified May 01, 2020, 04:19 AM UTC

    Pass 1 on second disk check is at 40%, next update at 2:45pm AET

  5. identified May 01, 2020, 04:47 AM UTC

    Pass 1 on disk check two is at 68%, next update at 3:15pm AET

  6. identified May 01, 2020, 05:18 AM UTC

    Pass 1 on disk check two is complete, Pass 2 is now at 85%. Next update at 3:45pm AET

  7. identified May 01, 2020, 05:51 AM UTC

    Pass 2 is now at 86%. Due to the large number of directories on this server the directory structure check can take some time to complete. Next update at 4:30pm AET

  8. identified May 01, 2020, 06:30 AM UTC

    Pass 2 is still at 86% with the check continuing. Next update at 5:15pm AET.

  9. identified May 01, 2020, 07:15 AM UTC

    Pass 2 is at 87% with the check continuing. Next update at 6pm AET.

  10. identified May 01, 2020, 08:00 AM UTC

    Pass 2 is still at 87% with the check continuing. Next update at 7pm AET.

  11. identified May 01, 2020, 09:04 AM UTC

    Pass 2 is still at 87% with the check continuing. Next update at 8pm AET.

  12. identified May 01, 2020, 10:05 AM UTC

    Pass 2 is still at 87% with the check continuing. Next update at 9pm AET.

  13. identified May 01, 2020, 11:01 AM UTC

    Pass 2 is still at 87% with the check continuing. Next update at 10pm AET.

  14. identified May 01, 2020, 12:08 PM UTC

    Pass 2 is still at 87% with the check continuing. Next update at 11pm AET.

  15. identified May 01, 2020, 01:06 PM UTC

    Pass 2 is at 90% with the check continuing. Next update at 3am AET.

  16. resolved May 01, 2020, 02:34 PM UTC

    All services have now been restored after the disk check completed successfully. Emails that have been sent to mailboxes on this server over the duration of the outage will either be reattempted by the senders mail server or will have bounced back to the sender directing them to resend the email at a later time. We will continue to monitor this service for stability.