MageMojo incident

Emergency Maintenance on 1 node in USEast (03594a)

Major Resolved View vendor source →

MageMojo experienced a major incident on August 7, 2022 affecting Webscale STRATUS - Northern Virginia, lasting 3h 57m. The incident has been resolved; the full update timeline is below.

Started
Aug 07, 2022, 12:03 PM UTC
Resolved
Aug 07, 2022, 04:00 PM UTC
Duration
3h 57m
Detected by Pingoru
Aug 07, 2022, 12:03 PM UTC

Affected components

Webscale STRATUS - Northern Virginia

Update timeline

  1. identified Aug 07, 2022, 12:03 PM UTC

    AWS has informed us about the detected degradation of the underlying hardware hosting your Amazon EC2 instance associated with this instance. Our Engineers are working on recovery and failover to new hardware.

  2. identified Aug 07, 2022, 01:44 PM UTC

    We are continuing to work on a fix for this issue.

  3. monitoring Aug 07, 2022, 02:42 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. monitoring Aug 07, 2022, 03:39 PM UTC

    We are continuing to monitor for any further issues.

  5. resolved Aug 07, 2022, 04:00 PM UTC

    This incident has been resolved.

  6. postmortem Aug 09, 2022, 08:09 PM UTC

    The Root cause of this incident for a small subset of stores was due to a bad/degraded AWS EC2 instance. Due to this degradation we have activated HA failover mechanism and balanced stores on new hardware nodes from our fleet. Please feel free to contact support if you have any questions.