MageMojo incident

Emergency Maintenance on 1 node in USEast (4b69ad1)

Major Resolved View vendor source →

MageMojo experienced a major incident on August 13, 2022 affecting Webscale STRATUS - Northern Virginia, lasting 2h 52m. The incident has been resolved; the full update timeline is below.

Started
Aug 13, 2022, 11:41 AM UTC
Resolved
Aug 13, 2022, 02:34 PM UTC
Duration
2h 52m
Detected by Pingoru
Aug 13, 2022, 11:41 AM UTC

Affected components

Webscale STRATUS - Northern Virginia

Update timeline

  1. investigating Aug 13, 2022, 11:41 AM UTC

    We are currently investigating this issue.

  2. identified Aug 13, 2022, 12:56 PM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Aug 13, 2022, 01:44 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Aug 13, 2022, 02:34 PM UTC

    This incident has been resolved.

  5. postmortem Aug 15, 2022, 11:55 AM UTC

    The Root cause of this incident for a small subset of stores was due to a bad/degraded AWS EC2 instance. Due to this degradation we have activated HA failover mechanism and balanced stores on new hardware nodes from our fleet. Please feel free to contact support if you have any questions.