MageMojo incident

One node in USEast under emergency maintenance

Notice Resolved View vendor source →

MageMojo experienced a notice incident on July 5, 2021 affecting Webscale STRATUS - Northern Virginia, lasting 2m. The incident has been resolved; the full update timeline is below.

Started
Jul 05, 2021, 11:32 AM UTC
Resolved
Jul 05, 2021, 11:34 AM UTC
Duration
2m
Detected by Pingoru
Jul 05, 2021, 11:32 AM UTC

Affected components

Webscale STRATUS - Northern Virginia

Update timeline

  1. monitoring Jul 05, 2021, 11:32 AM UTC

    A fix has been implemented and we are monitoring the results.

  2. resolved Jul 05, 2021, 11:34 AM UTC

    This incident has been resolved.

  3. postmortem Jul 05, 2021, 12:12 PM UTC

    An investigation concluded that a comprehensive kernel bug hit the ZFS filesystem and caused the issue with one of the nodes in our fleet. The problem is identified as similar to the [https://github.com/openzfs/zfs/issues/10642](https://github.com/openzfs/zfs/issues/10642) bug already reported. We have captured kernel stack traces during this event, and a solution for prevention is under investigation.