Grid AI incident

Outage in Runs & Experiments

Critical Resolved View vendor source →

Grid AI experienced a critical incident on September 30, 2021 affecting Grid Service, lasting 1h 47m. The incident has been resolved; the full update timeline is below.

Started
Sep 30, 2021, 04:12 PM UTC
Resolved
Sep 30, 2021, 05:59 PM UTC
Duration
1h 47m
Detected by Pingoru
Sep 30, 2021, 04:12 PM UTC

Affected components

Grid Service

Update timeline

  1. identified Sep 30, 2021, 04:12 PM UTC

    We have identified an issue in which Runs & Experiments are unable to run due to a failure in an upcoming feature. All clusters are affected, including Grid Cloud and BYOC. We will be turning the feature off and updating all clusters.

  2. identified Sep 30, 2021, 04:12 PM UTC

    We are continuing to work on a fix for this issue.

  3. monitoring Sep 30, 2021, 05:08 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Sep 30, 2021, 05:59 PM UTC

    We have applied an update to all clusters and service is now fully restored.