Grid AI incident

Cluster Upgrades Having Intermittent Errors

Minor Resolved View vendor source →

Grid AI experienced a minor incident on September 1, 2021 affecting Grid Service, lasting 4h 55m. The incident has been resolved; the full update timeline is below.

Started
Sep 01, 2021, 06:27 PM UTC
Resolved
Sep 01, 2021, 11:22 PM UTC
Duration
4h 55m
Detected by Pingoru
Sep 01, 2021, 06:27 PM UTC

Affected components

Grid Service

Update timeline

  1. investigating Sep 01, 2021, 06:27 PM UTC

    Our internal cluster update system is having intermittent errors. This errors are affecting two operations: * Sessions, which are paused after a few minutes of operation * Creating or updating clusters, which can be interrupted mid-way We are actively working on a solution.

  2. identified Sep 01, 2021, 06:27 PM UTC

    We have identified the issue and are working on a solution.

  3. monitoring Sep 01, 2021, 07:48 PM UTC

    We have applied a fix and are monitoring system recovery.

  4. resolved Sep 01, 2021, 11:22 PM UTC

    The incident has been resolved. Sessions are no longer paused.