Grain incident

Elevated failure rates with ad-hoc recording sessions

Minor Resolved View vendor source →

Grain experienced a minor incident on November 28, 2023 affecting Grain Recorder, lasting 21h 56m. The incident has been resolved; the full update timeline is below.

Started
Nov 28, 2023, 05:41 PM UTC
Resolved
Nov 29, 2023, 03:38 PM UTC
Duration
21h 56m
Detected by Pingoru
Nov 28, 2023, 05:41 PM UTC

Affected components

Grain Recorder

Update timeline

  1. investigating Nov 28, 2023, 05:41 PM UTC

    The team is observing elevated failure rates with ad-hoc (i.e. unscheduled) recorder sessions due to failure to scale the ad-hoc pool. The team is investigating the root cause while scaling up the ad-hoc pool to handle the load.

  2. monitoring Nov 28, 2023, 06:33 PM UTC

    Error rates have subsided, we are still monitoring for further issues.

  3. resolved Nov 29, 2023, 03:38 PM UTC

    Failure rates have remained at zero during the monitoring period.