Grain incident

Recorder failing to join meetings in a timely manner

Grain experienced a minor incident on September 8, 2023 affecting Grain Recorder, lasting 6h 16m. The incident has been resolved; the full update timeline is below.

Started: Sep 08, 2023, 03:13 PM UTC
Resolved: Sep 08, 2023, 09:30 PM UTC
Duration: 6h 16m
Detected by Pingoru: Sep 08, 2023, 03:13 PM UTC

Affected components

Grain Recorder

Update timeline

investigating Sep 08, 2023, 03:13 PM UTC

Recorders are not joining meetings at the scheduled start time. The team is investigating the root cause.
monitoring Sep 08, 2023, 06:14 PM UTC

A fix has been deployed. The root cause was found to be misconfigured auto-scaling logic leading to insufficient processing resources for scheduled bots. We are monitoring for further issues.
resolved Sep 08, 2023, 09:30 PM UTC

The incident appears to be fully resolved at this time. Additionally monitoring has been deployed to mitigate similar failures moving forward.