Grain incident

Webhook consumers failing causing app failures

Grain is currently experiencing a major incident affecting Recording Processing and Grain Recorder, which began 20h ago. The vendor's full update timeline is below.

Started
Oct 20, 2026, 05:38 PM UTC
Resolved
Ongoing
Duration
● 20h 12m
Detected by Pingoru
Oct 20, 2026, 05:38 PM UTC

Affected components

Recording ProcessingGrain Recorder

Update timeline

  1. investigating Oct 20, 2026, 05:38 PM UTC

    The team is actively rolling back a failed update for its webhook consumers. Webhooks are currently failing to process leading to UI to fail to update for bots and recording processing. No data loss has occurred.

  2. monitoring Oct 20, 2026, 06:05 PM UTC

    The team has deployed a rollback and webhooks are correctly process. The team is determining if any recording process needs manual intervention.

  3. monitoring Oct 20, 2026, 06:33 PM UTC

    The team is manually reprocessing all dangling completed recordings as they become available. All new recordings are processing correctly.