Grain incident

Platform Instability

Minor Resolved View vendor source →

Grain experienced a minor incident on March 23, 2026 affecting Grain Desktop App and Recording Processing and 1 more component, lasting 4d 5h. The incident has been resolved; the full update timeline is below.

Started
Mar 23, 2026, 03:08 PM UTC
Resolved
Mar 27, 2026, 09:02 PM UTC
Duration
4d 5h
Detected by Pingoru
Mar 23, 2026, 03:08 PM UTC

Affected components

Grain Desktop AppRecording ProcessingGrain Web App

Update timeline

  1. investigating Mar 23, 2026, 03:08 PM UTC

    The platform is currently experiencing instability, the team is deploying upgraded infrastructure and investigating the root cause.

  2. investigating Mar 23, 2026, 03:53 PM UTC

    The team has rolled out mitigations and we're not longer seeing node failures. However we are still investigating the root cause of the issue.

  3. monitoring Mar 24, 2026, 05:08 PM UTC

    The mitigation has prevented further instability but the team is still investigating the root cause of the issue. The issue caused intermittent and cascading excess memory pressure on the application nodes.

  4. resolved Mar 27, 2026, 09:02 PM UTC

    The team has deployed a fix around large memory spikes trigging out-of-memory failures. Since the latest fix, no further memory spikes have occurred.