Grain experienced a critical incident on March 24, 2026 affecting Grain Recorder, lasting 4h 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 24, 2026, 03:32 PM UTC
An outage with an upstream provider is causing failures for new recordings (both bot and local capture). The team is investigating potential mitigations.
- investigating Mar 24, 2026, 03:40 PM UTC
The team has moved to a new region for the upstream provider and new bot recordings are starting to succeed.
- investigating Mar 24, 2026, 05:02 PM UTC
We are seeing signs that the main region is coming back online. Local captures should start capturing correctly.
- monitoring Mar 24, 2026, 05:46 PM UTC
We've moved back to the main region and we're seeing new bots successfully be launched. Some issues are lingering around double scheduled bots and we are making adjustments to clean those up.
- monitoring Mar 24, 2026, 06:31 PM UTC
The team rolled back to the backup region due to continued higher than normal error rates.
- resolved Mar 24, 2026, 07:58 PM UTC
Our upstream provider has closed their incident. We are no longer observing any instability. Cleanup of captured but unprocessed recordings is underway.