Grain experienced a major incident on March 19, 2025 affecting Grain Recorder, lasting 70d 23h. The incident has been resolved; the full update timeline is below.
Affected components
Grain Recorder
Update timeline
- investigating Mar 19, 2025, 05:22 PM UTC
Our recording service is experiencing a high rate of failures, causing issues with scheduling and booting up new recorders. Some meetings may not be captured.
- monitoring Mar 19, 2025, 06:13 PM UTC
Mitigation efforts have brought the recording service to a stable place, we are monitoring for further issues and will be evaluating any possibilities for recovery of incorrectly handled recordings.
- monitoring Mar 19, 2025, 07:14 PM UTC
We've seen another uptick in errors after the top of the hour and new mitigations are being explored.
- resolved May 29, 2025, 04:24 PM UTC
This incident has been resolved.