Grain incident
Outages with upstream providers causing instability
Grain experienced a major incident on October 20, 2025 affecting Recording Processing and Grain Recorder, lasting 22h 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Oct 20, 2025, 01:09 PM UTC
AWS has deployed mitigations but we are still observing some issues with upstream providers being able to launch instances. We are continuing to monitor the situation.
- monitoring Oct 20, 2025, 01:59 PM UTC
We are continuing to monitor for any further issues.
- monitoring Oct 20, 2025, 02:55 PM UTC
AWS continues to be have failures launching new instances which is preventing scheduled and ad-hoc recorders from booting up. The team is working on spinning up resources in a new region.
- monitoring Oct 20, 2025, 04:07 PM UTC
We've deployed a patch to start scheduling the recording in a new AWS Region, we're seeing successful recording runs.
- monitoring Oct 20, 2025, 06:03 PM UTC
The patches for running the recorder in a new region had some unexpected side effects causing some recorders to continue to fail. Further mitigations have been deployed and processing appears to be moving forward successfully.
- resolved Oct 21, 2025, 11:30 AM UTC
This incident has been resolved.