Grain experienced a critical incident on December 19, 2023 affecting Grain Recorder, lasting 31d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 19, 2023, 03:33 PM UTC
Zoom has started blocking our Recorder from joining calls. We are in discussion with Zoom and we are attempting to work around the issue.
- investigating Dec 19, 2023, 03:54 PM UTC
We are on a conference call with Zoom engineers trying to work around the issue
- investigating Dec 19, 2023, 03:56 PM UTC
We've rolled out a potential short term mitigation and we're monitoring the solution.
- monitoring Dec 19, 2023, 06:19 PM UTC
Our temporary mitigation continues to appear to be successful. Zoom is still investigating the source of the issue as it appears to be a bug on their end causing our bot to be rejected by ~75% of calls. They have proposed a fix that we will be testing over the next 24 hours.
- resolved Jan 19, 2024, 05:40 PM UTC
The team worked with Zoom over the holidays to deploy a proper fix. No further issues have been observed.