Flow Capture experienced a minor incident on January 28, 2021 affecting Viewing/Playback, lasting 7h 55m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 28, 2021, 05:42 PM UTC
We have discovered an issue with DRM and playback on the AppleTV. We have identified the cause, and are working on a fix.
- identified Jan 28, 2021, 07:44 PM UTC
We have identified there is a window of assets affected from approximately 8pm January 27th to 10am January 28th. We are working to fix the issue for these assets. Any newly uploaded assets are unaffected by this issue and will playback normally on AppleTV. Assets uploaded before this time will also playback normally.
- monitoring Jan 29, 2021, 12:29 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 29, 2021, 01:37 AM UTC
This incident has been resolved.
- postmortem Feb 04, 2021, 07:18 PM UTC
**What happened?** On January 26th at 7:04pm \(PST\), Moxion became aware of a DRM playback issue from several DRM key request error reports. Playback for all DRM video was broken: when trying to play a video, a DRM key error message displayed. We launched an urgent investigation. At 7:46pm \(PST\) our engineering team identified an issue where our DRM provider had a region specific endpoint outage. In response, our team switched to an alternative DRM provider which we had been testing and were planning to migrate to. This allowed us to accelerate the incident resolution for any newly uploaded assets. At 8:20pm \(PST\), our old DRM provider resolved the endpoint outage and playback was restored for historical assets. On January 27th at 11:00pm \(PST\), we identified an issue with Apple TV DRM playback for newly uploaded assets. The recent switch to a new DRM provider had introduced this issue. In response, Moxion switched back to our old DRM provider and reprocessed the affected assets. At 4:30pm \(PST\), we deployed a hotfix for the root cause fixing Apple TV DRM playback on our new DRM provider allowing us to proceed with the migration. **Impact** DRM playback of all assets was broken from 7:00pm till 8:00pm Pacific on January 26th. Then, the DRM playback was restored for all newly uploaded assets but was still failing for all historic assets for another 30min. DRM playback on Apple TV for newly uploaded assets was further broken for 2.5 days and was resolved within hours after the issue had been reported. **Resolution** We enabled a new DRM provider during the outage of the old DRM provider, to resolve the first outage. We deployed a fix for the root cause of the new DRM provider related issue on Apple TV. **Corrective actions** We've improved our test coverage for the Apple TV platform. This will help us to fix potential issues before anything has been rolled out to production. We've planned updates to improve our DRM endpoint monitoring. This will help us to detect any errors with DRM playback as they come up and to act on them before they become a problem. **References** Further information about this incident is available on the status page \([https://status.moxion.io/](https://status.moxion.io/)\)