MediaSilo incident

System Outage: Upstream Services Impacted

Major Resolved View vendor source →

MediaSilo experienced a major incident on December 7, 2021 affecting Web Application and iOS App and 1 more component, lasting 1d. The incident has been resolved; the full update timeline is below.

Started
Dec 07, 2021, 04:30 PM UTC
Resolved
Dec 08, 2021, 04:36 PM UTC
Duration
1d
Detected by Pingoru
Dec 07, 2021, 04:30 PM UTC

Affected components

Web ApplicationiOS AppMacOS ApplicationAndroid AppWindows ApplicationFile IngestMediaSilo for tvOSVideo PlaybackImage and Document PreviewsSource File Download

Update timeline

  1. investigating Dec 07, 2021, 04:30 PM UTC

    Engineering has discovered an issue with our upstream provider which is impacting system performance. At this time encoding processes are paused to prevent failures.

  2. identified Dec 07, 2021, 10:22 PM UTC

    Our upstream provider has identified the root cause of the service outage and is continuing to work towards mitigation and resolution. Our services continue to experience intermittent errors as a result of this degradation event. Our engineers have paused encoding of all new uploads during this incident in an effort to prevent encoding errors. These files will be re-processed upon resolution of the incident

  3. monitoring Dec 08, 2021, 12:05 AM UTC

    Our upstream provider has resolved this incident and confirmed that service is operating normally. The engineers are continuing to monitor closely to ensure proper performance. Our system will begin to re-process all previously uploaded assets, a process which we expect will take several hours. New uploads will be impacted and long queue times are expected. In addition, the UI will not show % updates of encode progress.

  4. resolved Dec 08, 2021, 04:36 PM UTC

    All re-processing has been completed for previously uploaded assets. The incident is now resolved.