Dolby.io incident

LiveStream fails on Singapore Cluster and other clusters may also be affected

Major Resolved View vendor source →

Dolby.io experienced a major incident on August 27, 2023, lasting 9h 54m. The incident has been resolved; the full update timeline is below.

Started
Aug 27, 2023, 09:03 AM UTC
Resolved
Aug 27, 2023, 06:57 PM UTC
Duration
9h 54m
Detected by Pingoru
Aug 27, 2023, 09:03 AM UTC

Update timeline

  1. investigating Aug 27, 2023, 09:03 AM UTC

    LiveStream on Singapore cluster is failing from time to time. As a workaround please change the cluster region to Phoenix/Amsterdam/Bangalore.

  2. monitoring Aug 27, 2023, 09:39 AM UTC

    Identified and made changes on the Singapore node to avoid the issue. Currently, we are monitoring the issue.

  3. investigating Aug 27, 2023, 02:39 PM UTC

    We are encountering additional issues affecting new streams to be started and failing existing streams. The issue can be experienced across different clusters. We are investigating the issue.

  4. investigating Aug 27, 2023, 03:11 PM UTC

    We are experiencing heavy load and continuing to investigate this issue.

  5. monitoring Aug 27, 2023, 05:07 PM UTC

    The system is functional, and we are monitoring it.

  6. resolved Aug 27, 2023, 06:57 PM UTC

    The service is restored.