Livepeer Studio incident
Failure for Livestream Recordings and VOD Uploads
Livepeer Studio experienced a notice incident on June 11, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jun 12, 2024, 07:08 PM UTC
Livestream recordings and VOD uploads could not be processed from 2:30AM UTC to 5:02AM UTC due to an outage from a storage provider.
- postmortem Jun 12, 2024, 07:08 PM UTC
### Incident Overview **Date:** June 11th, 2024 **Time:** 2:30 AM UTC to 5:02 AM UTC **Duration:** 2 hours and 32 minutes **Current status:** The service has been fully restored [https://status.livepeer.studio/](https://status.livepeer.studio/) ### Summary On June 11th, from 2:30 AM UTC to 5:02 AM UTC, our services experienced a significant disruption due to an outage with our storage provider. This outage impacted our ability to record livestreams and upload Video on Demand \(VOD\) content. Unfortunately, any livestream recordings or VOD uploads attempted during this period are not available and we apologize for the inconvenience this incident may have caused. ### Impact * **Livestreams:** * Recordings of livestreams during the outage window are lost and unrecoverable. * **VOD Uploads:** * Any VOD content uploads attempted during the outage were unsuccessful and the data was lost. ### Root Cause The incident was triggered by an unexpected outage with our third-party storage provider, which is responsible for storing all livestream recordings and VOD uploads. The provider's systems experienced a failure that rendered them unable to handle our data storage needs during the specified time. ### Resolution Our team closely monitored the situation and worked with the storage provider to restore service as quickly as possible. Full functionality was restored by 5:02 AM UTC. ### Remediation and Future Prevention To prevent such incidents from impacting our services in the future, we are taking the following actions: 1. **Engagement with Storage Provider:** We are in ongoing conversations with our storage provider to address the reliability of their systems and to understand the root cause of their outage. 2. **Fallback Solutions:** We are actively developing a fallback option to ensure that no assets are lost in the event of a similar outage. This includes exploring alternative storage solutions and implementing redundancy measures.