Livepeer Studio experienced a minor incident on March 29, 2024 affecting Livepeer Streaming API and Livepeer Studio dashboard, lasting 36m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 29, 2024, 04:36 PM UTC
We are currently investigating into to this matter.
- resolved Mar 29, 2024, 05:13 PM UTC
We have identified the root cause and deployed a fix that affected only the United States region. Services have been restored.
- postmortem Apr 03, 2024, 03:27 PM UTC
# **Summary** This is a follow-up report describing the incident that occurred on 03/29/24 [https://status.livepeer.studio/incidents/qtl6n8ksdtv7](https://status.livepeer.studio/incidents/qtl6n8ksdtv7) ## Description Several users reported difficulties with new livestreams failing to ingest and existing livestreams either unable to playback or experiencing buffering issues. Following an investigation, the Livepeer Studio team determined that the root cause of these issues was outdated DNS configurations resulting from decommissioning nodes in certain North American regions for maintenance. This maintenance occurred during the early morning hours in the North American timezone of the same day, with users in specific regions experiencing later in the day. ## Impact * Livestreams: * New livestreams were not able to be ingested in certain North American regions * Viewers: * Playback for certain North American regions was not playing back livestreams while some were playing but experienced buffering issues. Regions: * North American Regions * New York * Miami * Seattle * Los Angeles ## Current status The service has been fully restored [https://status.livepeer.studio/](https://status.livepeer.studio/) ## Timeline * 12:16 PM EST - Reports from multiple users indicating livestreams were having issues with broadcasting and playback * 12:26 PM EST - The team from Livepeer Studio acknowledged this incident and started an investigation * 1:04 PM EST - The investigation from the Livepeer team led to a recent modification to the DNS configuration due to maintenance carried out on the nodes earlier that day * 1:13 PM EST - After monitoring the fix for the incident, the Livepeer Studio team concluded that the issue was resolved # Prevention * Livepeer Studio is now running the DNS configurations hourly to make sure it is consistently up to date * Livepeer Studio is improving alerting systems for catching ingest failures