Livepeer Studio experienced a minor incident on May 8, 2024, lasting 54m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating May 08, 2024, 07:41 PM UTC
We are currently investigating the degrading playback views for regions in fin and fin 2.
- resolved May 08, 2024, 08:36 PM UTC
We have identified and fixed the root cause of the problem. A post-incident report will be issued later.
- postmortem May 14, 2024, 03:55 PM UTC
# **Summary** This is a post-mortem describing the incident being investigated on 5/8/24 [https://status.livepeer.studio/incidents/2ljr9lffd5t8](https://status.livepeer.studio/incidents/2ljr9lffd5t8) # Incident ## Description Users reported experiencing either frequent rebuffering or an inability to view playbacks. Upon investigation, the Livepeer Studio team determined that the issue originated from the Finland region. It was caused by servers exceeding their allocated network bandwidth capacity. By redirecting traffic away from this region, the problem was resolved. ## Impact * Livestreams: * N/A * Viewers: * Some viewers were not able to view existing livestreams * Some viewers experienced high rebuffed rates while viewing livestreams * Regions: * Finland ## Current status The service has been fully restored [https://status.livepeer.studio/](https://status.livepeer.studio/) ## Timeline * 2:24 PM EST - An investigation was conducted by the Livepeer Studio team to find the cause of the issue * 3:46PM EST - The Livepeer studio team was still investigating and isolated the area where the root cause could be coming from * 4:16PM EST - The Livepeer Studio team identified the underlying cause of the problem with both servers in Finland having more traffic than it can handle. When traffic was then redirect away from those two servers, there was significant improvements with the rebuffering rate * 4:26 PM EST - After monitoring the fix for the incident, the Livepeer Studio team concluded that the issue was resolved ### Prevention * Enhance alerting for servers with low bandwidth * Improve systems for monitoring performance of servers in each region