api.video experienced a major incident on February 2, 2021 affecting Video On Demand, lasting 1d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 02, 2021, 02:45 PM UTC
Following yesterday's incident, the delivery infrastructure is still experiencing unstable functioning today. All of our technical teams are working to resolve the problem and stabilize it as quickly as possible.
- investigating Feb 02, 2021, 07:06 PM UTC
We are continuing to investigate this issue.
- resolved Feb 03, 2021, 05:20 PM UTC
The delivery platform has been stable for 24 hours now. Data access conflicts on the central database prior to caching on the local redis created a bottleneck and the stacking of TCP connections to provide the resource access responses. We have adapted the configuration to better manage these competitive accesses. 2ndQuadrant/EDB(the world's largest provider of PostgreSQL products and solutions) did a full Postgresql configuration review this afternoon to provide additional optimizations as well.