api.video experienced a major incident on May 6, 2021 affecting Web services and Video Player and 1 more component, lasting 46d 6h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 05, 2021, 01:51 PM UTC
We are currently investigating this issue.
- identified May 05, 2021, 01:55 PM UTC
The issue has been identified. Issue with the Redis cluster.
- identified May 05, 2021, 04:31 PM UTC
New issue has been identified. Issue with storage access. Working on it.
- identified May 06, 2021, 08:37 AM UTC
A serious issue has been found on our distributed storage configuration, generating long response time on assets. We have to remake our storage cluster. We will mitigate the problem as we migrate to the new cluster.
- identified May 27, 2021, 02:27 PM UTC
New storage cluster is currently in construction, the switch will start soon and will imply a downtime on the API As soon as the switch is done all new videos will work normally Right after, past videos will be migrated step by step until complete resolution
- identified Jun 01, 2021, 09:06 AM UTC
The new storage cluster is ready. The api will be in read-only temporarily during the switch. Check this page regularly, to get updated.
- identified Jun 01, 2021, 03:33 PM UTC
API is now in read-only during the switch
- identified Jun 01, 2021, 04:38 PM UTC
API in read-only still ongoing
- identified Jun 01, 2021, 07:42 PM UTC
API is back to normal. We are still experiencing encoding issues due to the switch. This is under investigation
- identified Jun 01, 2021, 08:31 PM UTC
Encoding problems have been resolved. All services are now linked to the new storage. New videos will be uploaded to the new storage. Videos uploaded before the switch will be retranscoded to the new storage.
- identified Jun 04, 2021, 11:25 AM UTC
Videos uploaded from the 3rd of June are not impacted, and our service is fully operational for them. Videos uploaded prior to June 3 may experience degraded service. We are working on it and preparing to migrate them to our new data storage.
- resolved Jun 21, 2021, 02:58 PM UTC
Majority of videos are now migrated to our new cluster and the platform is operating well. We will continue the migration of remaining videos in the coming days but degraded ones should be a minority. If you experience any issue don't hesitate to reach us though normal support request