An increase in errors and delays from our services

Update timeline

1. resolved Dec 19, 2024, 05:01 PM UTC

   Type: Incident Duration: 3 hours and 1 minute Affected Components: API - HTTP REST API, API - Dashboard Browsing, Data Ingestion Dec 19, 17:01:30 GMT+0 - Investigating - We are currently investigating this incident. Dec 19, 18:01:44 GMT+0 - Investigating - We have reverted the most recent release, and the error rates have gone down. We are still experiencing some slowdowns due to the task backlog that grew during the incident, but the system is mostly recovered. Runs that were created during the outage will be marked as timed out, and you may see errors related to those runs in your local clients if they are still in progress. New runs should not be affected, and should now work as expected. We are still actively investigating the root cause. Dec 19, 18:23:31 GMT+0 - Identified - We are still seeing issues as a result of the backlog, we are scaling up to address the backlog of tasks. We are continuing to investigate the cause in the meantime. Dec 19, 19:05:54 GMT+0 - Monitoring - Our backlog has recovered, and the service is back to normal. The root cause appears to have been changes in our now reverted deployment. We will continue to investigate the cause and provide more details after our investigation. Dec 19, 20:02:32 GMT+0 - Resolved - The system has recovered. We will continue to investigate the root cause and will update the description of the incident with the details when we have them.