Currents Outage History

This incident has been resolved, our workarounds are keeping everything in sync. We are monitoring the upstream dependency outage.

Type: Incident Duration: 2 hours and 20 minutes Affected Components: API - HTTP REST API, API - Dashboard Browsing May 4, 12:30:00 GMT+0 - Investigating - We are currently investigating this incident. May 4, 14:04:43 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. May 4, 14:49:30 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 1 hour and 17 minutes Affected Components: Data Pipeline Dec 15, 17:23:24 GMT+0 - Monitoring - The queue has now recovered, we will monitor the change as the extra processes we launched ramp back down to normal levels. Dec 15, 18:00:16 GMT+0 - Resolved - This incident has been resolved. Dec 15, 16:42:53 GMT+0 - Identified - Our background tasks queues have gotten behind and as a result run status in the dashboard and event notifications are currently delayed. This was due to a low auto-scaling cap put in place during our last incident (to limit database connections). This scaling issue has now been resolved but we will expect it to take some time for the system to catch up. And result notifications will likely continue to be delayed for the next hour or two.

Type: Incident Duration: 2 hours and 50 minutes Affected Components: Data Pipeline, Data Injestion Dec 13, 02:30:00 GMT+0 - Investigating - We are currently investigating this incident. Dec 13, 05:20:09 GMT+0 - Resolved - This incident has been resolved. The preliminary root cause is a sudden spike in concurrent executions, which affected the DB capacity. After increasing the capacity and processing the backlog the system is back to normal. A more thorough investigation and mitigation are to follow to improve system stability due to surge in concurrent requests.

Type: Incident Duration: 2 hours and 1 minute Affected Components: Data Ingestion Aug 21, 18:23:28 GMT+0 - Identified - Due to a network issue, uploads may be slow. Cypress uploads of logs in all regions are impacted (because our ingest server for these is in the impacted area), and playwright artifact uploads from the us east coast. Aug 21, 19:06:36 GMT+0 - Monitoring - The issue looks to have been resolved. We are continuing to monitor and confirm. Aug 21, 20:24:18 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 2 hours and 43 minutes Affected Components: Data Ingestion Aug 18, 15:33:32 GMT+0 - Investigating - We are currently investigating this incident. Aug 18, 16:46:15 GMT+0 - Monitoring - We are no longer experiencing the issue, but are still investigating the cause. Aug 18, 18:17:00 GMT+0 - Resolved - Incident resolved. We had a partial outage with our object storage provider that resulting in some uploads and requests timing out. We also had issues with our retry logic triggering access limit. We will look into improving this area.

Type: Incident Duration: 13 minutes Affected Components: Data Ingestion May 15, 19:17:38 GMT+0 - Resolved - This incident has been resolved. May 15, 19:05:00 GMT+0 - Identified - A failed deployment caused a 10-minutes outage. During that outage test reporters were not able to upload artifacts. After receiving the alerts we rolled back to failed deployment and the system went back to normal.

Type: Incident Duration: 1 hour and 4 minutes Affected Components: API - Dashboard Browsing, API - HTTP REST API May 9, 00:04:35 GMT+0 - Investigating - We are currently investigating this incident, where our elasticsearch cluster has become unhealthy, and is impacting the processing of results. May 9, 00:59:23 GMT+0 - Monitoring - We implemented a mitigation and are currently monitoring the result. As well as continuing to investigate the cause. May 9, 01:08:08 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 3 hours Affected Components: API - HTTP REST API Mar 21, 17:30:00 GMT+0 - Resolved - The error has been resolved, and the proper configuration restored. An internal issue has been logged to review the incident and look for future deployment improvements. Mar 21, 14:30:00 GMT+0 - Monitoring - Some incompatible settings from a failed deployment were left in place after a rollback, and our rest api endpoint was unreachable for the past 3hrs. Spot instance reset requests, and calls directly to our api endpoints were impacted. We implemented a fix and are currently monitoring the result.