Currents had 17 outages in the last 2 years — averaging 0.7 incidents per month.
There were 17 Currents outages since May 29, 2024. Each is summarised below — incident details, duration, and resolution information.
Type: Incident Duration: 2 hours and 20 minutes Affected Components: API - HTTP REST API, API - Dashboard Browsing May 4, 12:30:00 GMT+0 - Investigating - We are currently investigating this incident. May 4, 14:04:43 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. May 4, 14:49:30 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 1 hour and 17 minutes Affected Components: Data Pipeline Dec 15, 17:23:24 GMT+0 - Monitoring - The queue has now recovered, we will monitor the change as the extra processes we launched ramp back down to normal levels. Dec 15, 18:00:16 GMT+0 - Resolved - This incident has been resolved. Dec 15, 16:42:53 GMT+0 - Identified - Our background tasks queues have gotten behind and as a result run status in the dashboard and event notifications are currently delayed. This was due to a low auto-scaling cap put in place during our last incident (to limit database connections). This scaling issue has now been resolved but we will expect it to take some time for the system to catch up. And result notifications will likely continue to be delayed for the next hour or two.
Type: Incident Duration: 2 hours and 50 minutes Affected Components: Data Pipeline, Data Injestion Dec 13, 02:30:00 GMT+0 - Investigating - We are currently investigating this incident. Dec 13, 05:20:09 GMT+0 - Resolved - This incident has been resolved. The preliminary root cause is a sudden spike in concurrent executions, which affected the DB capacity. After increasing the capacity and processing the backlog the system is back to normal. A more thorough investigation and mitigation are to follow to improve system stability due to surge in concurrent requests.
Type: Incident Duration: 2 hours and 1 minute Affected Components: Data Injestion Aug 21, 18:23:28 GMT+0 - Identified - Due to a network issue, uploads may be slow. Cypress uploads of logs in all regions are impacted (because our ingest server for these is in the impacted area), and playwright artifact uploads from the us east coast. Aug 21, 19:06:36 GMT+0 - Monitoring - The issue looks to have been resolved. We are continuing to monitor and confirm. Aug 21, 20:24:18 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 2 hours and 43 minutes Affected Components: Data Injestion Aug 18, 15:33:32 GMT+0 - Investigating - We are currently investigating this incident. Aug 18, 16:46:15 GMT+0 - Monitoring - We are no longer experiencing the issue, but are still investigating the cause. Aug 18, 18:17:00 GMT+0 - Resolved - Incident resolved. We had a partial outage with our object storage provider that resulting in some uploads and requests timing out. We also had issues with our retry logic triggering access limit. We will look into improving this area.
Type: Incident Duration: 13 minutes Affected Components: Data Ingestion May 15, 19:17:38 GMT+0 - Resolved - This incident has been resolved. May 15, 19:05:00 GMT+0 - Identified - A failed deployment caused a 10-minutes outage. During that outage test reporters were not able to upload artifacts. After receiving the alerts we rolled back to failed deployment and the system went back to normal.
Type: Incident Duration: 1 hour and 4 minutes Affected Components: API - Dashboard Browsing, API - HTTP REST API May 9, 00:04:35 GMT+0 - Investigating - We are currently investigating this incident, where our elasticsearch cluster has become unhealthy, and is impacting the processing of results. May 9, 00:59:23 GMT+0 - Monitoring - We implemented a mitigation and are currently monitoring the result. As well as continuing to investigate the cause. May 9, 01:08:08 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 3 hours Affected Components: API - HTTP REST API Mar 21, 17:30:00 GMT+0 - Resolved - The error has been resolved, and the proper configuration restored. An internal issue has been logged to review the incident and look for future deployment improvements. Mar 21, 14:30:00 GMT+0 - Monitoring - Some incompatible settings from a failed deployment were left in place after a rollback, and our rest api endpoint was unreachable for the past 3hrs. Spot instance reset requests, and calls directly to our api endpoints were impacted. We implemented a fix and are currently monitoring the result.
Type: Incident Duration: 2 hours and 4 minutes Affected Components: API - HTTP REST API, API - Dashboard Browsing, Data Ingestion Jul 29, 15:21:22 GMT+0 - Investigating - We are currently investigating this incident. We're seeing a spike in processing load, that has resulting in a slowdown in results processing. This is having other cascading effects on how results and run progress is being shown in our system, including causing run timeouts. Jul 29, 17:24:53 GMT+0 - Resolved - Incident is now resolved. We've identified the poorly performing query that was the cause. Jul 29, 16:03:28 GMT+0 - Monitoring - We've increased our task processing capacity, and have restored performance. We continue to investigate and monitor.
Type: Incident Duration: 1 hour and 52 minutes Affected Components: API - Dashboard Browsing May 29, 15:31:13 GMT+0 - Identified - Our in-app support chat is currently experiencing issues. If you need support, please reach out using our support email. Our chat provider's incident status can be followed here: May 29, 16:35:44 GMT+0 - Monitoring - Chat support should now be working, but may be a little slow. If you have trouble with it loading, please try refreshing the page at least once and trying again. Our email support is always available if you are running into troubles. May 29, 17:23:04 GMT+0 - Resolved - This incident has been resolved.