Currents experienced a major incident on July 20, 2023, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jul 20, 2023, 08:42 PM UTC
Type: Incident Duration: 9 hours and 23 minutes Affected Components: API - Dashboard Browsing, API - HTTP REST API, , API → Jul 20, 23:43:33 GMT+0 - Monitoring - The cluster rebalancing is complete, enabling all the ES queries, monitoring the performance and resolution. Jul 21, 06:05:37 GMT+0 - Resolved - This incident has been resolved. Jul 20, 20:42:20 GMT+0 - Investigating - We are dealing with a partial outage with our analytics systems. Insights and performance metrics are temporarily unavailable. Jul 20, 22:55:30 GMT+0 - Identified - The root cause was identified as a failed allocator for the ElasticSearch cluster. A failure in one of the ElasticSearch nodes caused and increased CPU and load for the whole cluster, we are relocating the data to a different node and rebalancing the shards between the new nodes. We turned off some search queries to temporarily reduce the cluster load and reduce the recovery time.