Currents experienced a major incident on July 20, 2023, lasting 9h 23m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Jul 20, 2023, 08:42 PM UTC
We are dealing with a partial outage with our analytics systems. Insights and performance metrics are temporarily unavailable..
- identified Jul 20, 2023, 10:55 PM UTC
The root cause was identified as a failed allocator for the ElasticSearch cluster. A failure in one of the ElasticSearch nodes caused and increased CPU and load for the whole cluster, we are relocating the data to a different node and rebalancing the shards between the new nodes. We turned off some search queries to temporarily reduce the cluster load and reduce the recovery time. .
- monitoring Jul 20, 2023, 11:43 PM UTC
The cluster rebalancing is complete, enabling all the ES queries, monitoring the performance and resolution..
- resolved Jul 21, 2023, 06:05 AM UTC
This incident has been resolved..