Cockroach Labs experienced a major incident on December 22, 2023, lasting 1d. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Dec 22, 2023, 02:01 AM UTC
We are currently seeing some operations on AWS clusters failing, including cluster creation and modification. We've investigated and found a root cause and are working on a mitigation.
- identified Dec 22, 2023, 05:07 PM UTC
We've identified a root cause and are working on a solution. Current impact is limited to vertical scaling of AWS clusters only.
- monitoring Dec 22, 2023, 10:27 PM UTC
Vertical scaling of clusters on AWS may take longer than expected.
- resolved Dec 23, 2023, 02:46 AM UTC
All cluster operations should now succeed. We are monitoring for any additional failures.