ClickHouse incident
Creation of new instances in AWS is delayed or stuck
ClickHouse experienced a minor incident on April 8, 2026 affecting ap-south-1 and us-east-1 and 1 more component, lasting —. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 08, 2026, 04:09 PM UTC
Status: Investigating We spotted an issue where creation of new ClickHouse instances is delayed or stuck. Team is investigating the issue and already applying a temporary remediation. Affected components ap-southeast-1 (Degraded performance) ap-southeast-2 (Degraded performance) us-east-2 (Degraded performance) us-west-2 (Degraded performance) ap-northeast-2 (Degraded performance) il-central-1 (Degraded performance) ap-south-1 (Degraded performance) eu-west-2 (Degraded performance) us-east-1 (Degraded performance) eu-central-1 (Degraded performance) eu-west-1 (Degraded performance) ap-northeast-1 (Degraded performance)
- identified Apr 08, 2026, 04:21 PM UTC
Status: Identified We identified the issue and applying the remediation. The issue was caused by manual operation performed on the AWS Route53 configuration earlier today. It caused increase in the rate of requests to Route53 API, leading to throttling and retries (including provisioning of new ClickHouse instances). Affected components ap-northeast-1 (Degraded performance) us-west-2 (Degraded performance) ap-south-1 (Degraded performance) eu-west-2 (Degraded performance) ap-northeast-2 (Degraded performance) il-central-1 (Degraded performance) us-east-1 (Degraded performance) ap-southeast-1 (Degraded performance) eu-central-1 (Degraded performance) ap-southeast-2 (Degraded performance) eu-west-1 (Degraded performance) us-east-2 (Degraded performance)
- monitoring Apr 08, 2026, 04:49 PM UTC
Status: Monitoring We applied the fix for a subset of regions and see signs of improvements. We are gradually applying the fix to all other regions and monitoring the status. Affected components us-west-2 (Degraded performance) eu-west-2 (Degraded performance) us-east-1 (Degraded performance) ap-southeast-1 (Degraded performance) ap-southeast-2 (Degraded performance) us-east-2 (Degraded performance) ap-northeast-2 (Degraded performance) il-central-1 (Degraded performance) ap-south-1 (Degraded performance) eu-central-1 (Degraded performance) eu-west-1 (Degraded performance) ap-northeast-1 (Degraded performance)
- monitoring Apr 08, 2026, 07:20 PM UTC
Status: Monitoring ClickHouse continues to scale external-dns in each region. The following regions are complete and operating normally: us-east-1, af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2, ap-south-1 Additional regions are in progress or will begin shortly, and the ETA to complete the remaining regions is estimated as 3-5 hours: us-east-2, us-west-2, eu-central-1, eu-west-1, eu-west-2, ap-southeast-1, ap-southeast-2, il-central-1 Affected components ap-south-1 (Operational) eu-west-2 (Degraded performance) eu-central-1 (Degraded performance) ap-southeast-2 (Degraded performance) eu-west-1 (Degraded performance) us-east-2 (Degraded performance) ap-northeast-2 (Operational) il-central-1 (Degraded performance) us-east-1 (Operational) ap-southeast-1 (Degraded performance) ap-northeast-1 (Operational) us-west-2 (Degraded performance)
- resolved Apr 08, 2026, 10:24 PM UTC
Status: Resolved This issue is resolved. External-DNS has been scaled for all impacted regions. Affected components us-east-1 (Operational) ap-northeast-2 (Operational) il-central-1 (Operational) eu-west-2 (Operational) ap-southeast-1 (Operational) eu-central-1 (Operational) ap-southeast-2 (Operational) eu-west-1 (Operational) us-east-2 (Operational) ap-northeast-1 (Operational) us-west-2 (Operational) ap-south-1 (Operational)