Creation of new instances in AWS is delayed or stuck

Update timeline

1. investigating Apr 08, 2026, 04:09 PM UTC

   Status: Investigating We spotted an issue where creation of new ClickHouse instances is delayed or stuck. Team is investigating the issue and already applying a temporary remediation. Affected components ap-southeast-1 (Degraded performance) ap-southeast-2 (Degraded performance) us-east-2 (Degraded performance) us-west-2 (Degraded performance) ap-northeast-2 (Degraded performance) il-central-1 (Degraded performance) ap-south-1 (Degraded performance) eu-west-2 (Degraded performance) us-east-1 (Degraded performance) eu-central-1 (Degraded performance) eu-west-1 (Degraded performance) ap-northeast-1 (Degraded performance)

2. identified Apr 08, 2026, 04:21 PM UTC

   Status: Identified We identified the issue and applying the remediation. The issue was caused by manual operation performed on the AWS Route53 configuration earlier today. It caused increase in the rate of requests to Route53 API, leading to throttling and retries (including provisioning of new ClickHouse instances). Affected components ap-northeast-1 (Degraded performance) us-west-2 (Degraded performance) ap-south-1 (Degraded performance) eu-west-2 (Degraded performance) ap-northeast-2 (Degraded performance) il-central-1 (Degraded performance) us-east-1 (Degraded performance) ap-southeast-1 (Degraded performance) eu-central-1 (Degraded performance) ap-southeast-2 (Degraded performance) eu-west-1 (Degraded performance) us-east-2 (Degraded performance)

3. monitoring Apr 08, 2026, 04:49 PM UTC

   Status: Monitoring We applied the fix for a subset of regions and see signs of improvements. We are gradually applying the fix to all other regions and monitoring the status. Affected components us-west-2 (Degraded performance) eu-west-2 (Degraded performance) us-east-1 (Degraded performance) ap-southeast-1 (Degraded performance) ap-southeast-2 (Degraded performance) us-east-2 (Degraded performance) ap-northeast-2 (Degraded performance) il-central-1 (Degraded performance) ap-south-1 (Degraded performance) eu-central-1 (Degraded performance) eu-west-1 (Degraded performance) ap-northeast-1 (Degraded performance)

4. monitoring Apr 08, 2026, 07:20 PM UTC

   Status: Monitoring ClickHouse continues to scale external-dns in each region. The following regions are complete and operating normally: us-east-1, af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2, ap-south-1 Additional regions are in progress or will begin shortly, and the ETA to complete the remaining regions is estimated as 3-5 hours: us-east-2, us-west-2, eu-central-1, eu-west-1, eu-west-2, ap-southeast-1, ap-southeast-2, il-central-1 Affected components ap-south-1 (Operational) eu-west-2 (Degraded performance) eu-central-1 (Degraded performance) ap-southeast-2 (Degraded performance) eu-west-1 (Degraded performance) us-east-2 (Degraded performance) ap-northeast-2 (Operational) il-central-1 (Degraded performance) us-east-1 (Operational) ap-southeast-1 (Degraded performance) ap-northeast-1 (Operational) us-west-2 (Degraded performance)

5. resolved Apr 08, 2026, 10:24 PM UTC

   Status: Resolved This issue is resolved. External-DNS has been scaled for all impacted regions. Affected components us-east-1 (Operational) ap-northeast-2 (Operational) il-central-1 (Operational) eu-west-2 (Operational) ap-southeast-1 (Operational) eu-central-1 (Operational) ap-southeast-2 (Operational) eu-west-1 (Operational) us-east-2 (Operational) ap-northeast-1 (Operational) us-west-2 (Operational) ap-south-1 (Operational)