Elevated error rate in AWS

Update timeline

1. investigating Jan 31, 2026, 10:03 AM UTC

   Status: Investigating We are investigating a partial outage affecting services in the AWS region us-east-1.

2. identified Jan 31, 2026, 10:55 AM UTC

   Status: Identified Our engineering team has identified the root cause of the service disruption affecting ClickHouse some instances in US-EAST-1. We are currently deploying a fix across the region. ClickHouse instances should begin resuming normal operations within the next 10-30 minutes. We will continue to monitor closely and provide updates as services are restored. Affected components us-east-1 (Partial outage) ClickPipes - us-east-1 (Partial outage)

3. resolved Jan 31, 2026, 11:51 AM UTC

   Status: Resolved This issue is now fixed. Affected components us-east-1 (Operational) ClickPipes - us-east-1 (Operational)

4. investigating Feb 01, 2026, 09:27 AM UTC

   Status: Investigating We are investigating an elevated error rate in AWS us-east-1 and eu-west-2. Affected components us-east-1 (Partial outage)

5. resolved Feb 01, 2026, 09:45 AM UTC

   Status: Resolved All components are operational after the restart of an internal component (Cilium) in the affected regions. The team is working on an RCA. Affected components us-east-1 (Operational)

6. investigating Feb 02, 2026, 09:35 AM UTC

   Status: Investigating We are investigating an elevated error rate in AWS eu-central-1 and ap-southeast-1. Affected components ClickPipes - us-east-1 (Operational) eu-central-1 (Partial outage) eu-west-2 (Operational) us-east-1 (Operational) ap-southeast-1 (Partial outage)

7. monitoring Feb 02, 2026, 10:23 AM UTC

   Status: Monitoring The team mitigated the issue in affected regions and continues to investigate the root cause. Affected components us-east-1 (Operational) ap-southeast-1 (Operational) ClickPipes - us-east-1 (Operational) eu-central-1 (Operational) eu-west-2 (Operational)

8. monitoring Feb 02, 2026, 06:13 PM UTC

   Status: Monitoring Additional configuration changes have been applied to prevent the recurrence of this issue. The team continues to monitor the systems. Affected components eu-central-1 (Operational) eu-west-2 (Operational) us-east-1 (Operational) ap-southeast-1 (Operational) ClickPipes - us-east-1 (Operational)

9. monitoring Feb 02, 2026, 10:47 PM UTC

   Status: Monitoring We believe this is related to a known bug in the version of our networking stack currently in use and are mitigating the issue by resetting this layer across all AWS regions. The team continues to monitor all systems closely. Affected components eu-west-2 (Operational) us-east-1 (Operational) ap-southeast-1 (Operational) ClickPipes - us-east-1 (Operational) eu-central-1 (Operational)

10. monitoring Feb 03, 2026, 03:06 AM UTC

    Status: Monitoring We have completed the reset of the networking stack across all regions and continue to monitor all systems closely. Affected components eu-west-2 (Operational) us-east-1 (Operational) ap-southeast-1 (Operational) ClickPipes - us-east-1 (Operational) eu-central-1 (Operational)

11. resolved Feb 03, 2026, 10:09 AM UTC

    Status: Resolved We have confirmed that all systems are stable and operating normally. The networking configuration has been fully restored across all regions, and we have implemented additional monitoring and safeguards to prevent recurrence. We will continue to monitor systems closely over the coming days as part of our standard post-incident procedures, and share an RCA as soon as possible. Affected components us-east-1 (Operational) ap-southeast-1 (Operational) ClickPipes - us-east-1 (Operational) eu-central-1 (Operational) eu-west-2 (Operational)