A significant fraction of Clickhouse replicas marked as unhealthy

Update timeline

1. investigating Jul 28, 2026, 06:44 AM UTC

   Status: Investigating Load balancers seem to be down, everything is affected Affected components Sink (Degraded performance) Sinks (Degraded performance) Copy (Degraded performance) GCS Connector (Degraded performance) Sinks (Degraded performance) GCS Connector (Degraded performance) Login (Degraded performance) Kafka (Degraded performance) Events API (Degraded performance) Sinks (Degraded performance) Query API (Degraded performance) Events API (Degraded performance) Copy (Degraded performance) Login (Degraded performance) DynamoDB Connector (Degraded performance) Sinks (Degraded performance) Events API (Degraded performance) Kafka (Degraded performance) CSV Imports (Degraded performance) Events API (Degraded performance) DynamoDB Connector (Degraded performance) Login (Degraded performance) Query API (Degraded performance) GCS Connector (Degraded performance) Query API (Degraded performance) API Endpoints (Degraded performance) S3 Connector (Degraded performance) GCS Connector (Degraded performance) GCS Connector (Degraded performance) Events API (Degraded performance) Events API (Degraded performance) Events API (Degraded performance) API Endpoints (Degraded performance) S3 Connector (Degraded performance) S3 Connector (Degraded performance) DynamoDB Connector (Degraded performance) S3 Connector (Degraded performance) ClickHouse HTTP Interface (Degraded performance) Copy (Degraded performance) API Endpoints (Degraded performance) Kafka (Degraded performance) S3 Connector (Degraded performance) ClickHouse HTTP Interface (Degraded performance) Login (Degraded performance) Login (Degraded performance) Login (Degraded performance) Query API (Degraded performance) Kafka (Degraded performance) Copy (Degraded performance) Sinks (Degraded performance) BI Connector (Degraded performance) CSV Imports (Degraded performance) GCS Connector (Degraded performance) S3 Connector (Degraded performance) Copy (Degraded performance) Login (Degraded performance) Login (Degraded performance) Query API (Degraded performance) Copy (Degraded performance) ClickHouse HTTP Interface (Degraded performance) ClickHouse HTTP Interface (Degraded performance) ClickHouse HTTP Interface (Degraded performance) ClickHouse HTTP Interface (Degraded performance) S3 Connector (Degraded performance) Sinks (Degraded performance) ClickHouse HTTP Interface (Degraded performance) Query API (Degraded performance) CSV Imports (Degraded performance) CSV Imports (Degraded performance) Sinks (Degraded performance) Kafka (Degraded performance) Query API (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Events API (Degraded performance) GCS Connector (Degraded performance) Copy (Degraded performance) Query API (Degraded performance) API Endpoints (Degraded performance) ClickHouse HTTP Interface (Degraded performance) API Endpoints (Degraded performance) BI Connector (Degraded performance) GCS Connector (Degraded performance) API Endpoints (Degraded performance) API Endpoints (Degraded performance) S3 Connector (Degraded performance) API Endpoints (Degraded performance) Copy (Degraded performance) Kafka (Degraded performance)

2. monitoring Jul 28, 2026, 07:20 AM UTC

   Status: Monitoring We've marked as healthy the database replicas that were incorrectly marked as unhealthy. We're confirming the cause and have a few suspects. We keep monitoring the situation. Affected components Copy (Operational) Query API (Operational) Sinks (Operational) Sinks (Operational) Copy (Operational) ClickHouse HTTP Interface (Operational) S3 Connector (Operational) Sinks (Operational) Events API (Operational) GCS Connector (Operational) Events API (Operational) GCS Connector (Operational) API Endpoints (Operational) Copy (Operational) GCS Connector (Operational) Query API (Operational) Login (Operational) Kafka (Operational) ClickHouse HTTP Interface (Operational) BI Connector (Operational) CSV Imports (Operational) Copy (Operational) DynamoDB Connector (Operational) Events API (Operational) Kafka (Operational) Login (Operational) ClickHouse HTTP Interface (Operational) Query API (Operational) Events API (Operational) CSV Imports (Operational) API Endpoints (Operational) Copy (Operational) API Endpoints (Operational) Login (Operational) Login (Operational) API Endpoints (Operational) S3 Connector (Operational) Kafka (Operational) Kafka (Operational) ClickHouse HTTP Interface (Operational) Query API (Operational) Events API (Operational) Events API (Operational) Sink (Operational) Events API (Operational) API Endpoints (Operational) ClickHouse HTTP Interface (Operational) Sinks (Operational) S3 Connector (Operational) Events API (Operational) Login (Operational) Query API (Operational) GCS Connector (Operational) Sinks (Operational) Sinks (Operational) API Endpoints (Operational) Copy (Operational) Sinks (Operational) Query API (Operational) API Endpoints (Operational) S3 Connector (Operational) Query API (Operational) Copy (Operational) API Endpoints (Operational) Kafka (Operational) GCS Connector (Operational) Login (Operational) S3 Connector (Operational) ClickHouse HTTP Interface (Operational) CSV Imports (Operational) Query API (Operational) ClickHouse HTTP Interface (Operational) Kafka (Operational) S3 Connector (Operational) S3 Connector (Operational) BI Connector (Operational) CSV Imports (Operational) GCS Connector (Operational) S3 Connector (Operational) GCS Connector (Operational) Kafka (Operational) Copy (Operational) Kafka (Operational) Login (Operational) ClickHouse HTTP Interface (Operational) Login (Operational) DynamoDB Connector (Operational) DynamoDB Connector (Operational) GCS Connector (Operational)

3. monitoring Jul 28, 2026, 07:38 AM UTC

   Status: Monitoring We have narrowed the root cause to an automatic Ubuntu package upgrade triggering the restart of our database services on many servers, and those being marked unhealthy as a result. Service should be generally restored already. Affected components API Endpoints (Operational) Query API (Operational) CSV Imports (Operational) Kafka (Operational) Copy (Operational) Events API (Operational) S3 Connector (Operational) GCS Connector (Operational) Query API (Operational) S3 Connector (Operational) Events API (Operational) GCS Connector (Operational) Login (Operational) Login (Operational) ClickHouse HTTP Interface (Operational) Sinks (Operational) Login (Operational) BI Connector (Operational) DynamoDB Connector (Operational) GCS Connector (Operational) Copy (Operational) Query API (Operational) ClickHouse HTTP Interface (Operational) Kafka (Operational) Query API (Operational) S3 Connector (Operational) API Endpoints (Operational) Kafka (Operational) Sinks (Operational) CSV Imports (Operational) GCS Connector (Operational) Query API (Operational) Sinks (Operational) Sinks (Operational) GCS Connector (Operational) ClickHouse HTTP Interface (Operational) S3 Connector (Operational) S3 Connector (Operational) ClickHouse HTTP Interface (Operational) Copy (Operational) DynamoDB Connector (Operational) API Endpoints (Operational) GCS Connector (Operational) BI Connector (Operational) API Endpoints (Operational) Sinks (Operational) S3 Connector (Operational) Login (Operational) Login (Operational) Sink (Operational) GCS Connector (Operational) Events API (Operational) CSV Imports (Operational) Copy (Operational) Query API (Operational) GCS Connector (Operational) Sinks (Operational) Login (Operational) Copy (Operational) Sinks (Operational) Kafka (Operational) Events API (Operational) API Endpoints (Operational) Query API (Operational) Kafka (Operational) Copy (Operational) Events API (Operational) Login (Operational) Copy (Operational) ClickHouse HTTP Interface (Operational) ClickHouse HTTP Interface (Operational) ClickHouse HTTP Interface (Operational) Events API (Operational) CSV Imports (Operational) S3 Connector (Operational) Kafka (Operational) Events API (Operational) Kafka (Operational) ClickHouse HTTP Interface (Operational) API Endpoints (Operational) Login (Operational) S3 Connector (Operational) Events API (Operational) Query API (Operational) DynamoDB Connector (Operational) Copy (Operational) API Endpoints (Operational) API Endpoints (Operational) Kafka (Operational)

4. resolved Jul 28, 2026, 08:04 AM UTC

   Status: Resolved We've confirmed that service has been restored fully. Affected components CSV Imports (Operational) Copy (Operational) BI Connector (Operational) Kafka (Operational) Copy (Operational) ClickHouse HTTP Interface (Operational) Kafka (Operational) Kafka (Operational) GCS Connector (Operational) S3 Connector (Operational) Copy (Operational) S3 Connector (Operational) S3 Connector (Operational) Login (Operational) CSV Imports (Operational) DynamoDB Connector (Operational) DynamoDB Connector (Operational) Kafka (Operational) GCS Connector (Operational) Sinks (Operational) GCS Connector (Operational) Kafka (Operational) S3 Connector (Operational) GCS Connector (Operational) API Endpoints (Operational) GCS Connector (Operational) S3 Connector (Operational) ClickHouse HTTP Interface (Operational) Query API (Operational) Events API (Operational) CSV Imports (Operational) Kafka (Operational) Query API (Operational) Query API (Operational) ClickHouse HTTP Interface (Operational) Login (Operational) API Endpoints (Operational) Query API (Operational) Login (Operational) API Endpoints (Operational) Events API (Operational) Events API (Operational) Sink (Operational) BI Connector (Operational) Events API (Operational) Copy (Operational) Events API (Operational) Login (Operational) Events API (Operational) Login (Operational) Query API (Operational) Kafka (Operational) Query API (Operational) Query API (Operational) Login (Operational) Login (Operational) Query API (Operational) ClickHouse HTTP Interface (Operational) ClickHouse HTTP Interface (Operational) API Endpoints (Operational) GCS Connector (Operational) Sinks (Operational) GCS Connector (Operational) API Endpoints (Operational) Sinks (Operational) S3 Connector (Operational) API Endpoints (Operational) API Endpoints (Operational) Events API (Operational) DynamoDB Connector (Operational) S3 Connector (Operational) Sinks (Operational) Sinks (Operational) ClickHouse HTTP Interface (Operational) CSV Imports (Operational) S3 Connector (Operational) Sinks (Operational) Login (Operational) Kafka (Operational) ClickHouse HTTP Interface (Operational) ClickHouse HTTP Interface (Operational) GCS Connector (Operational) Copy (Operational) API Endpoints (Operational) Sinks (Operational) Copy (Operational) Events API (Operational) Copy (Operational) Copy (Operational)