Tinybird incident

A significant fraction of Clickhouse replicas marked as unhealthy

Resolved after 1 hour 20 minutes, affecting GCP europe-west2 (Kafka) and GCP europe-west3 (Events API) and 1 more, on July 28, 2026.

monitoring Jul 28, 2026, 07:38 AM UTC

We have narrowed the root cause to an automatic Ubuntu package upgrade triggering the restart of our database services on many servers, and those being marked unhealthy as a result. Service should be generally restored already.

Started: Jul 28, 2026, 06:44 AM UTC
Resolved: Jul 28, 2026, 08:04 AM UTC
Duration: 1h 20m
Detected by Pingoru: Jul 28, 2026, 06:44 AM UTC

Affected components

GCP europe-west2 (Kafka)GCP europe-west3 (Events API)BI ConnectorAWS eu-central-1 (Sinks)GCP europe-west3 (GCS Connector)GCP europe-west2 (Copy)AWS eu-west-1 (GCS Connector)AWS eu-west-1 (Kafka)AWS us-east-1 (Kafka)AWS us-east-1 (CSV Imports)

Update timeline

investigating Jul 28, 2026, 06:44 AM UTC

Load balancers seem to be down, everything is affected
monitoring Jul 28, 2026, 07:20 AM UTC

We've marked as healthy the database replicas that were incorrectly marked as unhealthy. We're confirming the cause and have a few suspects. We keep monitoring the situation.
monitoring Jul 28, 2026, 07:38 AM UTC

We have narrowed the root cause to an automatic Ubuntu package upgrade triggering the restart of our database services on many servers, and those being marked unhealthy as a result. Service should be generally restored already.
resolved Jul 28, 2026, 08:04 AM UTC

We've confirmed that service has been restored fully.

Your product runs on more than Tinybird

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.