Is Tinybird down?

Status: Investigating We're investigating an issue when creating or deleting new replicas. This affects cluster replicas and jobs on-demand (populates and copies). Affected components Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage)

Status: Monitoring We have identified the issue and it should be fixed now. We're already seeing some new replicas being provisioned. We'll keep monitoring. Affected components Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage) Copy (Partial outage)

Status: Resolved The incident is now resolved. Affected components Copy (Operational) Copy (Operational) Copy (Operational) Copy (Operational) Copy (Operational) Copy (Operational) Copy (Operational) Copy (Operational) Copy (Operational) Copy (Operational)

Status: Investigating Load balancers seem to be down, everything is affected Affected components Sink (Degraded performance) Sinks (Degraded performance) Copy (Degraded performance) GCS Connector (Degraded performance) Sinks (Degraded performance) GCS Connector (Degraded performance) Login (Degraded performance) Kafka (Degraded performance) Events API (Degraded performance) Sinks (Degraded performance) Query API (Degraded performance) Events API (Degraded performance) Copy (Degraded performance) Login (Degraded performance) DynamoDB Connector (Degraded performance) Sinks (Degraded performance) Events API (Degraded performance) Kafka (Degraded performance) CSV Imports (Degraded performance) Events API (Degraded performance) DynamoDB Connector (Degraded performance) Login (Degraded performance) Query API (Degraded performance) GCS Connector (Degraded performance) Query API (Degraded performance) API Endpoints (Degraded performance) S3 Connector (Degraded performance) GCS Connector (Degraded performance) GCS Connector (Degraded performance) Events API (Degraded performance) Events API (Degraded performance) Events API (Degraded performance) API Endpoints (Degraded performance) S3 Connector (Degraded performance) S3 Connector (Degraded performance) DynamoDB Connector (Degraded performance) S3 Connector (Degraded performance) ClickHouse HTTP Interface (Degraded performance) Copy (Degraded performance) API Endpoints (Degraded performance) Kafka (Degraded performance) S3 Connector (Degraded performance) ClickHouse HTTP Interface (Degraded performance) Login (Degraded performance) Login (Degraded performance) Login (Degraded performance) Query API (Degraded performance) Kafka (Degraded performance) Copy (Degraded performance) Sinks (Degraded performance) BI Connector (Degraded performance) CSV Imports (Degraded performance) GCS Connector (Degraded performance) S3 Connector (Degraded performance) Copy (Degraded performance) Login (Degraded performance) Login (Degraded performance) Query API (Degraded performance) Copy (Degraded performance) ClickHouse HTTP Interface (Degraded performance) ClickHouse HTTP Interface (Degraded performance) ClickHouse HTTP Interface (Degraded performance) ClickHouse HTTP Interface (Degraded performance) S3 Connector (Degraded performance) Sinks (Degraded performance) ClickHouse HTTP Interface (Degraded performance) Query API (Degraded performance) CSV Imports (Degraded performance) CSV Imports (Degraded performance) Sinks (Degraded performance) Kafka (Degraded performance) Query API (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Events API (Degraded performance) GCS Connector (Degraded performance) Copy (Degraded performance) Query API (Degraded performance) API Endpoints (Degraded performance) ClickHouse HTTP Interface (Degraded performance) API Endpoints (Degraded performance) BI Connector (Degraded performance) GCS Connector (Degraded performance) API Endpoints (Degraded performance) API Endpoints (Degraded performance) S3 Connector (Degraded performance) API Endpoints (Degraded performance) Copy (Degraded performance) Kafka (Degraded performance)

Status: Monitoring We've marked as healthy the database replicas that were incorrectly marked as unhealthy. We're confirming the cause and have a few suspects. We keep monitoring the situation. Affected components Copy (Operational) Query API (Operational) Sinks (Operational) Sinks (Operational) Copy (Operational) ClickHouse HTTP Interface (Operational) S3 Connector (Operational) Sinks (Operational) Events API (Operational) GCS Connector (Operational) Events API (Operational) GCS Connector (Operational) API Endpoints (Operational) Copy (Operational) GCS Connector (Operational) Query API (Operational) Login (Operational) Kafka (Operational) ClickHouse HTTP Interface (Operational) BI Connector (Operational) CSV Imports (Operational) Copy (Operational) DynamoDB Connector (Operational) Events API (Operational) Kafka (Operational) Login (Operational) ClickHouse HTTP Interface (Operational) Query API (Operational) Events API (Operational) CSV Imports (Operational) API Endpoints (Operational) Copy (Operational) API Endpoints (Operational) Login (Operational) Login (Operational) API Endpoints (Operational) S3 Connector (Operational) Kafka (Operational) Kafka (Operational) ClickHouse HTTP Interface (Operational) Query API (Operational) Events API (Operational) Events API (Operational) Sink (Operational) Events API (Operational) API Endpoints (Operational) ClickHouse HTTP Interface (Operational) Sinks (Operational) S3 Connector (Operational) Events API (Operational) Login (Operational) Query API (Operational) GCS Connector (Operational) Sinks (Operational) Sinks (Operational) API Endpoints (Operational) Copy (Operational) Sinks (Operational) Query API (Operational) API Endpoints (Operational) S3 Connector (Operational) Query API (Operational) Copy (Operational) API Endpoints (Operational) Kafka (Operational) GCS Connector (Operational) Login (Operational) S3 Connector (Operational) ClickHouse HTTP Interface (Operational) CSV Imports (Operational) Query API (Operational) ClickHouse HTTP Interface (Operational) Kafka (Operational) S3 Connector (Operational) S3 Connector (Operational) BI Connector (Operational) CSV Imports (Operational) GCS Connector (Operational) S3 Connector (Operational) GCS Connector (Operational) Kafka (Operational) Copy (Operational) Kafka (Operational) Login (Operational) ClickHouse HTTP Interface (Operational) Login (Operational) DynamoDB Connector (Operational) DynamoDB Connector (Operational) GCS Connector (Operational)

Status: Monitoring We have narrowed the root cause to an automatic Ubuntu package upgrade triggering the restart of our database services on many servers, and those being marked unhealthy as a result. Service should be generally restored already. Affected components API Endpoints (Operational) Query API (Operational) CSV Imports (Operational) Kafka (Operational) Copy (Operational) Events API (Operational) S3 Connector (Operational) GCS Connector (Operational) Query API (Operational) S3 Connector (Operational) Events API (Operational) GCS Connector (Operational) Login (Operational) Login (Operational) ClickHouse HTTP Interface (Operational) Sinks (Operational) Login (Operational) BI Connector (Operational) DynamoDB Connector (Operational) GCS Connector (Operational) Copy (Operational) Query API (Operational) ClickHouse HTTP Interface (Operational) Kafka (Operational) Query API (Operational) S3 Connector (Operational) API Endpoints (Operational) Kafka (Operational) Sinks (Operational) CSV Imports (Operational) GCS Connector (Operational) Query API (Operational) Sinks (Operational) Sinks (Operational) GCS Connector (Operational) ClickHouse HTTP Interface (Operational) S3 Connector (Operational) S3 Connector (Operational) ClickHouse HTTP Interface (Operational) Copy (Operational) DynamoDB Connector (Operational) API Endpoints (Operational) GCS Connector (Operational) BI Connector (Operational) API Endpoints (Operational) Sinks (Operational) S3 Connector (Operational) Login (Operational) Login (Operational) Sink (Operational) GCS Connector (Operational) Events API (Operational) CSV Imports (Operational) Copy (Operational) Query API (Operational) GCS Connector (Operational) Sinks (Operational) Login (Operational) Copy (Operational) Sinks (Operational) Kafka (Operational) Events API (Operational) API Endpoints (Operational) Query API (Operational) Kafka (Operational) Copy (Operational) Events API (Operational) Login (Operational) Copy (Operational) ClickHouse HTTP Interface (Operational) ClickHouse HTTP Interface (Operational) ClickHouse HTTP Interface (Operational) Events API (Operational) CSV Imports (Operational) S3 Connector (Operational) Kafka (Operational) Events API (Operational) Kafka (Operational) ClickHouse HTTP Interface (Operational) API Endpoints (Operational) Login (Operational) S3 Connector (Operational) Events API (Operational) Query API (Operational) DynamoDB Connector (Operational) Copy (Operational) API Endpoints (Operational) API Endpoints (Operational) Kafka (Operational)

Status: Resolved We've confirmed that service has been restored fully. Affected components CSV Imports (Operational) Copy (Operational) BI Connector (Operational) Kafka (Operational) Copy (Operational) ClickHouse HTTP Interface (Operational) Kafka (Operational) Kafka (Operational) GCS Connector (Operational) S3 Connector (Operational) Copy (Operational) S3 Connector (Operational) S3 Connector (Operational) Login (Operational) CSV Imports (Operational) DynamoDB Connector (Operational) DynamoDB Connector (Operational) Kafka (Operational) GCS Connector (Operational) Sinks (Operational) GCS Connector (Operational) Kafka (Operational) S3 Connector (Operational) GCS Connector (Operational) API Endpoints (Operational) GCS Connector (Operational) S3 Connector (Operational) ClickHouse HTTP Interface (Operational) Query API (Operational) Events API (Operational) CSV Imports (Operational) Kafka (Operational) Query API (Operational) Query API (Operational) ClickHouse HTTP Interface (Operational) Login (Operational) API Endpoints (Operational) Query API (Operational) Login (Operational) API Endpoints (Operational) Events API (Operational) Events API (Operational) Sink (Operational) BI Connector (Operational) Events API (Operational) Copy (Operational) Events API (Operational) Login (Operational) Events API (Operational) Login (Operational) Query API (Operational) Kafka (Operational) Query API (Operational) Query API (Operational) Login (Operational) Login (Operational) Query API (Operational) ClickHouse HTTP Interface (Operational) ClickHouse HTTP Interface (Operational) API Endpoints (Operational) GCS Connector (Operational) Sinks (Operational) GCS Connector (Operational) API Endpoints (Operational) Sinks (Operational) S3 Connector (Operational) API Endpoints (Operational) API Endpoints (Operational) Events API (Operational) DynamoDB Connector (Operational) S3 Connector (Operational) Sinks (Operational) Sinks (Operational) ClickHouse HTTP Interface (Operational) CSV Imports (Operational) S3 Connector (Operational) Sinks (Operational) Login (Operational) Kafka (Operational) ClickHouse HTTP Interface (Operational) ClickHouse HTTP Interface (Operational) GCS Connector (Operational) Copy (Operational) API Endpoints (Operational) Sinks (Operational) Copy (Operational) Events API (Operational) Copy (Operational) Copy (Operational)

Status: Investigating Kafka connector is not consuming messages Affected components Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance)

Status: Investigating A Kafka connector update is affecting message consumption Affected components Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance)

Status: Identified We've identified the issue and are deploying a fix Affected components Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance)

Status: Monitoring The problem is fixed now ingestion is recovered Affected components Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance) Kafka (Degraded performance)

Status: Monitoring GCP europe-west3 still having problems, we're redeploying the region Affected components Kafka (Operational) Kafka (Operational) Kafka (Degraded performance) Kafka (Operational) Kafka (Operational) Kafka (Operational)

Status: Resolved Ingestion is up to date now Affected components Kafka (Operational) Kafka (Operational) Kafka (Operational) Kafka (Operational) Kafka (Operational) Kafka (Operational)

Status: Investigating We're investigating a sudden increase in error rate in the region. Affected components Events API (Partial outage) CSV Imports (Partial outage) Sinks (Partial outage) Login (Partial outage) S3 Connector (Partial outage) GCS Connector (Partial outage) BI Connector (Partial outage) Kafka (Partial outage) Copy (Partial outage) Query API (Partial outage) API Endpoints (Partial outage)

Status: Monitoring We've identified the issue and applied a fix. We're monitoring the situation. Affected components Query API (Partial outage) BI Connector (Partial outage) CSV Imports (Partial outage) Sinks (Partial outage) GCS Connector (Partial outage) API Endpoints (Partial outage) Events API (Partial outage) Kafka (Partial outage) Copy (Partial outage) Login (Partial outage) S3 Connector (Partial outage)

Status: Resolved The situation is now resolved. There shouldn't be more errors. Affected components BI Connector (Operational) API Endpoints (Operational) Events API (Operational) Kafka (Operational) Copy (Operational) CSV Imports (Operational) Sinks (Operational) Login (Operational) S3 Connector (Operational) GCS Connector (Operational) Query API (Operational)

Status: Investigating There’s a performance degradation in the public cluster on AWS us-east. Affected components Query API (Degraded performance) Copy (Degraded performance) Events API (Degraded performance)

Status: Investigating We keep working on identifying the root cause. There's impact in both reads and writes, and in all clusters in the region as the incident is affecting a shared component. We are storing the data that gets to the Events API to retry the inserts later. Affected components Copy (Partial outage) Events API (Partial outage) Query API (Partial outage)

Status: Identified We're still working on stabilising the region. We're struggling with Zookeeper, in charge of coordination between ClickHouse replicas, which is why write are a lot more affected than reads. We've just tweaked a couple parameters that hope will aid the situation. Affected components Events API (Partial outage) Query API (Degraded performance) Copy (Partial outage)

Status: Identified We've already identified the root cause and we're working through the fix. Real-time ingestion is now fully available, we're working on ingesting past data that hasn't been processed yet. Affected components Events API (Partial outage) Query API (Degraded performance) Copy (Partial outage)

Status: Monitoring We've identify the root cause and we are recovering the retries from the HFI ingestion at the moment. The platform should be stable now Affected components Events API (Degraded performance) Query API (Degraded performance) Copy (Degraded performance)

Status: Monitoring We keep ingesting the delayed real time data Affected components Events API (Operational) Query API (Operational) Copy (Degraded performance)

Status: Monitoring We're continuing to work through a backlog of delayed ingestion on our AWS US East 1 shared cluster following yesterday's degradation. Our coordination layer was resized last night and the bulk of the impact has cleared. Recovery is taking longer than expected, so affected workspaces may continue to see ingestion lag while we work through the remaining queue. Affected components Events API (Operational) Query API (Operational) Copy (Degraded performance)

Status: Monitoring The bulk of the earlier retries has now been recovered, and no new ingestion errors are being introduced on the customer side. Recovery is progressing more slowly than we'd like because we are currently being rate-limited by our upstream storage provider, which is limiting how quickly we can drain the remaining retries. No data has been lost, all affected inserts are being safely retried. We'll provide the next update as soon as we see meaningful progress on either front. Affected components Events API (Operational) Query API (Operational) Copy (Degraded performance)

Status: Monitoring We are still getting rate-limited by our upstream storage provider but we are draining retries faster than before because of some optimizations we have made on the system. We are working on accelerating them as much as possible but we still have some retries in the queue. No data has been lost, all affected inserts are being safely retried. Affected components Events API (Operational) Query API (Operational) Copy (Degraded performance)

Status: Resolved All data has been recovered and the system health is back to normal Affected components Events API (Operational) Query API (Operational) Copy (Operational)