ClickHouse Outage History

Status: Investigating We detected an issue with network configuration in eu-central-1 region which causes failures for streaming Clickpipes in this region. Team is working on remediation Affected components AWS eu-central-1 (Degraded performance)

Status: Investigating Investigating delays to provisioning and autoscaling We are investigating an issue affecting our Kubernetes infrastructure in Azure eastus2 region. Some customers may experience delays with provisioning and autoscaling of ClickHouse clusters. We are working closely with Azure support to restore normal behavior. We’ll provide another update within 30 minutes, or sooner if we have more information. Affected components eastus2 (Degraded performance)

Status: Investigating We are investigating an issue with Salesforce that is affecting our Support tab. Some users may experience errors when accessing or managing support requests. Retrying the action usually resolves the issue. Our team is working to restore normal service, and we will provide an update as soon as more information becomes available. Thank you for your patience. Affected components ClickHouse Cloud API (Degraded performance)

Status: Investigating ClickHouse is investigating degraded Distributed Cache performance affecting multiple ClickHouse Cloud customers. Distributed Cache is a feature currently in Private Preview. We will provide further updates as they become available. Affected components us-east-1 (Degraded performance)

Status: Investigating Due to networking issues in GCP us-central1 zone b (see https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev), ClickHouse Cloud services can be degraded as well. Symptoms can be temporarily unavailable metrics and monitoring dashboards. Creation of new services and ClickPipes pipelines can be temporarily unavailable as well. Existing ClickHouse Services are not affected and are operating normally. Affected components us-central1 (Degraded performance)

Status: Investigating After careful investigation we could not determine any impact of the GCP service degradation: https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev (in zone b) on ClickHouse Cloud functionality. Affected components us-central1 (Operational)

Status: Investigating It may take longer time to provision a new service. Affected components europe-west4 (Degraded performance)

Status: Investigating We are aware about service degradation in Azure eastus2 region, impacting only one Availability Zone. New service provisioning time might be delayed. Customers may see increased latency. Affected components eastus2 (Degraded performance)

Status: Investigating We are still working closely with CSP support to figure out the path forward. The Azure support confirmed there is an issue with some components provided by CSP, investigation ongoing. Affected components eastus2 (Degraded performance)

Status: Investigating Monitoring and billing is degraded in eu-west-1. Specifically: Monitoring dashboards will not contain the latest data Metric-based notification will not be sent Missing metrics in prometheus endpoints Billing metering is not currently updating Autoscaling recommendations are not being applied Affected components eu-west-1 (Degraded performance)

Status: Investigating We are investigating an issue impacting customer services in ap-southeast-1. Some services may experience degraded availability. Affected components ap-southeast-1 (Partial outage) eu-west-1 (Operational)

Status: Investigating As part of our weekly upgrade process, CDC ClickPipes in eu-central-1 became stuck in maintenance state. The team is working on a mitigation. Affected components AWS eu-central-1 (Partial outage)

Status: Investigating We are seeing degraded performance in AWS eu-central-1 starting around 4 pm PT. https://health.aws.amazon.com/health/status

Status: Investigating The backend services are not impacted but the cloud console UI is down. Affected components Authorization (Full outage) ClickHouse Cloud API (Full outage) SQLConsole (Full outage)

Status: Investigating ClickHouse Cloud team is investigating an issue with creating new ClickPipes in GCP. Provisioning of new pipes might be delayed or stuck.

Status: Investigating Due to extremely slow response time for some API calls to Azure infrastructure, new service provisioning may timeout before Cloud Control Plane or OpenAPI request gets response. Affected components eastus2 (Degraded performance) germanywestcentral (Degraded performance) westus3 (Degraded performance)

Status: Investigating Connectivity to services is affected. Affected components eu-west-1 (Degraded performance)

Status: Investigating Impact is on eu-west-1 region. Incident name had a typo in the beginning. Affected components eu-west-1 (Degraded performance)

Status: Investigating Issues logging into ClickHouse Agents reported Affected components ClickHouse Agents (Degraded performance)

Status: Investigating We are currently experiencing capacity constraints in the affected GCP region. As a result, users may encounter issues when creating new clusters or operating existing clusters in this region. Our team is actively working on mitigation to restore normal capacity and minimize customer impact. We will continue to provide updates as more information becomes available. Affected components us-east1 (Degraded performance)