Buildkite Outage History

## Service Impact On April 8th from 22:10 to 22:15 UTC and from 22:33 to 22:38 UTC all customers would have experienced an increase in latency browsing [buildkite.com](http://buildkite.com) and using Buildkite REST and GraphQL APIs, as well as latency creating triggered and scheduled builds of up to 12 minutes. A portion of customers also experienced increased latency and error rates for the Agent API, with the impact not being evenly distributed, causing some customers to experience a higher impact than others. Customers impacted experienced p99 latency of more than 1 second and error rates of 0.5%. Between 22:10 to 23:03 UTC a majority of customers experienced notification latency of between 5 and 33 minutes, which includes Github commit statuses and webhook delivery, as well as delays in processing incoming webhooks of up to 40 seconds. ## Incident Summary Our engineers noticed an increase in exceptions and shortly afterwards received an alert at 22:16 UTC for high CPU utilisation on a single node in a Redis cluster. Upon investigation we found a rate limiter was hot spotting on a single node within the cluster. Upon removing this limit at 22:33 UTC the impact of high Redis CPU utilization was mitigated, with CPU utilization falling to 5% for the affected node where it had been at 50% prior to the incident. Further investigations also revealed high load on a replica database was also contributing to high latency, which recovered at 22:37 UTC after a third replica was added. The rate limit which was responsible for high load on Redis had caused hot spotting on a single key because it was applied across all organizations. This limit was introduced in response to a previous incident but hadn’t been required since our work to horizontally shard our Pipelines database had distributed the load and enabled higher scalability in early 2024. ## Changes we've made These are the changes we’ve made in response to this incident: * Removed the global rate limit that was contributing to a significant proportion of load * Increased the number of read replicas for our customer information database to 3 and increased the instance size * We’ve reviewed our remaining rate limits and have confirmed that no other rate limits apply globally to all shards.

# Service Impact Between approximately 18:00 UTC and 22:50 UTC February 26, 2026, a subset of customers experienced increased latency when dispatching jobs to agents. Affected customers observed agents sitting idle for several minutes despite having matching jobs waiting in the queue. Job dispatch eventually succeeded, but with significantly elevated latency. The impact was concentrated on specific database shards but affected customers across multiple shards over the course of the incident. # Incident Summary A database maintenance task designed to improve job ordering performance was running across all production database shards. However, this task was itself contributing significant database load, which impacted normal job dispatch and pipeline upload operations. This increased database load caused dispatch operations to queue up, resulting in the observed delays in matching jobs to agents. The issue was compounded by a connection pooling service having several containers running on underperforming infrastructure, which reduced the available database throughput. Contributing factors: * The maintenance task consumed limited database resources, which conflicted with concurrent dispatch operations * The task was running simultaneously across all database shards, amplifying the impact * A connection pooling service had degraded capacity due to infrastructure imbalance # Changes we're making * The maintenance task has been paused and in future will be run during low-traffic periods and on individual shards rather than all shards simultaneously * The connection pooling service has been rebalanced to ensure consistent performance * We are improving our monitoring and dashboards to enable faster identification of lock contention issues during incidents

## Service Impact Between 01:33 UTC and 04:49 UTC on February 12, 2026, a subset of Sidekiq background workers experienced database connection failures, affecting several types of operations over approximately **3 hours and 16 minutes**. For builds where pipeline steps had already been uploaded, the impact was limited to delays. Job dispatching, notifications, and job completion processing were delayed but retried successfully. **All jobs that had already been uploaded were eventually dispatched**, and no jobs were lost. Across all shards, approximately 0.35% of job dispatches and 0.5% of notifications needed retries. However, the impact was not evenly distributed. The underlying issue affected some worker containers and not others, and because containers are assigned to process work for specific database shards, some shards experienced significantly higher error rates while others were completely unaffected. Because each customer's data is assigned to a specific shard, customers on the worst-affected shards would have seen a much higher failure rate than these fleet-wide figures suggest. For builds where a pipeline upload was still in progress, the upload itself could have failed entirely, resulting in a failed build. In these cases, customers would have needed to retry the build manually. ## Incident Summary We were in the process of migrating how environment variables are provided to our Sidekiq background workers running on AWS Fargate. The migration involved two changes: an application-level change to support loading environment configuration from S3, and an infrastructure change to have the Sidekiq services rely on that new loading mechanism. The application change was deployed on February 10 without issue. At this point we verified across all Sidekiq services that the environment is correct when loading from S3. This was done by invoking the S3 environment loading mechanism on a single container per Sidekiq service and comparing the result with existing Sidekiq containers. On February 12 at approximately 01:30 UTC, the infrastructure change was applied, and a subsequent application deployment caused new worker containers to launch using the S3 configuration. Under the new configuration, environment files are downloaded from S3 at boot time and written to a local path. However, some of our Sidekiq services use `sidekiqswarm`, which forks multiple child processes. Gem and application preloading is disabled for these services, meaning each child process independently boots the full Rails application \(including the S3 download step\) after the fork. When multiple child processes booted simultaneously, they each attempted to download the same S3 files and write them to the same local path concurrently. This introduced a race condition: one process could read a file right when another process zeroed it out before writing to it, resulting in a process missing environment variables. This meant that database URL environment variables, which tell the application how to connect to PostgreSQL over the network, were absent in some workers. Without it, the PostgreSQL client fell back to attempting a local Unix socket connection, which does not exist in the containerized environment. Sidekiq workers are responsible for a range of asynchronous operations including job dispatching, notifications, and job completion processing. Workers that could not connect to the database would fail to perform any of these operations and would automatically retry, either succeeding when the retry was performed on an unaffected worker, or after the configuration was rolled back. However, pipeline upload operations that timed out during the incident window could fail outright, causing the associated build to fail. The issue was not immediately detected for two reasons. First, the race condition meant it was effectively random whether a given container was affected. Approximately 0.35% of job dispatches and 0.5% of notifications needed retries across the incident window. Second, the failures did not trigger our existing alerts. When a worker hit a database connection failure, an internal error handling layer caught the error and silently rescheduled the job for retry instead of raising an exception. Our metrics layer saw no raised exception from these attempts, so they were indistinguishable from successful executions. An existing monitor that tracks P99 job dispatch duration saw no increase, because the failures happened almost instantly and looked identical to fast successes. The monitor is effective at detecting slow job processing, but this failure mode \(fast, silent, and rescheduled\) fell outside its coverage. A high volume of database connection errors was present in application logs, which is ultimately how responders identified the root cause during investigation. These factors — a low overall error rate, silent rescheduling, and a duration-based monitor that couldn't detect fast failures — contributed to the delay in detection. **Contributing factors:** * A race condition in the multi-process Sidekiq setup caused concurrent S3 environment file downloads to write to the same local path, allowing processes to read zero-byte files and boot without critical environment variables. * There was no startup assertion to fail fast if critical environment variables such as database configuration were missing or empty after the environment loading step. * Database connection errors in affected workers were caught by an internal error handling layer and silently rescheduled rather than raised as exceptions, preventing them from appearing in error metrics or alerting. * Existing monitoring was designed to detect slow job processing rather than fast failures, so this failure mode fell outside its coverage. **Key timestamps \(UTC\):** | Time | Event | | --- | --- | | 01:33 | Impact began — first database connection errors appeared | | 03:57 | Incident declared; investigation began | | 04:19 | Root cause identified; revert of infrastructure change initiated | | 04:37 | Revert deployed via application deployment | | 04:49 | Impact ended — error rates returned to zero | * **Total duration of impact: 3 hours and 16 minutes.** * **Time to detection: 2 hours and 24 minutes.** * **Time from detection to resolution: 52 minutes.** ## Changes we're making * **Startup validation for critical environment variables \(deployed\):** We’ve added a fail-fast assertion during application boot to verify that database URLs and other critical configuration values are present and non-empty after environment loading completes. * **Atomic S3 file downloads \(deployed\):** We’ve updated the S3 environment file download process to avoid the possibility of processes reading incomplete environment files. * **Improved job processing monitoring \(in progress\):** We are adding error-rate monitoring for job processing to complement our existing latency-based monitors, ensuring that workers which are picking up jobs but failing to complete them are detected regardless of how quickly the failure occurs. * **Incremental rollout for worker configuration changes \(in progress\)**: Future changes to how workers load critical configuration will be deployed to a subset of services first, allowing us to detect problems before they affect the full fleet.