Buildkite Outage History

Started May 15, 2026, 06:51 AM UTC· resolved May 15, 2026, 07:35 AM UTC

1. monitoring May 15, 2026, 06:51 AM UTC

   Ingestion of Test Engine execution data from an internal queue to a data store stalled, has been resumed, and is working through the backlog. Visibility of test executions from the past hour hours will be delayed for approximately a further one hour. This has been a recurring issue; an architectural change is coming soon to eliminate this failure mode.

2. resolved May 15, 2026, 07:35 AM UTC

   Processing of the backlog is complete.

Read the full incident report →

Started May 13, 2026, 03:14 PM UTC· resolved May 13, 2026, 03:34 PM UTC

1. investigating May 13, 2026, 03:14 PM UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. resolved May 13, 2026, 03:34 PM UTC

   Additional capacity was added to our redis caches. This triggered a failover between UTC 15:10 - 15:14 and there was a spike of errors on the REST and GraphQL APIs. Customers would have seen some errors in the Buildkite UI during this period as well. We have been monitoring the situation since then and things have returned to baseline.

Read the full incident report →

Started May 08, 2026, 01:12 AM UTC· resolved May 09, 2026, 04:34 AM UTC

1. investigating May 08, 2026, 01:12 AM UTC

   We're aware that AWS is reporting availability zone failures in us-east-1. We are monitoring the situation but so far there is no customer impact.

2. investigating May 08, 2026, 01:50 AM UTC

   A small subset of our customers are experiencing delayed notifications. We are actively provisioning additional capacity for these customers. Availability zone automatic failovers are occurring in response to the outage, and this is causing some brief error blips for some customers.

3. investigating May 08, 2026, 02:35 AM UTC

   We are continuing to actively monitor the impacts of this availability zone outage for Buildkite customers. Some transient errors are visible due to availability zone failover events.

4. investigating May 08, 2026, 04:08 AM UTC

   We have provisioned additional capacity in unaffected availability zones so that they are able to support the additional load. Automatic failovers continue to occur where necessary. Some latency and transient errors will be visible to customers.

5. investigating May 08, 2026, 05:10 AM UTC

   We are actively moving resources out of us-east-1c. Similar brief latency and error blips will be visible to customers while these manual failovers occur.

6. investigating May 08, 2026, 05:45 AM UTC

   We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips will unfortunately continue while these manual failovers occur.

7. investigating May 08, 2026, 07:04 AM UTC

   We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur.

8. investigating May 08, 2026, 07:22 AM UTC

   We are continuing to move infrastructure resources out of the affected AWS Availability Zone. Brief latency and error blips may continue while these manual failovers occur. (Apologies if you receive duplicated notifications for this update.)

9. monitoring May 08, 2026, 08:07 AM UTC

   Despite the ongoing AWS incident, our own services are now stable. We are continuing to monitor our services closely, and are ready for further action should the need arise. We are also watching AWS services closely as they recover.

10. resolved May 09, 2026, 04:34 AM UTC

    The upstream AWS incident in us-east-1 has been resolved by AWS, and all Buildkite services are operating normally. No further customer impact is expected. We appreciate your patience during this incident.

11. postmortem May 13, 2026, 05:07 AM UTC

    ## Service impact On 8th May 2026 UTC, between 00:00 and 07:30 UTC, some customers would have seen intermittent errors and latency spikes across many areas of the platform. ## Incident Summary The AWS availability zone incident in `use1-az4` triggered our automatic availability failover mechanisms on database and cache clusters, as per AZ-failure tolerant design. During the failover we saw some isolated request errors that were handled by client-side retries in the agent. Customer workloads were either entirely undisrupted or in the worst case saw elevated latency for a period of up to 5 minutes. Throughout the incident we monitored customer impact and prepared additional resources in a healthy availability zone to manually failover to if the automated systems proved insufficient. These were not necessary, and all our infrastructure self healed.

Read the full incident report →

Started May 07, 2026, 10:45 PM UTC· resolved May 08, 2026, 12:13 AM UTC

1. investigating May 07, 2026, 10:45 PM UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. identified May 07, 2026, 11:13 PM UTC

   We've identified the issue and are working on applying mitigations. At this time we can confirm inbound and outbound webhooks, and notifications are delayed.

3. monitoring May 07, 2026, 11:50 PM UTC

   We have now monitoring the incident. We are seeing most customers have recovered, and some showing signs of recovery.

4. resolved May 08, 2026, 12:13 AM UTC

   All customer workloads have now recovered.

5. postmortem May 21, 2026, 05:53 AM UTC

   ## Service Impact On 2026-05-07 between 22:22 UTC and 00:14 UTC the next day \(~1 hour 52 minutes\), Buildkite customers experienced delays across several Pipelines features: * Inbound webhook processing was delayed. * Outbound webhook delivery and build notifications were delayed. * Job dispatch was delayed for jobs queued during the window. No data was lost. Webhooks and notifications were queued for retry and delivered after the underlying database recovered. Customers with high webhook volumes or time-sensitive build dispatches were most affected. Some customers continued to see lingering latency for a short period after the underlying database recovered, while queued work drained. ## Incident Summary The Pipelines product depends on a shared database that brokers inbound API and webhook requests across all Pipelines shards. At 22:22 UTC, this database's writer instance began saturating under contention on internal locks and could no longer keep up with the volume of work directed at it. Because every Pipelines shard depends on this single shared database, the slowdown affected all customers. Contributing factors: * The shared database's writer was provisioned with very little spare capacity, leaving no headroom to absorb a load spike. * A specific category of background worker, the one that processes incoming webhooks, shares a queue with workers that dispatch builds, so back-pressure on webhook processing cascaded into delayed builds. * A small number of slow external webhook endpoints held worker capacity open inside long-running transactions, amplifying load on the shared database. This is a pattern we have seen before. * Our detection at the database tier was not specific enough to catch the saturation directly; we detected the problem ~12 minutes after impact began, via a downstream signal on Sidekiq queue latency. Mitigation: we provisioned a substantially larger database replica, failed over to it, and re-enabled processing shard-by-shard. We then resized a second replica to match, restoring our ability to fail over again if needed. We initially suspected the incident might be upstream of a separate availability-zone event experienced by our cloud provider. We have since ruled this out: the cloud provider's availability-zone event was declared after this incident was resolved, and the provider has confirmed the affected database instance was not impacted by that event. ## Changes we're making * **Decoupling inbound webhook ingestion from the shared database.** We are currently working through shard isolating inbound webhooks. The risk had already been identified and work had begun to mitigate the risk. In a stroke of bad fortune, it hadn’t been fully rolled out before we incurred the spike in load. Once this has been completed the underlying databases will be one step closer to be broken. * **Operator controls to pause queues during incidents.** We have shipped admin controls to pause individual Sidekiq sets. We’ve also improved our remediation tooling to give our on-callers more control over the ingestion pipeline for managing back-pressure on this specific shared database. * **Database-tier alerting.** We are adding alerts on database write latency and lock-wait activity, so we detect saturation at the source rather than via downstream queue latency. * **Capacity normalisation.** We are normalising autoscaling capacity across shards, so a spike to a single shard's queue can be absorbed by autoscaling rather than cascading.

Read the full incident report →

Started May 07, 2026, 08:48 AM UTC· resolved May 07, 2026, 09:37 AM UTC

1. investigating May 07, 2026, 08:48 AM UTC

   We've spotted that something has gone wrong. We're currently investigating the issue with new builds not starting.

2. investigating May 07, 2026, 09:09 AM UTC

   We've identified issue with job acquiring endpoint. We're rolling back now. We'll provide next update in ~20 minutes.

3. identified May 07, 2026, 09:24 AM UTC

   We're currently seeing recovery at 50% rate. We'll provide next update soon.

4. resolved May 07, 2026, 09:37 AM UTC

   We've reverted a change that caused stale environment variables provided to acquire job used in Hosted Agents, agent-stack-k8s and other agent implementations using acquire job.

5. postmortem May 15, 2026, 01:15 AM UTC

   ## Service Impact On May 7th between 08:14 and 09:25 UTC customers using [hosted agents](https://buildkite.com/docs/agent/buildkite-hosted), the [k8s stack](https://buildkite.com/docs/agent/self-hosted/agent-stack-k8s) and the Buildkite agent [acquire job feature](https://buildkite.com/docs/agent/cli/reference/start#run-a-single-job) experienced failures when starting a job, resulting in an error message `Missing agent. See: buildkite-agent bootstrap --help`. ## Incident Summary As part of our efforts to improve the performance of our platform we shipped a change to how database commits were grouped together. This change inadvertently caused certain job environment variables to be omitted when a job was assigned via the acquire method. These variables are key to our integration with hosted agents as well as customers using our Kubernetes stack and their omission caused any jobs launched via this method to fail. While we had test coverage that ensured these variables were populated, these tests did not exercise the code path used by acquire job effectively enough to indicate this problem before it was deployed. Furthermore, while parts of the change were placed behind a feature flag, the refactor of the code that caused this bug were not. Attempts to restore service by rolling back this change were hindered by the current revision being selected for deploy instead of the previous. The initial rollback was triggered at around 08:40 UTC, but it wasn’t until 09:05 we realised the mistake and began the rollback to the correct revision. The rollback started deploying at 09:16 and at 09:25 service was fully restored. ‌  ## Changes we're making We have added additional rollback gates making it easier to identify when the incorrect revisions has been selected. Our test suite will be expanded to include contract tests for APIs used by Hosted Agent and the Kubernetes stack. Additionally, we are configuring Hosted Agent synthetic tests to automatically page on-call engineers when failures occur, improving our response times.

Read the full incident report →

Started May 06, 2026, 03:57 AM UTC· resolved May 06, 2026, 05:26 AM UTC

1. investigating May 06, 2026, 03:57 AM UTC

   A process writing test results to our Test Engine data store stalled, we've restarted the process and are seeing it catching up. We expect to be fully caught up on the backlog within the next couple of hours.

2. monitoring May 06, 2026, 04:21 AM UTC

   We've identified the issue and the system is currently processing the backlog of test executions

3. resolved May 06, 2026, 05:26 AM UTC

   Processing of test execution ingestion data has successfully caught up.

Read the full incident report →

Started Apr 28, 2026, 06:00 PM UTC· resolved Apr 28, 2026, 07:16 PM UTC

1. investigating Apr 28, 2026, 06:00 PM UTC

   We're observing increased error rates and dispatch latency for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. identified Apr 28, 2026, 06:26 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Apr 28, 2026, 06:45 PM UTC

   We have mitigated the issue causing increased Hosted Agents dispatch latency and intermittent timeout errors for a subset of customers. We identified abnormal workload activity that was placing elevated load on a supporting service, and have now blocked that activity and applied additional protections. Service metrics have returned to normal, and we are continuing to monitor closely.

4. resolved Apr 28, 2026, 07:16 PM UTC

   Previously elevated loads with Hosted Agents dispatch have fully recovered.

Read the full incident report →

Started Apr 22, 2026, 09:19 PM UTC· resolved Apr 22, 2026, 10:59 PM UTC

1. investigating Apr 22, 2026, 09:19 PM UTC

   We are currently investigating reports of authentication failures with the remote MCP server.

2. investigating Apr 22, 2026, 10:07 PM UTC

   We are continuing to investigate errors when authenticating to the remote MCP server.

3. monitoring Apr 22, 2026, 10:44 PM UTC

   We have rolled back a change on the remote MCP server that was contributing to authentication failures.

4. resolved Apr 22, 2026, 10:59 PM UTC

   The issue is resolved.

Read the full incident report →

Started Apr 22, 2026, 02:32 AM UTC· resolved Apr 22, 2026, 05:07 AM UTC

1. monitoring Apr 22, 2026, 02:32 AM UTC

   We noticed a lag in data processing, but our systems are operational and currently working through the backlog. We expect to be fully caught up within the next couple of hours.

2. resolved Apr 22, 2026, 05:07 AM UTC

   The backlog has been cleared and all systems are fully operational. Thank you for your patience.

Read the full incident report →

Started Apr 08, 2026, 10:26 PM UTC· resolved Apr 08, 2026, 11:12 PM UTC

1. investigating Apr 08, 2026, 10:26 PM UTC

   We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

2. monitoring Apr 08, 2026, 10:40 PM UTC

   We have identified and fixed the issue. We are monitoring and seeing signs of improvement.

3. resolved Apr 08, 2026, 11:12 PM UTC

   We experienced an issue that caused a brief increase in errors for the Agent API. This also impacted latency for notifications. All notifications were stored in a queue and processed. Latency is now back to normal.

4. postmortem Apr 21, 2026, 06:04 AM UTC

   ## Service Impact On April 8th from 22:10 to 22:15 UTC and from 22:33 to 22:38 UTC all customers would have experienced an increase in latency browsing [buildkite.com](http://buildkite.com) and using Buildkite REST and GraphQL APIs, as well as latency creating triggered and scheduled builds of up to 12 minutes. A portion of customers also experienced increased latency and error rates for the Agent API, with the impact not being evenly distributed, causing some customers to experience a higher impact than others. Customers impacted experienced p99 latency of more than 1 second and error rates of 0.5%. Between 22:10 to 23:03 UTC a majority of customers experienced notification latency of between 5 and 33 minutes, which includes Github commit statuses and webhook delivery, as well as delays in processing incoming webhooks of up to 40 seconds. ## Incident Summary Our engineers noticed an increase in exceptions and shortly afterwards received an alert at 22:16 UTC for high CPU utilisation on a single node in a Redis cluster. Upon investigation we found a rate limiter was hot spotting on a single node within the cluster. Upon removing this limit at 22:33 UTC the impact of high Redis CPU utilization was mitigated, with CPU utilization falling to 5% for the affected node where it had been at 50% prior to the incident. Further investigations also revealed high load on a replica database was also contributing to high latency, which recovered at 22:37 UTC after a third replica was added. The rate limit which was responsible for high load on Redis had caused hot spotting on a single key because it was applied across all organizations. This limit was introduced in response to a previous incident but hadn’t been required since our work to horizontally shard our Pipelines database had distributed the load and enabled higher scalability in early 2024. ## Changes we've made These are the changes we’ve made in response to this incident: * Removed the global rate limit that was contributing to a significant proportion of load * Increased the number of read replicas for our customer information database to 3 and increased the instance size * We’ve reviewed our remaining rate limits and have confirmed that no other rate limits apply globally to all shards.

Read the full incident report →

Started Mar 31, 2026, 07:51 AM UTC· resolved Mar 31, 2026, 08:34 AM UTC

1. investigating Mar 31, 2026, 07:51 AM UTC

   We have received reports from customers that they are unable to start builds on Hosted Agents. Their builds are immediately cancelled. We are investigating.

2. identified Mar 31, 2026, 08:15 AM UTC

   We have identified the issue and are rolling out a fix.

3. resolved Mar 31, 2026, 08:34 AM UTC

   We have deployed the fix and we have confirmed customer builds are working. If you encounter any further issues please contact support.

Read the full incident report →

Started Mar 27, 2026, 07:02 AM UTC· resolved Mar 27, 2026, 08:53 AM UTC

1. investigating Mar 27, 2026, 07:02 AM UTC

   We're seeing an increase in 504 errors when viewing pipeline builds. We're investigating this now.

2. identified Mar 27, 2026, 07:18 AM UTC

   We've identified a change which we think is the cause of this issue, and we're in the process of reverting it.

3. monitoring Mar 27, 2026, 08:08 AM UTC

   The deploy to revert this change is complete and builds are loading normally. We will continue to monitor for any other issues.

4. resolved Mar 27, 2026, 08:53 AM UTC

   The incident is now resolved. We are no longer seeing errors when viewing pipelines.

Read the full incident report →

Started Mar 25, 2026, 02:26 PM UTC· resolved Mar 25, 2026, 03:50 PM UTC

1. investigating Mar 25, 2026, 02:26 PM UTC

   We are currently investigating this issue.

2. identified Mar 25, 2026, 02:30 PM UTC

   The issue has been identified to be related to Networking and affecting Git Mirror cloning.

3. monitoring Mar 25, 2026, 02:59 PM UTC

   The networking issue has been resolved, dispatch of Hosted Agents has returned to normal levels and no further issues with Git cloning. We are monitoring the situation.

4. resolved Mar 25, 2026, 03:50 PM UTC

   This incident is now resolved. We are no longer seeing further networking issues with Hosted Agents, which affected delays in creating them for Jobs, resolving external traffic and interactions with Cache - affecting Git Mirror Cloning.

Read the full incident report →

Started Mar 11, 2026, 07:50 PM UTC· resolved Mar 11, 2026, 09:14 PM UTC

1. investigating Mar 11, 2026, 07:50 PM UTC

   We are investigating reports of elevated queue times with hosted agents.

2. monitoring Mar 11, 2026, 08:44 PM UTC

   We identified increased demand affecting hosted agent queue times. We have added additional capacity and are seeing recovery of hosted agent queue times.

3. resolved Mar 11, 2026, 09:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 10, 2026, 01:21 AM UTC· resolved Mar 10, 2026, 09:34 AM UTC

1. investigating Mar 10, 2026, 01:21 AM UTC

   We've observed periodic test splitting plan timing out and falling back to non-intelligent splitting. Performance appears to be back to normal as of an hour ago. We are continuing to investigate the root cause and solve the underlying issue.

2. monitoring Mar 10, 2026, 02:25 AM UTC

   We have implemented several mitigation and continue working on fixing the underlying cause. Our team is actively monitoring the situation to ensure the stability. We will provide further updates as we make progress on resolving this issue.

3. resolved Mar 10, 2026, 09:34 AM UTC

   Our mitigations have resolved the elevated latency and likelihood of suboptimal fallback test plans. We have also identified and fixed a blind-spot in our automated alerting, which was previously unable to detect this scenario as an issue. Work continues this week to resolve the underlying performance issue by restructuring how the relevant data is ingested and accessed.

Read the full incident report →

Started Mar 07, 2026, 12:21 AM UTC· resolved Mar 07, 2026, 01:05 AM UTC

1. investigating Mar 07, 2026, 12:21 AM UTC

   We are investigating the elevated latency issue for Test Engine. Processing the backlog of test executions is taking longer than expected, so elevated ingestion latency remains.

2. monitoring Mar 07, 2026, 12:56 AM UTC

   We've identified the issue and the system is currently processing the backlog of test executions

3. resolved Mar 07, 2026, 01:05 AM UTC

   Processing of test execution ingestion data has successfully caught up.

Read the full incident report →

Started Mar 06, 2026, 04:30 AM UTC· resolved Mar 06, 2026, 04:30 AM UTC

1. resolved Mar 06, 2026, 08:54 AM UTC

   Buildkite Hosted Agents experienced degraded start-time performance due to a network partition issue in the Hosted Agents control plane. A small subset of customers may have seen delayed job starts during 04:40-04:50 UTC and 05:06-05:16 UTC. The issue has been resolved and we are monitoring to confirm stability.

Read the full incident report →

Started Mar 05, 2026, 10:14 PM UTC· resolved Mar 06, 2026, 10:23 AM UTC

1. investigating Mar 05, 2026, 10:14 PM UTC

   We're investigating slow artifact uploads. This is isolated to artifacts, dispatch remains unaffected.

2. monitoring Mar 06, 2026, 08:02 AM UTC

   Latency for artifact uploads has remained at normal levels for some time now, and we now have a mitigation in place for a common source of load going forward. We are continuing to monitor.

3. resolved Mar 06, 2026, 10:23 AM UTC

   With artifact upload latency continuing to be stable, we are resolving this incident.

Read the full incident report →

Started Mar 03, 2026, 09:51 PM UTC· resolved Mar 04, 2026, 05:24 AM UTC

1. investigating Mar 03, 2026, 09:51 PM UTC

   We're seeing elevated job dispatch latency and Agent API latency across multiple shards. We're investigating.

2. investigating Mar 03, 2026, 10:41 PM UTC

   We're still experiencing latency issues for agent api and job dispatch. We continue to investigate and identify the root cause.

3. investigating Mar 03, 2026, 11:21 PM UTC

   We continue to experience high latency on some services. We're continuing to identify root causes.

4. monitoring Mar 04, 2026, 12:11 AM UTC

   We've made some changes to address the issue and are seeing signs of recovery. We continue to monitor the situation.

5. monitoring Mar 04, 2026, 01:06 AM UTC

   We've seen a small number of unrelated issues, each affecting a subset of customers. Most impact is resolved, but we are continuing to monitor impact for a small number of remaining customers. We are in touch with those customers directly.

6. monitoring Mar 04, 2026, 03:29 AM UTC

   We continue to observe high latency on isolated infrastructure serving Agent API endpoints for a subset of customers. We are provisioning additional capacity to address this latency, and have informed impacted customers.

7. resolved Mar 04, 2026, 05:24 AM UTC

   We have completed the provisioning of additional capacity mentioned in our last update, and error rates and response times have returned to normal. This incident is now resolved.

Read the full incident report →

Started Feb 26, 2026, 07:10 PM UTC· resolved Feb 27, 2026, 02:30 AM UTC

1. identified Feb 26, 2026, 07:10 PM UTC

   Some customers are experiencing increased latency for jobs being assigned to agents. We have identified the cause and are working on mitigations.

2. monitoring Feb 26, 2026, 07:36 PM UTC

   We're seeing signs of recovery and will continue to monitor.

3. investigating Feb 26, 2026, 11:41 PM UTC

   We're seeing ongoing latency impact across for a subset of customers. Some customers are seeing signs of improvement, but we are continuing to investigate the issue.

4. monitoring Feb 27, 2026, 12:57 AM UTC

   We've seen recovery for the remaining subset of customers. We will continue to monitor.

5. resolved Feb 27, 2026, 02:30 AM UTC

   We have seen a full recovery of service, and have a good understanding of the underlying cause. We will publish a post-incident review next week.

6. postmortem Mar 04, 2026, 02:33 AM UTC

   # Service Impact Between approximately 18:00 UTC and 22:50 UTC February 26, 2026, a subset of customers experienced increased latency when dispatching jobs to agents. Affected customers observed agents sitting idle for several minutes despite having matching jobs waiting in the queue. Job dispatch eventually succeeded, but with significantly elevated latency. The impact was concentrated on specific database shards but affected customers across multiple shards over the course of the incident. # Incident Summary A database maintenance task designed to improve job ordering performance was running across all production database shards. However, this task was itself contributing significant database load, which impacted normal job dispatch and pipeline upload operations. This increased database load caused dispatch operations to queue up, resulting in the observed delays in matching jobs to agents. The issue was compounded by a connection pooling service having several containers running on underperforming infrastructure, which reduced the available database throughput. Contributing factors: * The maintenance task consumed limited database resources, which conflicted with concurrent dispatch operations * The task was running simultaneously across all database shards, amplifying the impact * A connection pooling service had degraded capacity due to infrastructure imbalance # Changes we're making * The maintenance task has been paused and in future will be run during low-traffic periods and on individual shards rather than all shards simultaneously * The connection pooling service has been rebalanced to ensure consistent performance * We are improving our monitoring and dashboards to enable faster identification of lock contention issues during incidents

Read the full incident report →

Started Feb 26, 2026, 12:43 AM UTC· resolved Feb 26, 2026, 02:44 AM UTC

1. investigating Feb 26, 2026, 12:43 AM UTC

   We're observing increased latency on secrets endpoints for a subset of our customers. We're currently investigating and will provide status updates as they become available.

2. monitoring Feb 26, 2026, 12:53 AM UTC

   We've increased the compute available to the secrets service, and have seen response times return to normal levels.

3. resolved Feb 26, 2026, 02:44 AM UTC

   Response times have returned to normal. This incident is now resolved.

Read the full incident report →

Started Feb 12, 2026, 01:33 AM UTC· resolved Feb 12, 2026, 01:33 AM UTC

1. resolved Feb 13, 2026, 02:59 AM UTC

   Between 2026-02-12 01:33 and 04:49 UTC, an elevated error rate in job dispatch was observed. This affected a subset of jobs. Affected jobs were unable to be dispatched during the disruption window, but were eventually dispatched due to automatic retries. A subset of pipeline step uploads were also affected; the agent would check for successful step upload over a 5-minute period before timing out. This occurred due to a bug in a configuration change that resulted in some Sidekiq instances not receiving all required database connection configuration. The configuration change was subsequently reverted, with further investigation underway.

2. postmortem Feb 19, 2026, 02:03 AM UTC

   ## Service Impact Between 01:33 UTC and 04:49 UTC on February 12, 2026, a subset of Sidekiq background workers experienced database connection failures, affecting several types of operations over approximately **3 hours and 16 minutes**. For builds where pipeline steps had already been uploaded, the impact was limited to delays. Job dispatching, notifications, and job completion processing were delayed but retried successfully. **All jobs that had already been uploaded were eventually dispatched**, and no jobs were lost. Across all shards, approximately 0.35% of job dispatches and 0.5% of notifications needed retries. However, the impact was not evenly distributed. The underlying issue affected some worker containers and not others, and because containers are assigned to process work for specific database shards, some shards experienced significantly higher error rates while others were completely unaffected. Because each customer's data is assigned to a specific shard, customers on the worst-affected shards would have seen a much higher failure rate than these fleet-wide figures suggest. For builds where a pipeline upload was still in progress, the upload itself could have failed entirely, resulting in a failed build. In these cases, customers would have needed to retry the build manually. ## Incident Summary We were in the process of migrating how environment variables are provided to our Sidekiq background workers running on AWS Fargate. The migration involved two changes: an application-level change to support loading environment configuration from S3, and an infrastructure change to have the Sidekiq services rely on that new loading mechanism. The application change was deployed on February 10 without issue. At this point we verified across all Sidekiq services that the environment is correct when loading from S3. This was done by invoking the S3 environment loading mechanism on a single container per Sidekiq service and comparing the result with existing Sidekiq containers. On February 12 at approximately 01:30 UTC, the infrastructure change was applied, and a subsequent application deployment caused new worker containers to launch using the S3 configuration. Under the new configuration, environment files are downloaded from S3 at boot time and written to a local path. However, some of our Sidekiq services use `sidekiqswarm`, which forks multiple child processes. Gem and application preloading is disabled for these services, meaning each child process independently boots the full Rails application \(including the S3 download step\) after the fork. When multiple child processes booted simultaneously, they each attempted to download the same S3 files and write them to the same local path concurrently. This introduced a race condition: one process could read a file right when another process zeroed it out before writing to it, resulting in a process missing environment variables. This meant that database URL environment variables, which tell the application how to connect to PostgreSQL over the network, were absent in some workers. Without it, the PostgreSQL client fell back to attempting a local Unix socket connection, which does not exist in the containerized environment. Sidekiq workers are responsible for a range of asynchronous operations including job dispatching, notifications, and job completion processing. Workers that could not connect to the database would fail to perform any of these operations and would automatically retry, either succeeding when the retry was performed on an unaffected worker, or after the configuration was rolled back. However, pipeline upload operations that timed out during the incident window could fail outright, causing the associated build to fail. The issue was not immediately detected for two reasons. First, the race condition meant it was effectively random whether a given container was affected. Approximately 0.35% of job dispatches and 0.5% of notifications needed retries across the incident window. Second, the failures did not trigger our existing alerts. When a worker hit a database connection failure, an internal error handling layer caught the error and silently rescheduled the job for retry instead of raising an exception. Our metrics layer saw no raised exception from these attempts, so they were indistinguishable from successful executions. An existing monitor that tracks P99 job dispatch duration saw no increase, because the failures happened almost instantly and looked identical to fast successes. The monitor is effective at detecting slow job processing, but this failure mode \(fast, silent, and rescheduled\) fell outside its coverage. A high volume of database connection errors was present in application logs, which is ultimately how responders identified the root cause during investigation. These factors — a low overall error rate, silent rescheduling, and a duration-based monitor that couldn't detect fast failures — contributed to the delay in detection. **Contributing factors:** * A race condition in the multi-process Sidekiq setup caused concurrent S3 environment file downloads to write to the same local path, allowing processes to read zero-byte files and boot without critical environment variables. * There was no startup assertion to fail fast if critical environment variables such as database configuration were missing or empty after the environment loading step. * Database connection errors in affected workers were caught by an internal error handling layer and silently rescheduled rather than raised as exceptions, preventing them from appearing in error metrics or alerting. * Existing monitoring was designed to detect slow job processing rather than fast failures, so this failure mode fell outside its coverage. **Key timestamps \(UTC\):** | Time | Event | | --- | --- | | 01:33 | Impact began — first database connection errors appeared | | 03:57 | Incident declared; investigation began | | 04:19 | Root cause identified; revert of infrastructure change initiated | | 04:37 | Revert deployed via application deployment | | 04:49 | Impact ended — error rates returned to zero | * **Total duration of impact: 3 hours and 16 minutes.** * **Time to detection: 2 hours and 24 minutes.** * **Time from detection to resolution: 52 minutes.** ## Changes we're making * **Startup validation for critical environment variables \(deployed\):** We’ve added a fail-fast assertion during application boot to verify that database URLs and other critical configuration values are present and non-empty after environment loading completes. * **Atomic S3 file downloads \(deployed\):** We’ve updated the S3 environment file download process to avoid the possibility of processes reading incomplete environment files. * **Improved job processing monitoring \(in progress\):** We are adding error-rate monitoring for job processing to complement our existing latency-based monitors, ensuring that workers which are picking up jobs but failing to complete them are detected regardless of how quickly the failure occurs. * **Incremental rollout for worker configuration changes \(in progress\)**: Future changes to how workers load critical configuration will be deployed to a subset of services first, allowing us to detect problems before they affect the full fleet.

Read the full incident report →

Started Feb 03, 2026, 03:36 PM UTC· resolved Feb 03, 2026, 06:44 PM UTC

1. investigating Feb 03, 2026, 03:36 PM UTC

   We are currently investigating delays with receiving Webhooks from GitHub.

2. investigating Feb 03, 2026, 04:20 PM UTC

   We are continuing to investigate this issue.

3. monitoring Feb 03, 2026, 05:05 PM UTC

   We're continuing to see delays in webhook delivery from GitHub, which may result in delays in Builds triggering and Commit Status updates.

4. monitoring Feb 03, 2026, 06:15 PM UTC

   We’re seeing improvements with webhook delivery from GitHub, and we continue to monitor the issue.

5. resolved Feb 03, 2026, 06:44 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 31, 2026, 10:06 AM UTC· resolved Jan 31, 2026, 11:35 AM UTC

1. investigating Jan 31, 2026, 10:06 AM UTC

   A third party database provider is experiencing an outage which is resulting in Test Engine services being unavailable. We will report updates as we receive them.

2. investigating Jan 31, 2026, 10:54 AM UTC

   We have been notified by the third party provider that they are currently restoring services. We will update when we have more information.

3. monitoring Jan 31, 2026, 11:15 AM UTC

   Our third party database provider has recovered their systems and we are have worked through our backlog of test executions. We are continuing to monitor to ensure our systems are healthy.

4. resolved Jan 31, 2026, 11:35 AM UTC

   We believe this issue is now resolved and all Test Engine systems are functioning. Ingestion of test data was delayed during this incident but we believe there has been no data loss. Customers using Buildkite Test Engine Client (bktec) to run tests will have experienced slower builds during the incident due to bktec using a sub-optimal test planning strategy while Test Engine APIs were unavailable.

Read the full incident report →

Started Jan 24, 2026, 11:00 PM UTC· resolved Jan 25, 2026, 08:00 AM UTC

1. resolved Jan 26, 2026, 07:21 AM UTC

   A maintenance process incorrectly marked some connected agents as lost between January 23–25 UTC. Agents on Elastic CI Stack recovered automatically; others may need a service restart. We've deployed a fix to prevent this from recurring.

2. postmortem Jan 26, 2026, 07:23 AM UTC

   ## Service Impact Agents that were in a connected or stopping state before January 23, 06:50 UTC—and remained in that state through January 25, 08:05 UTC—were incorrectly transitioned to lost, causing them to disconnect. **Recovery:** Agents running on Buildkite Elastic CI Stack for AWS automatically recovered within ~10 minutes. For other orchestration methods, restarting the agent service \(`systemctl restart buildkite-agent` or equivalent\) will restore connectivity. ## Incident Summary On January 23 06:50 UTC, we deployed a change to migrate agent heartbeat data to a new storage format. On January 25 at 08:05 UTC, a background maintenance process that monitors agent health could not locate heartbeat records for agents still using the previous format. These agents were incorrectly identified as unresponsive and marked as lost. Only agents in connected or stopping states were affected. Agents actively running jobs were unaffected. ## Changes we're making We are adding an immediate safeguard to the maintenance process so that agents in a connected state are never automatically marked as lost. Instead, mismatches will trigger an internal alert for investigation—ensuring we catch anomalies without customer impact. **We are restructuring our internal agent state storage to support staged rollouts, limiting the blast radius of future maintenance operations to a small subset of agents before wider deployment.** **Customer outreach:** We are proactively contacting affected customers whose agents have not automatically recovered to confirm whether manual intervention is needed.

Read the full incident report →