Delayed Hosted Agents instance creation
Timeline · 1 update
- resolved Oct 19, 2026, 05:39 PM UTC
We are getting reports from a few customers that Hosted Agent jobs are delayed in starting. We are investigating.
Buildkite Outage History
Buildkite is up right nowBuildkite had 53 outages in the last 2 years totaling 56h 53m of downtime — averaging 2.2 incidents per month.
There were 53 Buildkite outages since October 15, 2025 totaling 56h 53m of downtime. Each is summarised below — incident details, duration, and resolution information.
Degraded checkout performance with Hosted Agents
Affected: Hosted Agents
Timeline · 3 updates
- investigating Oct 06, 2026, 06:19 PM UTC
We've received reports of degraded checkout performance with GitHub repositories when using Hosted Agents and are investigating.
- identified Oct 06, 2026, 07:07 PM UTC
We continue to observe degraded checkout performance. Our engineering teams are working on establishing an alternative network transit path to alleviate networking performance issues.
- resolved Oct 06, 2026, 11:59 PM UTC
The issue is now resolved. Network maintenance from our provider caused unexpected side effects that impacted outgoing traffic from Hosted Agents.
Small number of incoming GitHub Webhook failing
Affected: GitHub Webhooks
Timeline · 3 updates
- identified Oct 01, 2026, 06:10 AM UTC
Since 2026-09-30 20:52:36 UTC on 30 September, small amount of incoming GitHub webhook deliveries to Buildkite pipelines are being rejected, so the related builds are not created. For people who use GitHub.com may affect ability to receive status checks. We have identified the cause and are deploying a mitigation. This does not affect GitHub Enterprise Server and other SCM providers.
- monitoring Oct 01, 2026, 06:33 AM UTC
The mitigation is rolling out and we are seeing recovery. We will continue to monitor and provide an update.
- resolved Oct 01, 2026, 06:57 AM UTC
The error rate has returned to normal levels.
Delayed notifications
Affected: GitHub Commit Status NotificationsEmail NotificationsSlack NotificationsWebhook Notifications
Timeline · 4 updates
- investigating Sep 30, 2026, 01:03 AM UTC
We are investigating delays to build and job notifications for a subset of customers.
- identified Sep 30, 2026, 01:19 AM UTC
We've resolved the underlying error, and outbound notifications queues are now in the process of catching up for impacted customers.
- monitoring Sep 30, 2026, 01:43 AM UTC
Outbound notification queues have now fully caught up for all impacted customers.
- resolved Sep 30, 2026, 02:32 AM UTC
Outbound notification latency remains at expected levels. Some customers may have received duplicate notifications before the underlying error was resolved.
macOS hosted agent jobs not starting
Timeline · 1 update
- resolved Sep 30, 2026, 06:30 AM UTC
Between 11:45 and 15:00 (GMT+10) on 28 September, macOS hosted agent jobs could not start because of a misconfigured agent image update. This has been fixed and new jobs are running normally. Jobs from that window still waiting in the Scheduled state won't start on their own. Please cancel and retry them.
Issues with webhook ingestion, GraphQL queries
Timeline · 4 updates
AWS us-east-1 outage
Timeline · 2 updates
- monitoring Sep 21, 2026, 11:47 PM UTC
We're aware of a major AWS outage in us-east-1, we are taking action to prevent impact and monitoring the situation.
- resolved Sep 22, 2026, 12:32 AM UTC
We did not experience any customer impact during this AWS us-east-1 outage.
Increased latency and error rates
Affected: Agent API
Timeline · 3 updates
- investigating Sep 21, 2026, 10:35 PM UTC
We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
- monitoring Sep 21, 2026, 11:12 PM UTC
We observed a slight increase in latency for build creation isolated to a few customers, this has now passed and we are seeing signs of improvement.
- resolved Sep 21, 2026, 11:46 PM UTC
This incident has been resolved.
Buildkite service disruption
Affected: WebAgent APIREST APIJob Queue
Timeline · 6 updates
Hosted Agents - Increased latency and error rates
Affected: MacOSLinux (ARM64)Linux (AMD64)Hosted Agents
Timeline · 3 updates
- investigating Sep 04, 2026, 01:42 AM UTC
We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as they become available.
- monitoring Sep 04, 2026, 02:23 AM UTC
We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring.
- resolved Sep 04, 2026, 02:39 AM UTC
Jobs running on Hosted Agents are now being dispatched promptly.
Scheduling delay for hosted agents
Affected: Hosted Agents
Timeline · 4 updates
- investigating Sep 03, 2026, 06:00 AM UTC
We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
- investigating Sep 03, 2026, 06:09 AM UTC
Some job dispatches to hosted agents are timing out, leading to jobs starting late. We're currently investigating and will update again soon.
- monitoring Sep 03, 2026, 06:44 AM UTC
We're seeing recovery in hosted job scheduling rates. Your jobs should run, but there may be some delays as we work through any backlogs.
- resolved Sep 03, 2026, 07:06 AM UTC
Jobs running on Hosted Agents are now being dispatched promptly.
Delays in Hosted Agents job dispatch
Affected: MacOSLinux (ARM64)Linux (AMD64)Hosted Agents
Timeline · 5 updates
- investigating Sep 02, 2026, 02:45 PM UTC
We're investigating an issue with job dispatch to our Hosted Agents.
- investigating Sep 02, 2026, 03:16 PM UTC
Our engineers are continuing to investigate an issue with delayed job dispatch for Hosted Agents. Job dispatch for self-hosted agents remains unaffected.
- identified Sep 02, 2026, 03:29 PM UTC
We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs.
- monitoring Sep 02, 2026, 03:43 PM UTC
We are continuing to process through the backlog of delayed jobs and monitoring.
- resolved Sep 02, 2026, 03:51 PM UTC
Job dispatch for Hosted Agents is no longer experiencing delays.
Delivery issues with email notifications
Affected: Email Notifications
Timeline · 4 updates
- identified Sep 01, 2026, 03:32 PM UTC
We are experiencing delivery issues with email notifications due to an issue with an upstream provider.
- identified Sep 01, 2026, 04:19 PM UTC
Our on-call engineers are working on failing over to an alternative upstream email provider to restore outbound email notifications. No other Buildkite services have been impacted by this disruption.
- monitoring Sep 01, 2026, 04:38 PM UTC
We are seeing successful email deliveries and are monitoring processing of our outbound email queue.
- resolved Sep 01, 2026, 04:52 PM UTC
Our outbound email queue has finished processing all previously failed deliveries. Email notifications are processing successfully.
Test Engine ingestion delayed, catching up
Affected: Ingestion
Timeline · 3 updates
- monitoring Aug 27, 2026, 10:56 AM UTC
Internal processing of uploaded test execution data has been delayed, and is currently catching up. No data has been dropped, and all data should be processed over the next hour.
- monitoring Aug 27, 2026, 11:23 AM UTC
Catch-up processing continues, most of the backlog is processed, ingestion latency is decreasing.
- resolved Aug 27, 2026, 11:47 AM UTC
Processing of uploaded test execution data has caught up and is back to normal.
Issues identified with buildkite-agent registration
Affected: Agent API
Timeline · 5 updates
- investigating Aug 26, 2026, 01:42 AM UTC
We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
- investigating Aug 26, 2026, 02:02 AM UTC
We identified issues affecting certain customers when the buildkite-agent attempts to register with Buildkite. When agents are not able to register, builds may start to back up. We've identified a potential root case and are attempting changes to fix it.
- investigating Aug 26, 2026, 02:36 AM UTC
We have rolled out a few changes to fix the underlying issues and are seeing signs of recovery. We'll keep monitoring until all systems are fully recovered and communicate once we are confident everything is stable again.
- monitoring Aug 26, 2026, 02:43 AM UTC
Recovery is underway; we'll keep monitoring until recovery is complete.
- resolved Aug 26, 2026, 03:13 AM UTC
Mitigation strategies worked as expected and now agent registration is back to normal. All systems are stable now.
Unexpected agent disconnection for some customers
Affected: Agent API
Timeline · 3 updates
- monitoring Aug 25, 2026, 06:41 AM UTC
At 06:04 UTC we deployed a defective request routing change to `agent-edge.buildkite.com`, which caused some customers' agents to abruptly disconnect. At 06:09 UTC we noticed the problem, and immediately reverted the change. We began to observe recovery at 06:10 UTC, and are continuing to monitor.
- monitoring Aug 25, 2026, 07:42 AM UTC
Agent connectivity is recovered, but some agents are not reconnecting automatically after the earlier forced disconnects. We recommend restarting agent service to make agents reconnect.
- resolved Aug 25, 2026, 08:16 AM UTC
This incident has been resolved.
Slow web UI
Affected: WebWeb
Timeline · 4 updates
- investigating Aug 13, 2026, 03:41 PM UTC
We've spotted that something has gone wrong and there's elevated latency. We're currently investigating the issue, and will provide an update soon.
- identified Aug 13, 2026, 04:14 PM UTC
We've identified the cause of the issue and are actively mitigating.
- monitoring Aug 13, 2026, 04:29 PM UTC
We have identified and fixed the issue. We are monitoring and seeing signs of improvement.
- resolved Aug 13, 2026, 04:40 PM UTC
We think the impact from the issue is over.
Pipelines latency and webhook errors
Timeline · 1 update
- resolved Aug 05, 2026, 11:13 PM UTC
Between 22:03 - 22:16 UTC customers experienced delayed job dispatch, build processing and notifications. Additionally there were errors receiving webhooks during this period.
Latency on Pipelines
Affected: Job Queue
Timeline · 6 updates
Increased latency on REST and GraphQL APIs
Affected: REST APIRemote MCP Server
Timeline · 4 updates
- investigating Jun 11, 2026, 11:18 PM UTC
We're observing increased latency for all our customers. We're currently investigating and will provide status updates as they become available.
- monitoring Jun 11, 2026, 11:49 PM UTC
We've isolated the issue to elevated load on our REST API service and are working to mitigate. The agent and stacks API isn’t affected.
- monitoring Jun 11, 2026, 11:51 PM UTC
We've isolated the issue to elevated load on our REST API service and have mitigated the issue. The agent and stacks API isn’t affected.
- resolved Jun 12, 2026, 12:16 AM UTC
The mitigation applied before the last update had the intended effect, and we have seen recovery in REST API latency.
Increased latency and error rates for Agent API
Affected: Agent API
Timeline · 2 updates
- investigating Jun 11, 2026, 12:32 AM UTC
We're observing increased latency and error rates for a subset of our customers on the Agent API. We're currently investigating and will provide status updates as they become available.
- resolved Jun 11, 2026, 12:51 AM UTC
Between 00:05 - 00:34 UTC, a subset of customers experienced increased latency and timeout errors on the Agent API. This impacts job assignment. At peak impact, we saw an error rate of 1.3% of requests and job acceptance latency up to 53s.
Email deliveries are delayed
Timeline · 1 update
- resolved May 31, 2026, 11:45 PM UTC
We have received reports email deliveries have not been working, affecting signup and invite emails as well as build notification emails. This issue has now been resolved.
Delayed Test Engine ingestion processing
Affected: Ingestion
Timeline · 2 updates
- monitoring May 15, 2026, 06:51 AM UTC
Ingestion of Test Engine execution data from an internal queue to a data store stalled, has been resumed, and is working through the backlog. Visibility of test executions from the past hour hours will be delayed for approximately a further one hour. This has been a recurring issue; an architectural change is coming soon to eliminate this failure mode.
- resolved May 15, 2026, 07:35 AM UTC
Processing of the backlog is complete.
Error rates increasing
Affected: WebREST APIRemote MCP Server
Timeline · 2 updates
- investigating May 13, 2026, 03:14 PM UTC
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
- resolved May 13, 2026, 03:34 PM UTC
Additional capacity was added to our redis caches. This triggered a failover between UTC 15:10 - 15:14 and there was a spike of errors on the REST and GraphQL APIs. Customers would have seen some errors in the Buildkite UI during this period as well. We have been monitoring the situation since then and things have returned to baseline.