- Detected by Pingoru
- Oct 06, 2026, 06:19 PM UTC
- Resolved
- Oct 06, 2026, 11:59 PM UTC
- Duration
- 5h 40m
Affected: Hosted Agents
Timeline · 3 updates
-
investigating Oct 06, 2026, 06:19 PM UTC
We've received reports of degraded checkout performance with GitHub repositories when using Hosted Agents and are investigating.
-
identified Oct 06, 2026, 07:07 PM UTC
We continue to observe degraded checkout performance. Our engineering teams are working on establishing an alternative network transit path to alleviate networking performance issues.
-
resolved Oct 06, 2026, 11:59 PM UTC
The issue is now resolved. Network maintenance from our provider caused unexpected side effects that impacted outgoing traffic from Hosted Agents.
Read the full incident report →
- Detected by Pingoru
- Oct 01, 2026, 06:10 AM UTC
- Resolved
- Oct 01, 2026, 06:57 AM UTC
- Duration
- 47m
Affected: GitHub Webhooks
Timeline · 3 updates
-
identified Oct 01, 2026, 06:10 AM UTC
Since 2026-09-30 20:52:36 UTC on 30 September, small amount of incoming GitHub webhook deliveries to Buildkite pipelines are being rejected, so the related builds are not created. For people who use GitHub.com may affect ability to receive status checks. We have identified the cause and are deploying a mitigation. This does not affect GitHub Enterprise Server and other SCM providers.
-
monitoring Oct 01, 2026, 06:33 AM UTC
The mitigation is rolling out and we are seeing recovery. We will continue to monitor and provide an update.
-
resolved Oct 01, 2026, 06:57 AM UTC
The error rate has returned to normal levels.
Read the full incident report →
- Detected by Pingoru
- Sep 30, 2026, 01:03 AM UTC
- Resolved
- Sep 30, 2026, 02:32 AM UTC
- Duration
- 1h 28m
Affected: GitHub Commit Status NotificationsEmail NotificationsSlack NotificationsWebhook Notifications
Timeline · 4 updates
-
investigating Sep 30, 2026, 01:03 AM UTC
We are investigating delays to build and job notifications for a subset of customers.
-
identified Sep 30, 2026, 01:19 AM UTC
We've resolved the underlying error, and outbound notifications queues are now in the process of catching up for impacted customers.
-
monitoring Sep 30, 2026, 01:43 AM UTC
Outbound notification queues have now fully caught up for all impacted customers.
-
resolved Sep 30, 2026, 02:32 AM UTC
Outbound notification latency remains at expected levels. Some customers may have received duplicate notifications before the underlying error was resolved.
Read the full incident report →
- Detected by Pingoru
- Sep 28, 2026, 01:30 AM UTC
- Resolved
- Sep 28, 2026, 01:30 AM UTC
- Duration
- —
Timeline · 1 update
-
resolved Sep 30, 2026, 06:30 AM UTC
Between 11:45 and 15:00 (GMT+10) on 28 September, macOS hosted agent jobs could not start because of a misconfigured agent image update. This has been fixed and new jobs are running normally. Jobs from that window still waiting in the Scheduled state won't start on their own. Please cancel and retry them.
Read the full incident report →
- Detected by Pingoru
- Sep 22, 2026, 08:07 PM UTC
- Resolved
- Sep 22, 2026, 08:50 PM UTC
- Duration
- 42m
Timeline · 4 updates
Read the full incident report →
Notice September 21, 2026 - Detected by Pingoru
- Sep 21, 2026, 11:47 PM UTC
- Resolved
- Sep 22, 2026, 12:32 AM UTC
- Duration
- 45m
Timeline · 2 updates
-
monitoring Sep 21, 2026, 11:47 PM UTC
We're aware of a major AWS outage in us-east-1, we are taking action to prevent impact and monitoring the situation.
-
resolved Sep 22, 2026, 12:32 AM UTC
We did not experience any customer impact during this AWS us-east-1 outage.
Read the full incident report →
- Detected by Pingoru
- Sep 21, 2026, 10:35 PM UTC
- Resolved
- Sep 21, 2026, 11:46 PM UTC
- Duration
- 1h 10m
Affected: Agent API
Timeline · 3 updates
-
investigating Sep 21, 2026, 10:35 PM UTC
We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
-
monitoring Sep 21, 2026, 11:12 PM UTC
We observed a slight increase in latency for build creation isolated to a few customers, this has now passed and we are seeing signs of improvement.
-
resolved Sep 21, 2026, 11:46 PM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- Sep 09, 2026, 07:07 PM UTC
- Resolved
- Sep 10, 2026, 01:54 AM UTC
- Duration
- 6h 46m
Affected: WebAgent APIREST APIJob Queue
Timeline · 6 updates
Read the full incident report →
- Detected by Pingoru
- Sep 04, 2026, 01:42 AM UTC
- Resolved
- Sep 04, 2026, 02:39 AM UTC
- Duration
- 56m
Affected: MacOSLinux (ARM64)Linux (AMD64)Hosted Agents
Timeline · 3 updates
-
investigating Sep 04, 2026, 01:42 AM UTC
We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as they become available.
-
monitoring Sep 04, 2026, 02:23 AM UTC
We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring.
-
resolved Sep 04, 2026, 02:39 AM UTC
Jobs running on Hosted Agents are now being dispatched promptly.
Read the full incident report →
- Detected by Pingoru
- Sep 03, 2026, 06:00 AM UTC
- Resolved
- Sep 03, 2026, 07:06 AM UTC
- Duration
- 1h 5m
Affected: Hosted Agents
Timeline · 4 updates
-
investigating Sep 03, 2026, 06:00 AM UTC
We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
-
investigating Sep 03, 2026, 06:09 AM UTC
Some job dispatches to hosted agents are timing out, leading to jobs starting late. We're currently investigating and will update again soon.
-
monitoring Sep 03, 2026, 06:44 AM UTC
We're seeing recovery in hosted job scheduling rates. Your jobs should run, but there may be some delays as we work through any backlogs.
-
resolved Sep 03, 2026, 07:06 AM UTC
Jobs running on Hosted Agents are now being dispatched promptly.
Read the full incident report →
- Detected by Pingoru
- Sep 02, 2026, 02:45 PM UTC
- Resolved
- Sep 02, 2026, 03:51 PM UTC
- Duration
- 1h 5m
Affected: MacOSLinux (ARM64)Linux (AMD64)Hosted Agents
Timeline · 5 updates
-
investigating Sep 02, 2026, 02:45 PM UTC
We're investigating an issue with job dispatch to our Hosted Agents.
-
investigating Sep 02, 2026, 03:16 PM UTC
Our engineers are continuing to investigate an issue with delayed job dispatch for Hosted Agents. Job dispatch for self-hosted agents remains unaffected.
-
identified Sep 02, 2026, 03:29 PM UTC
We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs.
-
monitoring Sep 02, 2026, 03:43 PM UTC
We are continuing to process through the backlog of delayed jobs and monitoring.
-
resolved Sep 02, 2026, 03:51 PM UTC
Job dispatch for Hosted Agents is no longer experiencing delays.
Read the full incident report →
- Detected by Pingoru
- Sep 01, 2026, 03:32 PM UTC
- Resolved
- Sep 01, 2026, 04:52 PM UTC
- Duration
- 1h 19m
Affected: Email Notifications
Timeline · 4 updates
-
identified Sep 01, 2026, 03:32 PM UTC
We are experiencing delivery issues with email notifications due to an issue with an upstream provider.
-
identified Sep 01, 2026, 04:19 PM UTC
Our on-call engineers are working on failing over to an alternative upstream email provider to restore outbound email notifications. No other Buildkite services have been impacted by this disruption.
-
monitoring Sep 01, 2026, 04:38 PM UTC
We are seeing successful email deliveries and are monitoring processing of our outbound email queue.
-
resolved Sep 01, 2026, 04:52 PM UTC
Our outbound email queue has finished processing all previously failed deliveries. Email notifications are processing successfully.
Read the full incident report →
- Detected by Pingoru
- Aug 27, 2026, 10:56 AM UTC
- Resolved
- Aug 27, 2026, 11:47 AM UTC
- Duration
- 51m
Affected: Ingestion
Timeline · 3 updates
-
monitoring Aug 27, 2026, 10:56 AM UTC
Internal processing of uploaded test execution data has been delayed, and is currently catching up. No data has been dropped, and all data should be processed over the next hour.
-
monitoring Aug 27, 2026, 11:23 AM UTC
Catch-up processing continues, most of the backlog is processed, ingestion latency is decreasing.
-
resolved Aug 27, 2026, 11:47 AM UTC
Processing of uploaded test execution data has caught up and is back to normal.
Read the full incident report →
- Detected by Pingoru
- Aug 26, 2026, 01:42 AM UTC
- Resolved
- Aug 26, 2026, 03:13 AM UTC
- Duration
- 1h 30m
Affected: Agent API
Timeline · 5 updates
-
investigating Aug 26, 2026, 01:42 AM UTC
We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
-
investigating Aug 26, 2026, 02:02 AM UTC
We identified issues affecting certain customers when the buildkite-agent attempts to register with Buildkite. When agents are not able to register, builds may start to back up. We've identified a potential root case and are attempting changes to fix it.
-
investigating Aug 26, 2026, 02:36 AM UTC
We have rolled out a few changes to fix the underlying issues and are seeing signs of recovery. We'll keep monitoring until all systems are fully recovered and communicate once we are confident everything is stable again.
-
monitoring Aug 26, 2026, 02:43 AM UTC
Recovery is underway; we'll keep monitoring until recovery is complete.
-
resolved Aug 26, 2026, 03:13 AM UTC
Mitigation strategies worked as expected and now agent registration is back to normal. All systems are stable now.
Read the full incident report →
- Detected by Pingoru
- Aug 25, 2026, 06:41 AM UTC
- Resolved
- Aug 25, 2026, 08:16 AM UTC
- Duration
- 1h 34m
Affected: Agent API
Timeline · 3 updates
-
monitoring Aug 25, 2026, 06:41 AM UTC
At 06:04 UTC we deployed a defective request routing change to `agent-edge.buildkite.com`, which caused some customers' agents to abruptly disconnect. At 06:09 UTC we noticed the problem, and immediately reverted the change. We began to observe recovery at 06:10 UTC, and are continuing to monitor.
-
monitoring Aug 25, 2026, 07:42 AM UTC
Agent connectivity is recovered, but some agents are not reconnecting automatically after the earlier forced disconnects. We recommend restarting agent service to make agents reconnect.
-
resolved Aug 25, 2026, 08:16 AM UTC
This incident has been resolved.
Read the full incident report →
- Detected by Pingoru
- Aug 13, 2026, 03:41 PM UTC
- Resolved
- Aug 13, 2026, 04:40 PM UTC
- Duration
- 58m
Affected: WebWeb
Timeline · 4 updates
-
investigating Aug 13, 2026, 03:41 PM UTC
We've spotted that something has gone wrong and there's elevated latency. We're currently investigating the issue, and will provide an update soon.
-
identified Aug 13, 2026, 04:14 PM UTC
We've identified the cause of the issue and are actively mitigating.
-
monitoring Aug 13, 2026, 04:29 PM UTC
We have identified and fixed the issue. We are monitoring and seeing signs of improvement.
-
resolved Aug 13, 2026, 04:40 PM UTC
We think the impact from the issue is over.
Read the full incident report →
- Detected by Pingoru
- Aug 05, 2026, 10:00 PM UTC
- Resolved
- Aug 05, 2026, 10:00 PM UTC
- Duration
- —
Timeline · 1 update
-
resolved Aug 05, 2026, 11:13 PM UTC
Between 22:03 - 22:16 UTC customers experienced delayed job dispatch, build processing and notifications. Additionally there were errors receiving webhooks during this period.
Read the full incident report →
- Detected by Pingoru
- Aug 03, 2026, 05:11 AM UTC
- Resolved
- Aug 03, 2026, 05:36 AM UTC
- Duration
- 25m
Affected: Job Queue
Timeline · 6 updates
Read the full incident report →
- Detected by Pingoru
- Jun 11, 2026, 11:18 PM UTC
- Resolved
- Jun 12, 2026, 12:16 AM UTC
- Duration
- 57m
Affected: REST APIRemote MCP Server
Timeline · 4 updates
-
investigating Jun 11, 2026, 11:18 PM UTC
We're observing increased latency for all our customers. We're currently investigating and will provide status updates as they become available.
-
monitoring Jun 11, 2026, 11:49 PM UTC
We've isolated the issue to elevated load on our REST API service and are working to mitigate. The agent and stacks API isn’t affected.
-
monitoring Jun 11, 2026, 11:51 PM UTC
We've isolated the issue to elevated load on our REST API service and have mitigated the issue. The agent and stacks API isn’t affected.
-
resolved Jun 12, 2026, 12:16 AM UTC
The mitigation applied before the last update had the intended effect, and we have seen recovery in REST API latency.
Read the full incident report →
- Detected by Pingoru
- Jun 11, 2026, 12:32 AM UTC
- Resolved
- Jun 11, 2026, 12:51 AM UTC
- Duration
- 18m
Affected: Agent API
Timeline · 2 updates
-
investigating Jun 11, 2026, 12:32 AM UTC
We're observing increased latency and error rates for a subset of our customers on the Agent API. We're currently investigating and will provide status updates as they become available.
-
resolved Jun 11, 2026, 12:51 AM UTC
Between 00:05 - 00:34 UTC, a subset of customers experienced increased latency and timeout errors on the Agent API. This impacts job assignment. At peak impact, we saw an error rate of 1.3% of requests and job acceptance latency up to 53s.
Read the full incident report →
- Detected by Pingoru
- May 30, 2026, 12:30 AM UTC
- Resolved
- May 30, 2026, 12:30 AM UTC
- Duration
- —
Timeline · 1 update
-
resolved May 31, 2026, 11:45 PM UTC
We have received reports email deliveries have not been working, affecting signup and invite emails as well as build notification emails. This issue has now been resolved.
Read the full incident report →
- Detected by Pingoru
- May 15, 2026, 06:51 AM UTC
- Resolved
- May 15, 2026, 07:35 AM UTC
- Duration
- 44m
Affected: Ingestion
Timeline · 2 updates
-
monitoring May 15, 2026, 06:51 AM UTC
Ingestion of Test Engine execution data from an internal queue to a data store stalled, has been resumed, and is working through the backlog. Visibility of test executions from the past hour hours will be delayed for approximately a further one hour. This has been a recurring issue; an architectural change is coming soon to eliminate this failure mode.
-
resolved May 15, 2026, 07:35 AM UTC
Processing of the backlog is complete.
Read the full incident report →
- Detected by Pingoru
- May 13, 2026, 03:14 PM UTC
- Resolved
- May 13, 2026, 03:34 PM UTC
- Duration
- 19m
Affected: WebREST APIRemote MCP Server
Timeline · 2 updates
-
investigating May 13, 2026, 03:14 PM UTC
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
-
resolved May 13, 2026, 03:34 PM UTC
Additional capacity was added to our redis caches. This triggered a failover between UTC 15:10 - 15:14 and there was a spike of errors on the REST and GraphQL APIs. Customers would have seen some errors in the Buildkite UI during this period as well. We have been monitoring the situation since then and things have returned to baseline.
Read the full incident report →
- Detected by Pingoru
- May 08, 2026, 01:12 AM UTC
- Resolved
- May 09, 2026, 04:34 AM UTC
- Duration
- 1d 3h
Timeline · 11 updates
Read the full incident report →
- Detected by Pingoru
- May 07, 2026, 10:45 PM UTC
- Resolved
- May 08, 2026, 12:13 AM UTC
- Duration
- 1h 28m
Affected: Job QueueWebhook Notifications
Timeline · 5 updates
Read the full incident report →
