Buildkite incident

Increased latency and timeouts on REST and GraphQL API

Notice Resolved View vendor source →

Buildkite experienced a notice incident on September 19, 2025, lasting 1h 29m. The incident has been resolved; the full update timeline is below.

Started
Sep 19, 2025, 07:47 PM UTC
Resolved
Sep 19, 2025, 09:17 PM UTC
Duration
1h 29m
Detected by Pingoru
Sep 19, 2025, 07:47 PM UTC

Update timeline

  1. investigating Sep 19, 2025, 07:47 PM UTC

    We are investigating increased latency and timeouts with the REST and GraphQL endpoints.

  2. identified Sep 19, 2025, 08:30 PM UTC

    We have discovered a rogue puma process contributing to the high latency and have restarted it. We are now investigating other processes to determine if they are exhibiting the same rogue behaviour.

  3. monitoring Sep 19, 2025, 09:07 PM UTC

    We've identified a large increase in load which has now been resolved, and we're monitoring to ensure request latency returns to normal operating levels.

  4. resolved Sep 19, 2025, 09:17 PM UTC

    We are happy that request latency has dropped to normal levels so consider this issue resolved.