Buildkite incident

Delayed processing of execution uploads

Minor Resolved View vendor source →

Buildkite experienced a minor incident on September 30, 2025 affecting Ingestion, lasting 10h 43m. The incident has been resolved; the full update timeline is below.

Started
Sep 30, 2025, 07:44 PM UTC
Resolved
Oct 01, 2025, 06:28 AM UTC
Duration
10h 43m
Detected by Pingoru
Sep 30, 2025, 07:44 PM UTC

Affected components

Ingestion

Update timeline

  1. investigating Sep 30, 2025, 07:44 PM UTC

    We're investigating delayed processing of execution uploads in the ingestion pipeline. You may see partial data appearing for Runs and Tests in Test Engine.

  2. identified Sep 30, 2025, 09:04 PM UTC

    We have identified an issue with processing uploads to Test Engine and are working on a fix.

  3. monitoring Sep 30, 2025, 10:12 PM UTC

    We have determined the root cause and have fixed the issue. The system is now processing a backlog of execution data.

  4. resolved Oct 01, 2025, 06:28 AM UTC

    We think the impact from the issue is over.