Buildkite incident

Elevated ingestion latency for Test Engine

Minor Resolved View vendor source →

Buildkite experienced a minor incident on March 4, 2026 affecting Ingestion, lasting 8h 36m. The incident has been resolved; the full update timeline is below.

Started
Mar 04, 2026, 03:59 AM UTC
Resolved
Mar 04, 2026, 12:35 PM UTC
Duration
8h 36m
Detected by Pingoru
Mar 04, 2026, 03:59 AM UTC

Affected components

Ingestion

Update timeline

  1. monitoring Mar 04, 2026, 03:59 AM UTC

    We've identified and fixed the root cause and now monitoring the issue. Test execution may be delayed for up to 30 minutes.

  2. monitoring Mar 04, 2026, 08:00 AM UTC

    Processing the backlog of test executions is taking longer than expected, so elevated ingestion latency remains. We continue to monitor progress. We have also identified database schema improvements we can make that will increase throughput to avoid this scenario in future.

  3. resolved Mar 04, 2026, 12:35 PM UTC

    Processing of test execution ingestion data has successfully caught up.