Blacksmith incident

Missing job history in the Blacksmith dashboard for runs before 3:30pm EDT today

Minor Resolved

Blacksmith experienced a minor incident on July 2, 2026 affecting API, lasting 13h 57m. The incident has been resolved; the full update timeline is below.

Started
Jul 02, 2026, 09:03 PM UTC
Resolved
Jul 03, 2026, 11:01 AM UTC
Duration
13h 57m
Detected by Pingoru
Jul 02, 2026, 09:03 PM UTC

Affected components

API

Update timeline

  1. identified Jul 02, 2026, 09:03 PM UTC

    All job history rows from before 3:30pm EDT today were lost. Data up is being restored from a backup.

  2. monitoring Jul 03, 2026, 12:41 AM UTC

    We are in the process of backfilling the data and customers should begin to start seeing data from 12am and earlier on their observability pages. We are still working on a resolution for today's gap (12AM - 3:30PM) and will update as soon as that fix is put in place.

  3. resolved Jul 03, 2026, 11:01 AM UTC

    This incident has been resolved. We will continue to backfill certain impacted time periods but observability data for customers is now accurately represented in the workflow and job dashboards.