Hiro incident

Mainnet API Block Progression Stalled

Major Resolved View vendor source →

Hiro experienced a major incident on December 19, 2025 affecting Stacks Blockchain API and explorer.hiro.so, lasting 32m. The incident has been resolved; the full update timeline is below.

Started
Dec 19, 2025, 02:44 PM UTC
Resolved
Dec 19, 2025, 03:16 PM UTC
Duration
32m
Detected by Pingoru
Dec 19, 2025, 02:44 PM UTC

Affected components

Stacks Blockchain APIexplorer.hiro.so

Update timeline

  1. investigating Dec 19, 2025, 02:44 PM UTC

    We're investigating a stall in block progression for the Mainnet Stacks Blockchain API.

  2. identified Dec 19, 2025, 02:50 PM UTC

    The issue has been identified and a fix is being implemented.

  3. resolved Dec 19, 2025, 03:16 PM UTC

    This incident has been resolved, and a post-mortem will be added later today.

  4. postmortem Dec 19, 2025, 05:45 PM UTC

    # Summary On December 19, 2025, at approximately 8:47 AM ET, the mainnet Stacks Blockchain API stopped advancing at Bitcoin block 928,558 and Stacks block 5,385,227. This stall impacted the Explorer and any services relying on the mainnet Stacks Blockchain API. Block ingestion was restored at approximately 9:46 AM ET, after which the API resumed syncing. The API fully caught up to chain tip by 10:11 AM ET. # Impact The stall affected all consumers of the mainnet Stacks Blockchain API, including: * The Stacks Explorer * Chainhooks v2 * Any downstream services dependent on API block progression Detection and mitigation were delayed due to a misconfigured silenced alert in Hiro’s alerting system. As a result, engineers were notified later than expected, extending the duration of user impact. # Root cause The definitive root cause is still under investigation. However, initial analysis indicates that the Stacks Blockchain API entered an unexpected internal state regarding an internal block-synchronization system it depends on, preventing further block progression. Recovery required manual intervention at the database level to reset the API’s block sync state. For context, Hiro operates a custom internal block-sync system to improve sync performance and recovery behavior for our API services. We do not believe this condition would affect users syncing directly from a standard Stacks Core node. # Resolution Once discovered, action was immediately taken to communicate, diagnose, and rectify the mainnet Stacks Blockchain API stall via manual database intervention. Additionally, the silenced alert that should have notified Hiro engineers has been corrected. Should this happen again before a permanent fix is in place, our engineers will be notified appropriately and able to restore block progression much faster. Further action will be taken to verify any issues in how the Stacks Blockchain API interacts with our block-sync system and implement a permanent fix. # Action Items Below are some items Hiro will complete to prevent another mainnet Stacks Blockchain API stall under similar conditions, and to improve the speed at which it was recovered: * Investigate any issues in the Stacks Blockchain API that would lead to an unexpected state with our internal block-sync system, and implement a permanent fix. * Update any critical alerts to reduce the need for silencing specific instances of them, reducing the possibility for human error. * Add a new alert\(s\) for any gaps found.