Hedera incident

Hedera Mirrornode Outage

Major Resolved View vendor source →

Hedera experienced a major incident on October 26, 2022 affecting Mirror Node REST API Mainnet and Mirror Node HCS gRPC Mainnet, lasting 2h 52m. The incident has been resolved; the full update timeline is below.

Started
Oct 26, 2022, 08:34 PM UTC
Resolved
Oct 26, 2022, 11:26 PM UTC
Duration
2h 52m
Detected by Pingoru
Oct 26, 2022, 08:34 PM UTC

Affected components

Mirror Node REST API MainnetMirror Node HCS gRPC Mainnet

Update timeline

  1. investigating Oct 26, 2022, 08:34 PM UTC

    Hedera is observing an incident with mirrornode processing transactions. We are investigating the cause and working to mitigate the impact on applications.

  2. investigating Oct 26, 2022, 08:54 PM UTC

    We are continuing to investigate this issue.

  3. identified Oct 26, 2022, 09:35 PM UTC

    The transaction processing issue in mirrornode has been identified and we are working on a fix to deploy. Event streams, account balances, and recordstreams are still being uploaded and transactions are still being processed by consensus nodes, but they are not able to be processed by mirrornodes.

  4. identified Oct 26, 2022, 11:18 PM UTC

    We are continuing to work on a fix for this issue.

  5. monitoring Oct 26, 2022, 11:21 PM UTC

    Hedera core engineering has deployed the v0.66.4 software update to mirrornode and is continuing to monitor.

  6. resolved Oct 26, 2022, 11:26 PM UTC

    Hedera core engineering has confirmed that the mirrornode is now functional; mainnet transactions which occurred during the period of mirrornode downtime are now queryable. Mirrornode operators are advised to upgrade to version v0.66.4.