Avalanche experienced a critical incident on February 23, 2024 affecting Mainnet Primary Network, lasting 6h 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 23, 2024, 12:11 PM UTC
Developers across the community are currently investigating a stall in block finalization that is preventing blocks from being accepted on the Primary Network. Updates will be posted here as the issue is investigated.
- identified Feb 23, 2024, 03:59 PM UTC
🚨🚨 Attention Avalanche Validators 🚨🚨 Please upgrade your node to AvalancheGo to v1.11.1: https://github.com/ava-labs/avalanchego/releases/tag/v1.11.1 This release disables logic added in v1.10.18 that lead to validators sending an excessive amount of gossip to each other. Avalanche Validators provision a stake-weighted bandwidth allocation for each peer and this buggy logic led to each node saturating their allocation with useless transaction gossip. This dynamic prevented pull queries issued by the validator from being processed in a timely manner and led to consensus stalling (as no polls were being handled). As soon as a sufficient amount of stake upgrades to this release, consensus should return to normal.
- monitoring Feb 23, 2024, 04:36 PM UTC
Block finalization on the Primary Network is beginning to resume. Stability will fluctuate as validators continue updating to [email protected]. You can monitor block finalization here: https://subnets.avax.network/c-chain
- monitoring Feb 23, 2024, 05:36 PM UTC
A significant % of stake weight has upgraded to [email protected] and block finalization is stable. We will continue monitoring for further regressions.
- resolved Feb 23, 2024, 05:43 PM UTC
This incident has been resolved.