Filecoin experienced a critical incident on December 19, 2020 affecting Chain Height, lasting 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 19, 2020, 09:29 PM UTC
The Filecoin network is halted with most nodes stuck at height 336459 with mismatched state compute error due to indeterminate behavior. We have found the root cause and are testing the fix. Next Step: Miners will need to upgrade to lotus v1.4.0 (https://github.com/filecoin-project/lotus/releases/tag/v1.4.0) with the fix when it is released. Once miners upgrade, the chain will start to catch up.
- identified Dec 19, 2020, 10:16 PM UTC
Lotus v1.4.0 (https://github.com/filecoin-project/lotus/releases/tag/v1.4.0) has been released. Please upgrade ASAP and see the recovery instructions (https://github.com/filecoin-project/lotus/discussions/5232) for more information/troubleshooting.
- monitoring Dec 19, 2020, 11:26 PM UTC
The network has restarted and progressed to the expected block height (337120 @ 3:20pm PT). As more power joins the upgraded chain, the network will continue to stabilize. Check back for an update here when >50% of power is in sync.
- monitoring Dec 19, 2020, 11:27 PM UTC
We are continuing to monitor for any further issues.
- resolved Dec 20, 2020, 12:03 AM UTC
If you’re just catching up: check the upgrade and recovery instructions (https://github.com/filecoin-project/lotus/discussions/5232) to get back in sync. Miners have upgraded & the chain is now recovering smoothly. 🎉🙏💪 We don’t expect further action is required, but will continue monitoring. Expect degraded chain quality. We should expect the network to be somewhat degraded for the next few hours as miners continue to upgrade and power recovers (currently ~34% upgraded). We recommend pausing all non-critical messages until chain capacity is restored to normal levels - BaseFee may be volatile as the network recovers. Full recovery ETA. Conservatively, we will deem the network to be fully recovered & stable 2x finality from when the chain resumed. This means 1,400 epochs (11.6 hrs) from 337120 (2020-12-19 23:20 UTC). So, conservatively, full recovery at epoch 338520 (2020-12-20 11:00 UTC). For custodians, exchanges, wallets: The full recovery ETA is very conservative, and the chain is expected to be fully functional long before. Our recommendation for you would be to keep transfers out disabled until that time or until you gain comfort with chain status -- at least for larger amounts. Your own internal transfers, trading, and other fully off-chain activity should be fine.
- postmortem Jan 16, 2021, 11:10 PM UTC
Please see this post on [**Resolving the Dec 19 Chain Halt: Cause, Impact, & Take Aways **](https://filecoin.io/blog/resolving-the-chain-halt/)for cause and impact of this outage. More specific post mortems WIP from the actors & infra teams to describe specific learnings and mitigations for the future.