Optimism had 16 outages in the last 2 years — averaging 0.7 incidents per month.
There were 16 Optimism outages since June 20, 2024. Each is summarised below — incident details, duration, and resolution information.
Type: Incident Duration: 1 hour and 2 minutes Affected Components: Transaction Sequencing Mar 4, 23:54:08 GMT+0 - Investigating - Due to an investigation of network bottlenecks the Flashblocks streaming websocket endpoints are temporarily unavailable. Mar 5, 00:55:49 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 10 minutes Affected Components: Batch Submission, Public API, Withdrawals, , Deposits, Transaction Sequencing, Node Sync, Mainnet → Mar 4, 23:04:00 GMT+0 - Investigating - OP Mainnet has been observed to have various issues within its internal components causing delayed block building and sporadic chain stalls. We're investigating the incident. Mar 4, 23:10:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Mar 4, 23:14:00 GMT+0 - Resolved - This incident has been resolved, signified by the restored stable chain progression. Mar 4, 23:07:00 GMT+0 - Identified - The incident's root cause has been identified to be the problems with the networking infrastructure backing the communication between various critical block building components. We are continuing to work on a fix for this incident.
Type: Incident Duration: 4 hours and 2 minutes Affected Components: Transaction Sequencing Nov 19, 19:13:03 GMT+0 - Identified - The problem has been diagnosed and a fix is being worked on by the flashbots team. Nov 19, 21:30:24 GMT+0 - Monitoring - A fix has been applied and we are monitoring for stability. Nov 19, 21:36:43 GMT+0 - Resolved - This incident has been resolved, flashblocks are now operational again. Nov 19, 17:34:30 GMT+0 - Investigating - Following the activation of the Jovian hardfork, we detected some problems with op-rbuilder and are investigating the root cause. Nov 19, 17:45:20 GMT+0 - Investigating - To clarify, regular (non-flashblocks) usage is unaffected. Unsafe and safe head progression and the rest of the system are fully operational.
Type: Incident Duration: 22 minutes Affected Components: Public API Aug 19, 18:05:31 GMT+0 - Resolved - This incident has been resolved. Aug 19, 17:43:56 GMT+0 - Monitoring - We experienced an outage of our mainnet public endpoint during routine cluster upgrade. We implemented a fix and are currently monitoring the result.
Type: Incident Affected Components: Withdrawals May 14, 14:52:14 GMT+0 - Resolved - This is a test and not a real security incident. This is dedicated for testing our Incident Response Process in case of an emergency May 14, 15:08:28 GMT+0 - Resolved - Resolved DRILL TEST.
Type: Incident Duration: 16 minutes Affected Components: Withdrawals Apr 14, 15:47:34 GMT+0 - Investigating - This is test for testing our process for incident response this is a DRILL. Apr 14, 16:03:54 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 1 hour and 28 minutes Affected Components: Deposits, Withdrawals, Batch Submission, Node Sync, Public API, Transaction Sequencing Dec 5, 19:32:00 GMT+0 - Identified - OP Mainnet experienced a safe head stall from \~19:30- 21:00 UTC. The incident is already resolved and the safe head has caught up. Dec 5, 21:00:00 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 5 hours and 35 minutes Affected Components: Public API, Deposits, Node Sync, Withdrawals, Batch Submission, Transaction Sequencing Sep 6, 00:59:12 GMT+0 - Resolved - False alarm, no network degradation occurred Sep 6, 00:53:38 GMT+0 - Investigating - We are currently investigating this incident.
Type: Incident Duration: 19 minutes Affected Components: Node Sync, Transaction Sequencing Sep 3, 18:08:06 GMT+0 - Identified - We have identified the cause of the stall, and are deploying a mitigation. Nodes may need to wait for the next safe head increase before they resume syncing. Sep 3, 17:58:58 GMT+0 - Investigating - We are currently investigating this incident. Sep 3, 18:11:28 GMT+0 - Monitoring - We have deployed the mitigations and the network is syncing again. At this time the network is fully operational. Sep 3, 18:18:22 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 23 minutes Affected Components: Public API, Node Sync, Transaction Sequencing Aug 13, 17:23:25 GMT+0 - Resolved - This incident has been resolved, safe and unsafe chain are advancing normally. Thanks for your patience with us! Aug 13, 17:18:24 GMT+0 - Monitoring - The sequencer has caught up sequencing and the remaining network should pick up after the next safe head increase. This should resolve within the next few minutes. Aug 13, 17:00:00 GMT+0 - Investigating - We are investigating an OP Sepolia chain halt. All users are affected.
Type: Incident Duration: 16 minutes Affected Components: Withdrawals Aug 9, 16:03:16 GMT+0 - Identified - We have identified the problem and are working on a fix. Aug 9, 16:15:38 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Aug 9, 16:02:56 GMT+0 - Investigating - We are seeing a delay in our mainnet proposer instance submitting proposals on chain. Aug 9, 16:18:28 GMT+0 - Resolved - This incident has been resolved, proposals have resumed.
Type: Incident Duration: 1 day, 15 hours and 49 minutes Affected Components: Batch Submission Jun 29, 00:05:00 GMT+0 - Investigating - Due to heightened L2 block size on OP Sepolia, the batcher is seeing some performance degradation. This is causing the safe head to progress more slowly than usual. We are currently investigating. Jun 30, 14:28:07 GMT+0 - Monitoring - We have identified and applied a potential fix for the batcher performance issue. We will continue to monitor the situation. Jun 30, 15:54:23 GMT+0 - Resolved - The safe head has now resumed normal operation.
Type: Incident Duration: 32 minutes Affected Components: Transaction Sequencing Jun 26, 20:55:48 GMT+0 - Monitoring - We were conducting some cluster maintenance and had unexpected short downtime. We have implemented a fix and are currently monitoring the result. Jun 26, 21:28:08 GMT+0 - Resolved - This incident has been resolved.
Type: Incident Duration: 3 hours and 56 minutes Affected Components: Batch Submission, Transaction Sequencing Jun 20, 17:10:29 GMT+0 - Identified - We are currently experiencing elevated fees on OP Mainnet due to high blob fees on L1\. We are mitigating the issue by switching batch submission back to calldata. While the switch rolls out, users may experience higher fees than normal and the safe head may increase less often due to L1 congestion. Jun 20, 17:17:06 GMT+0 - Monitoring - We have performed the switch back to calldata. Users should observe a more stable safe head, as well as lower fees. We will switch back to blobs once the L1 congestion subsides. Jun 20, 18:26:00 GMT+0 - Monitoring - Blob data availability fees returned to stability. Batch submission is operational and utilizing blobs again. We will continue to monitor the L1 data fees for changes. Jun 20, 21:06:10 GMT+0 - Resolved - L2 fees are no longer elevated, L1 blob fees are trending down.