Node Sync Stall on GIWA Sepolia Testnet

Update timeline

1. investigating Aug 25, 2026, 02:35 AM UTC

   Nodes on GIWA Sepolia testnet other than the sequencer have stopped tracking the chain head following a sequencer leader transition. RPC endpoints and the block explorer are serving stale data. L2 block production is unaffected and there is no reorg. We are investigating.

2. resolved Aug 25, 2026, 02:36 AM UTC

   Between 11:36 and 14:05 UTC on 29 July 2026, nodes on GIWA Sepolia testnet other than the sequencer stalled and stopped tracking the chain head. RPC endpoints and the block explorer served stale data during this window. L2 block production was unaffected and there was no reorg.

3. postmortem Aug 25, 2026, 02:39 AM UTC

   ## Overview On 29 July 2026 at 11:36 UTC, the sequencer leader on GIWA Sepolia testnet failed over as designed. During this transition, one block was replicated to the consensus layer shared between sequencers but was not propagated over the peer-to-peer network. Nodes that did not receive this block were unable to connect subsequent blocks to their chain, causing their sync to stall. RPC responses and the block explorer remained fixed at the block height where the incident began. **L2 block production did not stop.** The failover behaved as intended and the new leader continued producing blocks immediately. **There was no reorg, no safe-head or finality violation, and no funds were at risk.** ## Impact * **Public RPC endpoints** — Block, transaction, and state queries were fixed at the height where the incident began. Reads continued to be served, though against a stale state. The Flashblocks endpoint did not respond normally. * **Transaction processing** — The submission path did not function normally. * **Block explorer** — Indexing stalled, so recent blocks and transactions were not displayed. **L2 block production and L1 batch submission were unaffected throughout.** The sequencer produced blocks without interruption, and those blocks were posted to L1 as normal. ## Timeline * **11:36:11** — The sequencer produced block 31979856 and propagated it over peer-to-peer. * **11:36:12–17** — A consensus log snapshot operation on the leader node ran longer than usual. During that window, the attempt to replicate a newly produced block to the consensus layer timed out. The consensus layer marked that sequencer unhealthy and transferred leadership. * **11:36:18** — The new leader produced and propagated block 31979858. From this point, the block sequence observed on the network was missing 31979857. * **11:36:20** — Other nodes received 31979858 but could not attach it to their chain because the preceding block was absent. * **11:42** — The batch containing 31979857 was posted to L1. * **11:44** — A node that maintained external peer-to-peer connectivity received the block and recovered on its own. * **13:58–14:05** — We changed the sync configuration on one internal node so it could obtain the missing block directly. Using that node as the starting point, the remaining nodes and the block explorer recovered in sequence. ## Root Cause ### 1. Trigger — Consensus layer snapshot delay GIWA runs sequencers in a high-availability configuration that uses consensus-based leader election. The elected leader produces blocks, and each block is replicated to the other sequencers in the cluster. At 11:36:12 UTC, a consensus log snapshot operation began on the leader node. This operation normally completes within a few hundred milliseconds; on this occasion it took approximately 4.8 seconds. Because the operation is blocking, the leader's attempt to replicate its newly produced block timed out while it was in progress. The consensus layer detected that the leader's block progression had fallen behind its threshold and transferred leadership. **Up to this point, the high-availability configuration behaved exactly as designed.** ### 2. One block existed only in the consensus layer A sequencer produces a block, replicates it to the consensus layer, and propagates it over the peer-to-peer network only after replication succeeds. Because propagation is conditional on successful replication, a replication timeout means propagation never runs. Block 31979857 fell on that boundary. Replication to the cluster completed, but the leader stepped down before receiving the acknowledgement, so the block never reached the propagation step. The new leader recognized the block recorded in the consensus layer as a valid predecessor and produced and propagated the next block from it. **31979857 is not an orphaned block.** The chain is continuous and the block is valid. What was missing was not the block itself but one propagation path, and other nodes temporarily lost the means to obtain a single intermediate block. ### 3. Why recovery was delayed The missing block could only be obtained through the execution layer's peer-to-peer network. Peer-to-peer block propagation pushes newly created blocks and provides no way to request past ones. Re-deriving the block from the L1 batch was also possible in principle, but on some nodes the derived result did not progress through to the execution layer. The difference between the node that recovered on its own at 11:44 and those that did not was **execution layer peer-to-peer connectivity.** The recovered node maintained connections to external nodes. The others had a restricted set of connection targets, and when all of those targets were stalled in the same way, there was nowhere to obtain the block from. The eight-minute delay has a specific explanation. At 11:36, no node anywhere on the network could serve this block over peer-to-peer. When the batch was posted to L1 at 11:42, external nodes obtained the block from L1, and only then did a peer holding it exist on the peer-to-peer network. **What was missing was not the number of peers but a peer that held the block.** In addition, some of the stalled nodes **failed silently rather than erroring.** Consensus layer block height continued to advance, so monitoring indicators showed these nodes as healthy. This delayed our assessment of the full scope of impact. ## Remediation **Recovery action \(13:58 UTC\)** — We changed the sync configuration on one internal node so it would obtain the missing block directly from the L1 batch. Once that node recovered and held the block, the remaining nodes recovered in sequence, and the block explorer returned to normal at 14:05 UTC. The following work is complete. * **Strengthened execution layer peer configuration** — Nodes in continuous operation now maintain stable connectivity to peers that can serve blocks. If peer-to-peer propagation is momentarily missed during a leader transition, a node requests the preceding block immediately and recovers automatically. * **Standardized sync configuration for nodes in continuous operation** — Consolidated onto a configuration in which the condition described in section 3 does not arise. * **Added alerting on divergence between layer block heights** — We now continuously monitor the difference between consensus layer and execution layer block height, which detects stalls that occur without errors. * **Added instrumentation for consensus layer snapshot duration** — Snapshot operation duration is now continuously measured, so delays relative to normal can be detected before they trigger a failover. The condition that produces a block replicated to the consensus layer but not propagated over peer-to-peer during a leader failover can arise structurally. The purpose of this work is not to eliminate that condition but to ensure that when it occurs, **nodes recover automatically without manual intervention.** No incident of this kind has occurred since these changes were applied. ## If You Operate a Node If you run a GIWA node yourself, we recommend **maintaining stable execution layer peer-to-peer connectivity.** Obtaining a missing block depends on the execution layer peer-to-peer network, so recovery on your own may not be possible if your connection targets are restricted or if they share the same failure condition. ## Notes * GIWA Sepolia is a test network intended for testing and validation. Its characteristics and terms of use are defined in the [Testnet Terms of Use](https://docs.giwa.io/terms-and-policies/testnet-terms-of-use). * The impact window above reflects the full incident duration and is not specific to any individual user or service. Actual impact varied depending on the endpoint used and how it was queried.