How we broke four validators by updating one

This is an incident report. The chain was down for about a day, and it was entirely self-inflicted.

What happened

We updated the explorer and RPC server on the primary validator to add governance views. Standard deployment — SSH in, upload new source, build, restart. The explorer looked great. What we didn't do: update the other three validators.

The divergence

The primary started producing blocks with the new code. The other three validators — still running old code — tried to sync those blocks. At block 83,623, the primary processed a transaction that produced different state under the new code. The state root in block 83,623's header didn't match what the peers computed locally.

The peers rejected the block. Then the next one. Then the next one. They got stuck in a loop: fetch block → apply → state root mismatch → retry. For over 200,000 blocks. The primary kept producing solo, advancing to block 290,871 while peers were stuck at 83,622.

The cascade

Wipe and re-sync would take 160 hours. Copying the 33MB data directory failed because pnpm's symlink-based node_modules didn't survive the copy. We ended up doing a fresh genesis — three times before getting it right, each attempt hitting a different issue.

The mempool bug

During recovery, we discovered the mempool had no deduplication and no eviction. Peer nodes forwarded faucet transactions to each other, duplicates accumulated to 60,000+, and the chain produced empty blocks. We shipped a fix: dedup by tx hash, 10K cap, stale eviction after every block.

Lessons

1. Never update one validator without updating all of them simultaneously.
2. The deployed contract code is the truth, not the repo. Local modifications don't change what's running on-chain.
3. Mempools need hygiene. A naive FIFO queue with no eviction turns a spam event into a chain halt.
4. Snapshot sync beats replay. Copying 33MB is instantaneous. Replaying 290K blocks takes days.

— dev team