Three incidents in two weeks. A 23-hour chain halt from a closed laptop. A code drift divergence that stuck three validators 200,000 blocks behind. A mempool bug that turned transaction spam into empty blocks.
Every single one of these would have been worse on mainnet. Some of them would have been catastrophic. That's the point.
The laptop incident exposed that our original consensus design assumed all validators stay online forever. When the Mac validator went offline (lid closed), the committee couldn't reach quorum. Fix: the liveness-aware committee now dynamically shrinks to the set of validators who proposed at least one of the last 30 blocks. Home validators can come and go.
The code drift incident exposed that deploying code to one validator without updating the others causes state divergence. Fix: an atomic deploy script is on the roadmap — one command pushes identical built artifacts to all validators and restarts them simultaneously.
The mempool bug exposed that a naive FIFO queue with no deduplication turns any source of duplicate transactions into a chain halt. Fix: the mempool now deduplicates by hash, has a 10K size cap, and evicts stale transactions after every block.
We also learned practical things about running a multi-validator testnet. pnpm symlinks don't travel between machines. Genesis is permanent — the system contracts deployed at genesis are baked into chain state. Snapshot sync beats replay: copying a 33MB state directory takes seconds; replaying 290K blocks takes days.
Atomic deploy script. Code version hash on /metadata so validators can compare versions. Auto-updater for the Operator app. The mempool is now production-grade. The chain is back up — four validators across three continents, producing 2-second blocks. The testnet did its job. — dev team
Join the launch list and we'll send you a note whenever there's a new devlog entry, a research drop, or a real milestone.