The midnight genesis reset

It's 1am. The testnet has been down since yesterday. We've tried four different recovery strategies. The mempool is full of 60,000 invalid transactions. The block builder is producing empty blocks. Three validators are stuck 200,000 blocks behind the primary. And we just discovered that the bond function we've been calling doesn't match the API that's actually deployed on-chain.

This is fine.

The setup

Earlier that day, we tried to restart the testnet validators after syncing their code. Each attempt uncovered a new problem:

• Attempt 1: permissions error on the data directory. Fixed with chmod 777.
• Attempt 2: genesis hash mismatch. Peers computed a different genesis than the primary because their dependencies differed.
• Attempt 3: mempool spam. 63,000 transactions, zero getting included in blocks.
• Attempt 4: bond transactions failing silently. Cannot convert undefined to a BigInt. The staking contract API had changed locally but the on-chain contract was still the old version.

The fix

Stop everything. Wipe all data on all four nodes. Start the primary. Temporarily disable the faucet rate limit. Fund three validators one at a time. Call bond(pubKey, amount) — the old API. Wait for confirmation. Four validators. Block 624. Mempool empty. Chain advancing.

What we shipped alongside

The mempool got a proper fix: deduplication by BLAKE3 hash, a 10,000-transaction cap, and stale eviction after every block. The empty-block bug is gone.

The lesson

Testnet exists for exactly this. We found a mempool bug that would have been catastrophic on mainnet — a spam attack could have halted block production indefinitely. All of this is now documented, fixed, and deployed. It's 2am. The chain is producing blocks. Time for bed. — dev team