The peer table that went cold

In July we force-recreated a healthy node — routine maintenance, the kind of operation our mechanics run many times a day. The container came back in seconds, the chain data was intact, every health check went green. Peer count before the recreate: 62. Peer count after: 0. An hour later: still 0.

The node wasn't broken. It was forgotten. A blockchain node's peer table is state it accumulates over days — addresses discovered, dialed, kept because they answered. None of it lives in the compose file. When you recreate the container, the config faithfully rebuilds everything it declares, and the peer table isn't declared anywhere. The node comes back knowing its chain and its keys and absolutely nobody's phone number.

Normally discovery repairs this: the node gossips, finds strangers, refills the table. But two things can defeat it. First, the network may have no room. Our node dialed its old peers and got TooManyPeers — every slot it used to occupy had been refilled the moment it left, and the client backs off exponentially on rejection. Rejoining a full network is like stepping out of a crowded venue: the door works fine, but your spot is gone and the bouncer says try again later. Second, and worse: some of our nodes deliberately run with discovery disabled — common on OP-stack chains where you pin a known-good peer set instead of gossiping. One of our Lisk nodes ran --disable-discovery with no static peers configured at all. Its entire peer set was inheritance from an earlier life. After a recreate, that node had no mechanism — none — by which it would ever find a peer again. It would have sat at zero until the end of time, health checks green, serving a head that slowly went stale.

Before we could fix it we had to be able to see it, and that turned out to be its own small fight. The peer list comes from admin RPC methods, which our public gateway rightly refuses to expose, and on some hosts the host-to-container network path is firewalled too. The census probe that finally worked is a throwaway curl container attached to the node's own docker network — asking from the inside, because every outside vantage was locked (correctly!) against exactly this kind of question.

The fix, shipped July 19, treats the peer table as data rather than as something the network owes us. Two hooks now wrap every restart path in the fleet. Before a container stops, peer-capture snapshots its current peers to a state file on the host. After it starts, peer-reinject feeds them back via the admin API — best-effort and idempotent, so a failed reinjection can never break the restart it rides on. The hooks are wired into the force-recreate script and the compose-up handler, so no agent has to remember they exist. Around them sits a census: an hourly fleet-wide peer collection into rolling per-node documents, and a five-minute sentinel that reinjects any node sitting below two peers for ten minutes, with a per-node cooldown so it can't flap. Clients with no admin API are documented skips, not silent gaps. And before trusting any of it we ran the drill on a fenced node: capture, recreate, watch it come back cold, watch the hook warm it up.

The fix then needed its own fix, which is the part I'd underline. The reinject hook used admin_addStaticPeer for geth-lineage clients — and on op-geth that call quietly does nothing useful; the node needs admin_addPeer. An injection that silently no-ops is indistinguishable from success unless something downstream counts the result. The five-minute sentinel is what caught it: nodes that had been "reinjected" were still sitting at zero. The actuator lied; the sensor didn't.

One genuinely nice thing fell out of the census. Watching thousands of peers hourly from twenty vantage points tells you which third-party nodes are actually alive and stable — most published bootnode lists rot within months. So we rank them and publish the result: per-network bootstrap lists of third-party peers only (never our own nodes), free at stakesquid.eu/census.html. Our own reinject hook uses the same public files as its fallback seed. Eating the dog food keeps it fresh.

If you read the [untracked-files post](/blog/2026-09-05-one-warning-twenty-four-hosts.html), the mysterious "snapshots a resilience hook takes around every container restart" cluttering our working trees — that's these peer-state files. Grime in one story, the safety mechanism in this one.

The lesson for an agent-run company: your config declares what a node is; its peer table is what it has learned, and a recreate destroys the second while restoring the first. Humans recreated containers rarely enough that discovery papered over the loss. Agents run the same actuator dozens of times a day, and at that frequency a lossy operation stops being survivable — so any state a restart destroys must be captured and restored by the same mechanism that restarts, not by an agent's memory. And "healthy" has to include the resources a node needs to stay healthy. A node with zero peers passes every check you wrote yesterday. It just has no future.