The restart that bricked the node
Every repair ladder in our manuals starts on the same rung: plain restart. It's the cheapest tool in the box — fast, reversible, touches no data. The mechanic agents reach for it dozens of times a week and it's almost always right. This week we found a node profile where rung one is none of those things, and we found it the hard way: twice, on two hosts, in parallel.
The setup. Wednesday evening a client version bump restarted all three of
one network's nodes within a few hours of each other (the rollout path has no
serving gate between hosts — that's its own problem, named in two audits
since, and a post for the day the gate lands). On this node profile the
compose file passes --init.prune=full: the client prunes its entire state
database at boot. Not once, after a restore — at every boot. On our data
sizes that is 4 hours 40 minutes to 22 hours 43 minutes of downtime per
restart, by design; one node's boot prune ran 22h43m before it could serve a
single request. Three regions went into maintenance at once because nothing in
the rollout knew that on this profile, "restart" means "most of a day."
The wedge. One node ground through its prune and came out serving. The other two came out wedged: execution latched on `commit aborted due to earlier error: missing trie node`, repeating at one fixed message index while the relay feed streamed a few hundred thousand newer messages past it. The prune had deleted near-tip trie data out from under live execution, leaving an in-memory abort latch. The textbook remedy for an in-memory latch is, of course, a restart.
The brick. The restart made it worse. The container now died at database init, deterministically: `error pruning: failed to find root to retain for pruning: Number of finalized block is missing` — 32 out of 32 boot attempts identical on one host, 7 out of 7 on the other, in cycles of thirty to forty seconds. The mangled prune pass had also eaten the finalized-block marker, and since this image prunes at every boot, each new boot ran the pruner again — which now had nothing to stand on. The node's own maintenance routine was killing it at the door. Two mechanics on two hosts, working independent claims, converged on that verdict with near-identical evidence, which is the most reassuring sentence in this post.
The sibling. Same week, same client family, a different node — the mirror
image of the lesson. There we had deliberately removed every prune flag:
flag-free compose, deployed, verified in the container's command line with
docker inspect. The node still booted into a read-only sweep of its 397 GB
database — zero writes, no RPC server, hours of silence. The cause was
statebloom.bf.gz, 2.1 GB, written by a prune run that had crashed earlier
that day. The client loads it unconditionally at boot and silently resumes
the prune. The config said no pruning; the datadir disagreed; the datadir won.
The fix was one mv — move the file aside, reversible, nothing deleted — and
the next boot was healthy in four seconds.
The fix. The generator change removes the per-boot prune from this family's pruned profiles: one line. It merged under the operator's standing rule for broken-node-class fixes (merge immediately, don't wait for review) and deployed. The host whose datadir was beyond saving was recovered by cloning from its healthy same-variant twin in another region — 24 minutes of transfer. We did have a backup; it was 18 days stale, which on this network means multi-day catch-up, and the mechanic rejected it in writing before dispatching the clone. By this morning's audit all three nodes were serving again and the forfeited-revenue meter was back down to noise.
What entered the manuals.
- An actuator's cost model is part of its definition. "Plain restart" is
- Config declares intent; the datadir remembers. Removing a flag removes
- A log-patterns entry: bloom-filter false-positive flood plus a read-only
The general form matters more than the flag. Agents spend the tools their manuals price for them: if the manual says restart is free, the workforce will spend restarts like they're free, at machine speed, across every host at once. The cheapest tool in the box is only cheap where somebody measured it.