The flag that vanished in an RC

This one is from the archive — June 19, before this blog existed — and we keep coming back to it internally because it is the cleanest end-to-end run the repair loop has ever produced. One container down, one flag to blame, and a diagnose → issue → PR → merge → deploy chain that closed in ninety minutes with agents at every step except one.

It starts with a dismissal. At 06:50 the mechanic's fleet survey saw our Aztec testnet node reporting error: could not parse rollup version and wrote it off: "status-script parse limitations, not node-down." That was not lazy. The fleet's health checks probe dozens of client implementations, several of which answer standard questions in nonstandard ways, and we had recently burned days on a check that couldn't parse a perfectly healthy node's response. An error message about parsing pattern-matches to a broken probe, not a broken node. The survey moved on.

An hour later the next survey pass found the same error still there, and this time the mechanic claimed it — but note what it did next. Still suspecting a probe bug, it didn't restart anything and didn't file anything. It dispatched a diagnosis job to establish ground truth on the host: real outage, or check-health false positive?

The verdict came back at 08:20 and the error message everyone was primed to distrust turned out to be true. The container was crash-looping — 71 restarts and counting — on the 5.0.0-rc.1 image, because the generated compose file still passed --archiver on the command line. Aztec's 5.0 release had removed that flag; the archiver now runs inside --node. The client printed a usage error and exited, forever. And the unparseable rollup version? The probe was trying to parse a version out of a backend that wasn't answering. The sensor was fine. The node was dead.

Then the chain ran. 08:20: the mechanic opened issue #59 against the compose generator with the exact fix spec — drop the flag, command becomes [--node, --network=testnet] — and dispatched an implementation job on a branch. 08:26: the implementation agent landed the change, two files, one line of it load-bearing. 08:50: the mechanic verified the build, opened PR #60, and posted a domain-context comment for the human who would review it: this is a hard outage, not cosmetic; blast radius is one testnet on one host; and — because that same host had a concurrent disk-critical claim open — a caution about deploy timing. 09:01: the operator merged. 09:20: the mechanic saw the merge, dispatched the deploy, and parked. By the next survey the node was off the error board and the same mechanic was already diagnosing an unrelated problem two hosts away. (The 09:50 tick, in the interest of full disclosure, died on an API rate limit and contributed nothing.)

First sighting to recovery, about three hours. Claim to deploy, ninety minutes. The one human action in the chain took eleven: a merge, arriving with the diagnosis, the risk assessment, and the deploy-timing caution already attached.

The part worth generalizing isn't the speed. It's why the node broke, and why the loop caught it anyway.

It broke without anyone making a mistake. The compose command had been correct for months; the client's new release candidate deleted a flag; our config became fatal retroactively. Testnets that track release candidates redeploy under you — this same network had already forced a genesis-breaking redeploy months earlier, and three weeks after this fix it wedged again on rc.2, whose verification-key tree didn't match the freshly upgraded rollup. Another one-line bump. Running the long tail means being on that treadmill for every chain that's still becoming itself, and "we changed nothing" is not a defense the treadmill accepts.

And it was caught because the loop refused to resolve ambiguity by pattern. The 06:50 dismissal and the 08:20 verdict looked at the same error string and reached opposite conclusions; the difference was that the second one paid for a measurement. An error that resembles a known false positive is exactly the costume a real outage will eventually wear — the more often a signal is noise, the more safely a fault can hide in it. Suspicion was correct both times. Dismissal was only safe once. The rule that survives: you can doubt a sensor all you like, but you retire the doubt with a probe, not a shrug.