Right version, wrong address
At 03:35 this morning the pager fired: flow collapse on one of the busiest networks we serve. Both replicas — deliberately on two separate hosts, so that one bad machine can never take the network down — had gone dark to the routing layer. Roughly 2,300 requests a minute were suddenly going elsewhere. One upstream was cut for 44 minutes, the other for 185.
Both machines were fine. What failed sat upstream of both of them, in the one place they share: the configuration repo. Two agents had each done something locally defensible there the previous afternoon, and the composition took the network down at three in the morning.
The bump
The network had a scheduled hardfork. The client release we had pinned stops at the fork block; the release that continues past it is v1.4.2. Keeping pins current through hardforks is exactly what our version-updater lane exists for, and at 12:48 the previous afternoon it did its job: read the release notes, bumped the pin to v1.4.2, said so in the commit message. The right version.What the release notes didn't say is that the project had moved its container image to a new repository. The old repository — the one our config pointed at — had quietly stopped receiving releases at v1.2.0. v1.4.2 existed, was published, and worked; it just lived at a different address. The updater wrote the right version into a dead repository, and the hosts had nothing new to pull.
The revert
At 15:54 a second run was dispatched to clean up, carrying a spec that said: the v1.4.2 image tag does not exist in the registry — revert the bump.That statement is true. In the abandoned repository, the tag genuinely does not exist. The run executed the revert on the spec's word — zero probes of its own — and the merge lane merged it three hours later. The fleet was now pinned, with full confidence and a clean audit trail, to the one release known not to survive the fork.
The revert had a second defect that the nightly lesson pass caught: the pre-state wasn't one version, it was three — a global default at v1.1.0, the testnet at v1.2.0, mainnet at v1.1.1 — and the revert flattened all three to v1.1.1. It had re-stated the spec's prose ("go back to v1.1.1") instead of inverting the named commit. A revert that changes pins nobody asked it to touch, while its summary calls the result "the known-good version before the bump," is stating a falsehood politely.
The night
The fork activated; the pinned release stopped following the chain; both replicas fell behind together, because both obey the same repo. At 03:00 the routing layer gave up on the first upstream, then the second. The delist sentinel auto-restarted the gateway — the right reflex for the common case, the wrong disease tonight. At 03:35 the flow-collapse sentinel paged.The diagnosis run that followed found what two runs had missed, and its fix commit at 03:54 says it plainly: the earlier bump *"failed with the false reason that v1.4.2 does not exist in the registry — the tag exists in a different repository."* The fix pointed the config at the project's new image repository. The operator — awake at four in the morning — merged it at 03:57, twenty-two minutes after the page: the one human step. Regeneration and deploy were verified on both hosts by 04:47, and by 05:25 the flattened defaults were repaired so the wrong pins can't resurface on the next regenerate.
Residuals, honestly: our forfeited-revenue sensor scored the day $0 — it is streak-based and blind to transient outages, so the pager was the only sensor that priced this night correctly. And a testnet follow-up was landed on main directly, outside the PR flow, flagged in-channel.
What entered the manuals
Three entries. An absent tag is only "unpublished" after a repository move has been ruled out — the registry probe is now inlined into the doctrine file, next to the rule it guards. A revert is the inverse of the named commit, never a re-statement of its prose: mirror the per-rung pre-state fromgit show,
and a revert of a hardfork bump re-breaks the chain at that fork, so the
summary must carry the fork block — the caller deploys knowingly or not at
all. And the audit named a new rising constraint: version bumps on our busiest
images need a staged rollout, because two replicas on two hosts is zero
redundancy against the one lane that configures both.The generalizable lesson is the second mechanism. In an agent pipeline, a true statement about the wrong namespace is more dangerous than a false one. A false claim trips over the next agent's checks; a true claim gets inherited downstream as if someone had verified it — the revert run treated "the tag does not exist" as a probe result when it was only a sentence in its inbox. Facts come with coordinates. A verification that doesn't say where it looked hasn't verified anything yet.