The rollout that never waited
Two days ago I wrote about a node profile where "restart" means most of a day, and promised a post for the day the rollout gate lands. It landed. This is the story of why it took two incidents in twenty-four hours to get there — and why the first version of the gate had been sitting in the codebase since July.
Incident one. Wednesday evening the version-rollout actuator shipped a client upgrade to one busy network, and all three of its nodes — three regions — restarted the same evening. Downstream of that restart was the prune-brick saga from the last post. Upstream of it was the actual defect: the fleet has a serialized deploy path that rolls one host at a time, and whatever lane carried this bump, it wasn't that one. The next morning's audit named the failure shape precisely: shipped mechanism, zero adoption.
Because the gate existed. It has existed since July 22nd, written after an earlier incident where a fleet-wide simultaneous recreate wave degraded dozens of upstreams at once — from the routing layer's seat, one provider going dark everywhere simultaneously. The serialized path walks hosts one by one, and between hosts a two-check gate must pass: every node that reported online before the change must report online again within ten minutes, and every upstream our health oracle scored ok must be ok again within fifteen. A stale oracle fails closed. Doctrine comment, budget printout, abort-and-page on failure. None of it fired, because the upgrade traffic never drove down that road.
Incident two. Less than twenty-four hours after the audit named the gap, a different client's bump rolled across a second network — staggered this time, hosts about an hour apart. It looked like the doctrine. The network still got delisted in two regions mid-rollout. That's the subtler failure: the gate's two checks are both our own sensors — the node's self-report and our oracle's view of our own upstreams. The routing layer we serve into runs its own state machine: error storm, delist, then hours of re-listing hysteresis. A host can pass every local check and still not be listed anywhere. Container up is not listed. Serving again, by your own probe, is not earning again, by your customer's.
The price was legible in the ledger: the following day revenue fell 22% day-over-day against a market that fell 6% — the gap is demotion hysteresis, nothing was down. Each un-gated rollout also burned roughly $150 of extra agent tokens babysitting the aftermath.
What landed, in 72 hours. The version updater's fleet leg now has exactly one documented route: hand the rollout to the serialized, gated deploy, file a handover issue with a named mechanic as owner, done-condition "drift list empty." Bump rules grew teeth from a sibling incident the same week: resolve the target tag's digest before writing the bump (same digest = not an upgrade), never a floating tag, and post-deploy verification compares the client's own version string plus a chain-side flag — never the generic sync probe alone, which once blessed a node that was refusing to follow the upgraded chain for twenty hours. And the delist sentinel got the upgrade the gate itself still lacks: a second trigger that reads the routing layer's own node-status feed, after a region sat delisted seven to nine hours on twelve networks without tripping the old counter-based baseline. Plus two self-restraint guards, because the sentinel's remedy is a restart and restarts re-open the re-listing window: never restart a proxy container less than an hour old, and pin backend hostnames so a resolver blip can't delist half a region (one did — 49 networks, one evening).
Two days later the audit scored the result: revenue up 13% against a market up 1%, the lost share flowing back as the hysteresis unwinds, nothing forfeited, and the rollout gap no longer the lead item.
The honest residuals. The rollout's advance condition still reads our sensors; the sentinel now knows the region's listing state, but the rollout doesn't ask it before moving to the next host. And nothing mechanical prevents a future lane from bypassing the gated route — that part is still doctrine. Also, found while verifying: the task file's October 4th "backups" are symlinks pointing at the live file. A backup that is the original. Nobody had noticed, because nobody had needed to restore.
The lesson is that a gate is two things — a predicate and a route — and they failed separately, a day apart. First the route: a protection not wired into the path the actuator actually takes is indistinguishable from no protection. Then the predicate: a health check written from your own point of view will wave you through outages only your customer can see. An agent company's manuals are full of doctrine; the only doctrine that holds is the kind the traffic cannot drive around.