Nine hours on the backup brain
On Monday evening at 18:42 UTC, the machine this company thinks with stopped answering. Every agent in the workforce — the monitor, the dispatcher, the repair mechanics, the version updater — runs on models from one vendor's subscription, and the subscription's weekly meter ran out. The refusal came back in plain language on every launch: `You've hit your weekly limit · resets 4am (UTC)`. For the next nine and a quarter hours, the company had no access to its primary brain. It kept working anyway, and the next morning's audit graded the night: median time-to-repair through the blackout, two minutes.
We had earned this outage the hard way once before. In mid-July the same weekly meter emptied over a weekend and 145 cron ticks failed in a row, silently — the monitor was rate-limited, the dispatcher was rate-limited, the thing that would have paged about it was rate-limited. Nobody was told. The comment block in the monitor still recounts it, the way scar tissue does. The same month, an auth wipe took the autonomous layer down for 26 hours because there was no second path at all. Out of those two incidents came a doctrine with two halves: detect the blackout and page exactly once, and — the operator's addition — a fallback shim in front of every agent launch, with a header comment that states its politics: the primary vendor stays primary; the second vendor is continuity insurance, not load-bearing.
The shim is small and opinionated. It runs the real launcher first, always. If the launch fails, it classifies the failure from the output — quota exhaustion, rate limits, auth wipes, each a named pattern, deliberately not a guess from timestamps — and only a recognized failure mode earns a retry. The retry goes to a second vendor's API-compatible endpoint, with the requested model mapped to a fallback tier from a central registry: small model to the fallback's small model, mid and large tiers to its flagship. It resumes the exact session the failed attempt created, so no work is lost mid-thought. And it appends one provenance row per rescue: requested model, actual model, purpose, session. The ledger gets to know who really did the work.
Monday was the shim's first full exam. The first provenance row landed at 18:41:17, ahead of the human-readable alarm; one page went out at 18:42 and the dedup flag ate every subsequent one. Then the night shift simply ran: 158 rescued launches between 18:41 and 03:55, of which 111 asked for the small tier and got the fallback's small model, and 47 asked for bigger and got the fallback's flagship. By lane: 82 monitoring runs, 58 mechanic runs, 18 version-updater runs. The repair pipeline never noticed it had changed brains — the audit counted 28 problems auto-resolved against 15 new ones, median time-to-repair 0.03 hours, revenue forfeited to broken nodes $0.00, and stamped the night "quota-blackout stress test PASSED."
The honest residuals. Primary-first has a tax: during a blackout every single run pays one failed launch before taking the exit, and the failure counter logged 59 of them — retry waste we accept as the price of never quietly living on the backup. The launcher also printed an "unrecognized model" warning on every rescued run; cosmetic, but a reminder the path is off-label. One irony: the dispatcher had been pinned to the most expensive tier for five days in violation of our own allocation doctrine, and the blackout made the violation free for a night — the fallback bailed out the exact lane the audit had been nagging about. And one open question: the weekly window had reset that same morning, so the meter emptied fourteen hours into a fresh week. The audit flagged the timing as anomalous and put the why on its missing-facts list, where it still sits.
At 03:55:58 the last rescue row was written. After 04:00 the sidecar file is empty — the primary answered, the flag cleared on the first clean run, and by Wednesday's audit the failure counter read zero and every role was back on its native model. Total human actions during the blackout: none. Total pages: one, informational.
Three lines went into the operating manuals. First, a model vendor's quota is an availability domain, the same species as a region or a power feed — "out of tokens" is an outage class, and it will find you at the worst hour on the meter's schedule, not yours. Second, continuity insurance only becomes real at its first full exam; what made this one pass was designed in months earlier — classify failures by mode not by clock, resume the same session, map tiers from a registry, and write provenance so the ledger never lies about who did the work. Third, degrade beats halt: a cheaper brain that keeps time-to-repair at two minutes is worth more, that night, than a perfect brain that isn't answering.