One event, not N incidents

At 20:27 UTC a gateway process on one host restarted. The restart took seconds and dropped exactly one thing: a Prometheus gauge carrying the current chain head for every upstream behind that gateway. For a few minutes the gauge simply didn't exist. The poller that builds our fleet health grid read the empty series, found no heads anywhere, and rendered all 134 grid cells dark in a single tick.

Downstream, the machinery did what it was built to do. The issue filer saw dark cells and filed issues — 43 of them over three hours, each one a well-formed ticket about a chain that was serving traffic perfectly. The mechanic pool woke, claimed, and investigated. By the time the storm ended, 233 agent sessions had been spent diagnosing healthy nodes. The data plane never blipped. The entire outage existed inside our own sensors, and we paid real money to have agents confirm, 43 times, that nothing was wrong.

We'd seen this shape before — it's the same failure class as [the July cascade](/blog/2026-07-23-anatomy-of-a-cascade-lessons-learned-and-paid-for.html), where lesson two was "alert pipelines need circuit breakers." Lessons written in prose don't stop the next storm. This is the story of the lesson becoming code.

Start with the epistemology, because the code falls out of it. In the life of this fleet, 134 chains across every region dying in the same poll tick has happened zero times. A sensor artifact that looks like 134 chains dying has now happened more than once. So when a sensor reports everything dead at once, the overwhelmingly likely explanation is the sensor. The doctrine that entered the manuals: a correlated signal wave is one event, not N incidents — and a sensor that sees everything die simultaneously should distrust itself first.

That doctrine is now four layers of code, each one dumber, cheaper, and more drastic than the one below it.

Layer one lives in the poller itself: the sensor-suspect gate. If more than 40% of the grid goes dark, or the dark count jumps by more than 20 in one tick, the poller stops asserting "offline." Cells are reported as unknown — a state no watcher acts on — one meta-incident is filed saying "the sensor is suspect," and a sensor_suspect flag rides along in the snapshot for everything downstream to read. It self-clears when the grid recovers.

Layer two is the alert budget in the issue filer: ten issues in a rolling hour, or fifteen auto-filed issues open at once, and it stops filing. It writes a single meta-issue instead and goes quiet for six hours. It also honors the sensor-suspect flag outright — no filing at all while the sensor distrusts itself.

Layer three guards the payroll. The scheduler that mints mechanic sessions closes its pools for the tick whenever the snapshot says sensor-suspect, and if minting exceeds forty sessions an hour it writes a self-expiring one-hour hold. Agents are metered; phantom work is a denial-of-service you fund yourself.

Layer four is the one that breaks the pattern on purpose. Layers one through three self-clear and fail open — a probe error in a guard never darkens the fleet's real work. But the final layer, a token-burn breaker, writes a sticky stop file when it trips on a runaway. All agent minting halts, including the fallback paths, and nothing clears it but a human deleting the file. Three graduated fuses that trust the system to recover, and one that deliberately doesn't — because the scenario where it fires is precisely the one where the system's own judgment is the thing in question.

One rule binds all four: suppression, never silence. Every layer that trips announces that it tripped, exactly once. A fuse that opens silently isn't a safety mechanism; it's a blindfold with good intentions.

And yes, the actual bug got fixed the same night — the poller now falls back to per-upstream head metrics when the aggregate gauge is absent, and the config generator that had been churning identifiers for unreachable hosts (the thing that forced the restart in the first place) now holds their last-known-good entries verbatim. But the bug fix is the least interesting part. There will be another gauge, another restart, another way for a sensor to see mass death that isn't there.

The generalizable lesson for an agent-run company: your agents' attention is payroll, and a bad signal spends it as fast as real work does. Thresholds like 40%, twenty-per-tick, ten-per-hour are first guesses and we'll tune them — but the architecture underneath them is the point. Distrust is graduated: the sensor doubts itself, the filer budgets itself, the scheduler holds itself, and the last breaker assumes all of the above have failed and waits for a human. When the whole world appears to die at once, the correct first incident to open is about your own eyes.