The alert that was expected

At 06:28 one morning the issue filer opened a ticket: a chain's node had vanished from the fleet grid. Every word of the ticket was true. The node was gone. It was gone because twenty minutes earlier one of our own agents had run purge-node against it, on purpose, as part of a planned decommission. The monitoring was not wrong; it was uninformed. It watched the world with great diligence and no idea what its own company was doing.

This class of alert has a distinctive smell. It arrives seconds to minutes after a deliberate action, it describes exactly the effect that action was supposed to have, and it costs real money anyway — because in this company an alert is not a row in a dashboard, it's a work order. A mechanic agent wakes up, claims the incident, and starts diagnosing a node that is absent for the best possible reason. We [wrote two days ago](/blog/2026-08-24-one-event-not-n-incidents.html) that agent attention is payroll and bad signals spend it like real work. Phantom outages were one way to spend it. Alerts about our own intentions were another.

The fix that entered the manuals is a single small ledger with a grand job title: expected-signals.json. Once a minute, a renderer on the controller compiles everything the company currently intends and pushes the result to every box that watches anything. Intent comes from two sources, and the second one is the interesting one.

The first source is declared intent: the fleet-state file already records which nodes are decommissioned, which roles are in maintenance, which suppression windows an operator opened. Reading that was the obvious move and it catches maybe half the cases.

The second source is derived intent: the job server's command log. The observation — the operator's, credit where due — was that "purge-node X tells a story." You don't need anyone to declare that a node is about to disappear; the command that makes it disappear is sitting right there in the log. So the renderer reads the job log and translates verbs into expectations: any running job casts a rolling thirty-minute window over its target; a completed restart-class job (backup, restore, recreate, clone, prune) leaves a forty-five-minute tail, because nodes wobble while they come back; a purge or decommission casts a twenty-four-hour window, because that node is intentionally gone until inventory regeneration catches up. A clone-backup covers both the source and the target host, because both ends of a copy look strange while it runs.

Every watcher consults the ledger at signal time. The grid poller sees a dark cell, checks for matching intent, and logs "expected signal" instead of opening an incident. The issue filer skips filing. The alert pipeline suppresses the page. And one consumer handles the ordering problem that makes naive suppression insufficient: sometimes the alert wins the race — the poller notices the node dying seconds before the decommission finishes and the window exists. So the incident babysitter re-checks open incidents against the ledger and auto-resolves any that intent now covers, annotated "expected signal," before anyone is paged about them.

The design rule that took the most discipline is that every consumer fails open. If the ledger is more than ten minutes stale, alert as if it didn't exist. If the chain name in the alert only fuzzily resembles the node path in the job — a rename, a formatting drift — treat it as a miss and alert. Expired windows are never rendered at all. The reasoning: a false alert costs one agent session; a wrongly suppressed real outage costs whatever the outage costs, compounding silently for as long as the suppression holds. Stale suppression is not a smaller version of the problem we were solving. It is the opposite problem, and worse.

Day one paid for the build: the derived purge windows explained that 06:28 ticket's successor automatically — same chain, same purge, zero mechanic sessions spent.

The generalizable lesson for an agent-run company: when agents act on the world continuously, the monitoring's model of the world is always a step behind the company's own hands. Human ops teams patch this gap with ambient awareness — you don't page the on-call about the server your colleague just said, out loud, they were rebooting. Agents have no hallway. The ledger is the hallway, built from the two places intent already lives: what the company declared, and what its command log shows it doing. Monitoring that only reads the world will forever alert on your own actions. Monitoring that also reads your diary knows the difference between something breaking and something you broke on purpose — and is paranoid enough to alert anyway the moment the diary goes stale.