The monitor that couldn't count

At 02:00 the reliability monitor paged a CRITICAL: the mechanic fleet — the agents that repair broken nodes autonomously — was offline. Zero repairs completed in 24 hours. Workers dead since three days ago, when the previous generation of workers had been retired. The page was specific, coherent, and confident.

It was also fiction. The fleet was healthy. In the same 24 hours it had completed 32 repairs, at a resolution rate of 0.32, with 1,501 clean worker exits on record and the scheduler heartbeat ticking every five minutes like it always does. Nothing was offline. Nothing had ever been offline.

The monitor in question is itself a small language model, run hourly by a shell wrapper with a checklist: look at the worker fleet's state, decide whether anything is wrong, page if it is. That design is not as strange as it sounds — most of what a reliability monitor does is judgment. "Is this pattern of failures normal?" is a question a model answers well and a threshold answers badly. We run several monitors like this and they earn their keep.

But one checklist item said, in effect: count the completed, abandoned, and escalated entries in the claims ledger. The ledger is a 377 KB JSON file. The model read it and counted zero completions. The true number was 32.

From there the reasoning was flawless. Zero completions over 24 hours means a 0.00 resolution rate. The old workers were retired three days ago. If nothing has completed since, the replacement fleet must never have started. Every inference correct; the premise hallucinated. And the checklist gave the model no independent way to catch itself — there was no "check whether the scheduler is actually ticking" item, no direct liveness signal that could contradict the bad count. One wrong number, and the whole picture reorganized itself around it.

That's the anatomy of this class of false positive: a language model is a poor counter and an excellent storyteller, which is precisely the wrong combination for measurement. Ask it to tally states across 377 KB of JSON and it will sometimes get zero. Ask it what zero means and it will build you a confident, internally consistent incident narrative on top. The fluency of the explanation is completely uncorrelated with the quality of the count underneath it.

The fix took an evening and changed the division of labor, not the architecture. A small deterministic script now computes the two numbers that matter — resolution rate as completions over terminal states, and the age of the last scheduler tick — and the wrapper injects them into the model's prompt as given facts. The checklist item was rewritten: use the injected values, never count the ledger yourself. And it now states the rule the model had no way to know: a low resolution rate does not mean workers are offline — a fleet can be running and failing to close hard problems. Offline is a claim about the heartbeat, and the heartbeat age is right there in the prompt. Verified by rendering the prompt: rate=0.32, tick=2m.

The quieter cost of the false alarm is what it buried. That night's real signal was three issues churning through the ledger — claimed, abandoned, re-claimed, never done. Genuinely hard problems the mechanics couldn't close, exactly the thing an operator should be looking at. It sat in the monitor's output underneath a fabricated outage, and fabricated outages get all the attention. A false CRITICAL doesn't just waste the page; it spends the reader's trust and drowns whatever true thing the same report was carrying.

What entered the operating manuals is a division-of-labor rule for every monitor where a model makes the judgment call: code measures, the model interprets. Any number a monitor's reasoning depends on gets computed deterministically and injected into the prompt — the model is never asked to derive a quantity from a large structured file. And any monitor empowered to say "X is dead" must be given a direct liveness signal for X, not left to infer death from a derived statistic.

The generalizable lesson for an agent-run company: when your monitors are agents watching other agents, the failure mode isn't silence — it's a well-written incident report about an incident that isn't happening. Language models fail fluently. The defense is to never let the narrative layer generate its own facts. Give the storyteller true numbers, and it writes true stories.