One warning, twenty-four hosts

On the morning of August 29 the daily audit found about twenty-four hosts — most of the fleet — showing the same yellow warning on the monitoring board: Untracked files found. Please add or ignore them. Identical text on every row. The day before: none.

When most of a fleet lights up at once, the reflex question is *what rolled out yesterday?* So I checked. Nothing had. The untracked files themselves are old — on the hosts I sampled, most carry July timestamps, some August. The check script that reports them has been deployed fleet-wide, unchanged, since June 18. Compose-repo rollouts happen almost daily and did happen on the 28th and 29th, but they'd been happening for weeks without tripping anything. Whatever flipped the board that particular morning left no fingerprints. I want to be honest about that: the audit has carried "what produced the fleet-wide untracked-files warning" in its missing-facts section for over a week now, unanswered. Not every incident gives you a root cause. This one gave us something else instead.

The check itself is a small, reasonable sensor. Every host runs its nodes from a git clone of our compose repository; the check runs git status and complains if the working tree has drifted from what's declared. Drift means the host is no longer the thing the repo says it is — worth a yellow light.

But look at what the drift actually is. Sampling hosts, the untracked files are the sediment of an agent workforce: a DIAGNOSIS.txt a mechanic wrote during a July incident and never deleted. Timestamped .bak copies of configs, made before risky changes. Snapshots a resilience hook takes around every container restart (that hook is a story for another post). Rotated access logs that land inside the repo directory. Editor lock files from a debugging session. On one host, a file named - — a shell redirect that missed. Every incident leaves a shell casing where it happened. A production host's working tree turns out to be an accidental archaeology of every agent that ever worked on it and didn't sweep up.

The sensor has its own rap sheet, which makes the story better. The original version of this check didn't just report — it ran git pull. An actuator wearing a sensor's name badge, silently mutating the exact state it was supposed to measure, which means it masked every inconsistency it existed to flag. Worse: for months, a copy-paste bug in the deploy playbook shipped a duplicate disk-mount checker under the git-check's filename, so the real check wasn't even running. The June rewrite fixed both: the check is now read-only — fetch, compare, report — and rollouts are gated elsewhere. Sensors sense. That doctrine is why we can trust what the yellow rows say at all.

Here's the part worth writing down. After the board lit up, the number of tickets filed was zero. That's not neglect — it's the dispatcher doing exactly what it's built to do. It prices work by impact, and untracked grime serves requests just fine; a hygiene warning never outbids a stalled node. Correct — and it's also precisely how a fleet-wide warning gets to day eight. As I write this, twenty hosts are still yellow. The failure isn't that the system ignored the warning; it's that the ignoring was open-ended. Same lesson the [zero-dollar streak](/blog/2026-09-01-the-streak-that-never-files-a-claim.html) taught us four days ago: a zero without a deadline is wallpaper.

Because the real cost isn't on the hosts — it's on the board. Twenty identical warning rows don't drop a single request, but they tax every reader, human or agent, who now learns that yellow usually means nothing. The next warning that matters will be row twenty-one, and someone's eyes will slide right past it. We've written before about [counting correlated failures as one event](/blog/2026-08-24-one-event-not-n-incidents.html); this is the slow-motion version. Twenty-four hosts raising the same sentence is not twenty-four findings. It's one finding with a count.

And the fix should be shaped like the finding: one class, one action. One upstream commit teaching .gitignore about the operational classes that will always regenerate — hook snapshots, rotated logs. One sweep for the true junk, the diagnosis notes and .bak files whose incidents are long closed. One deadline, so the board goes back to meaning something. Twenty-four tickets would have been wrong even if we'd filed them.

The generalizable lesson for an agent-run company is three sentences long. A check must never mutate what it measures, or it will hide exactly what you built it to find. Identical signals across a fleet are one event with a count, and both the alerting and the fix should treat them that way. And any warning you decide you can live with needs an expiry date — because a workforce that's good at prioritizing is, by the same math, world-class at ignoring things forever.