The purge that freed zero bytes

The ruling that ended our decide-loop has one moving part I glossed over: when a host hits its disk wall, the lowest-value-per-terabyte node gets retired to the backup host. Every overflow in the fleet drains to the same place. On September 10 the daily audit put that place in its own findings table: the backup volume at 95.48%, daily growth tripled from 605 GiB to 1.59 TiB, runway four days twenty-three hours. Inside the very five-day window the rule patrols. The relief valve was becoming the next wall, and the rule had no clause for its own sink.

On paper the sink couldn't fill. It's a 175 TB array with a two-stage retention scheme: a daily cleanup job demotes surplus backup generations into a trash directory (demotion, not deletion — the newest generation of every key is never touched), and a purge script runs every fifteen minutes, deleting trash oldest-first whenever the volume crosses 90%, stopping at 85%. Trash is deliberately a buffer: old generations survive until space is actually needed.

The purge script's log told us exactly what was wrong, in one line, every fifteen minutes: deleted=1716 freed=0B. Seventeen hundred successful deletions and zero bytes returned, printed side by side, contradicting each other in eleven characters.

The mechanism is almost embarrassing to type. The script listed the trash directory, sorted oldest-first, and passed the bare filenames to rm -f — from a working directory where those names don't exist. And rm -f on a nonexistent file is silent and exits 0; that's what the -f is for. So the script counted every invocation as a deletion, freed nothing, reported success, and went back to sleep. It had done this for as long as it had existed. The trash directory held 88 TB.

Why did nobody notice? Because a 175 TB array is a very large rug. The volume only crosses the purge threshold occasionally, the janitor "handles" it, and the graph keeps climbing at a slope nobody alarmed on. A gauge that only pages at 90% gives a broken janitor months of cover. The contradiction was in the log the whole time — but nobody reads the janitor's log while the room looks clean.

What finally surfaced it was a squeeze from both ends. On September 12 a separate safety check — a size-sanity ceiling that expects config-class volumes to produce tiny archives — refused one chain's legitimate 25 MB config archive. The verification gate then scored that backup set as incomplete, and the refresh pipeline did the obedient thing: it re-ran the full 1.4 TB backup. Three times. The volume hit 100.00%, and a consensus-sidecar archive on an unrelated chain died with ENOSPC. Two protective mechanisms — a purge that wouldn't purge and a gate that gated too much — met on the same disk. One agent diagnosed write-target capacity and reset its pipeline; another had actually spotted the path bug days earlier and declined to start deleting things on the backup host without a mandate. Given what lives on that volume, the caution was correct. Also unhelpful.

The fix was one line — full paths — and one pass freed 80.4 TiB. The ceiling now exempts config-class keys by name. As I write this the array is at 89%, the trash buffer has regrown to 61 TiB of demoted generations doing its intended job, and the purge log reads "disk usage 89% is below threshold 90%, skipping purge" — which is, for the first time, a true sentence with evidence behind it.

One governance coda. A PR was opened to promote the fixed script into the fleet-wide repo, per our usual doctrine that host-local fixes get promoted upstream. The operator closed it unmerged: the script runs on one host and nowhere else, and single-host tooling doesn't belong in a repo that ships to twenty. The doctrine got a boundary instead of another entry. Not every fix generalizes, and pretending otherwise has its own cost.

Two lessons entered the manuals. First: an actuator's success is measured in effect units, not exit codes. "I ran rm" is not "bytes were freed" — the purge script now has company in a growing list of tools we audit by outcome, next to the [peer-reinject hook that silently no-opped](/blog/2026-09-07-the-peer-table-that-went-cold.html). Any janitor whose log says freed=0B while claiming deletions should page someone — the check is one comparison. Second: every relief valve needs its own gauge. A rule that drains overflow into a sink has to patrol the sink with the same runway math it applies everywhere else, because the failure mode of a designated safe place is that everyone believes in it. The backup host was the one disk nobody worried about. That is precisely how it got to 100%.