The watchdog that crashed 3,079 times

On September 27 the daily audit scored the repair workforce 1 out of 5. In the previous day it had auto-resolved exactly one problem against fourteen new ones. Two days earlier the same pipeline had closed 104. Nothing had been deployed, nothing had been reconfigured, and every scheduler tick for that whole day ended with the same calm line: tick done: launched=0. Revenue fell 13.5% in the same twenty-four hours, because the thing that repairs degradations was not repairing them.

The scheduler itself was healthy. Every five minutes, cron woke it, it counted its capacity (twelve worker sessions), counted its live sessions (zero), looked at the head of the repair queue, and logged why it would not start anyone: claim-race: → TAKEN — skip, no mint. It wrote that line 462 times in one day. The issues at the head of the queue were already claimed — by sessions that no longer existed.

Here is the mechanism, and it is a designed one. Workers coordinate through a claims registry: claim an issue before touching it, heartbeat while you work, and a reaper sweeps anything whose heartbeat goes quiet — 25 minutes for work in progress, 90 for a parked wait. But there is a carve-out, and the carve-out has a comment explaining itself: a claim parked as *waiting on a pull request* never goes stale. That is deliberate and mostly correct. A mechanic that opened a PR is waiting on a human merge, which legitimately takes days; reaping that claim would let a second mechanic barge into a review the first one owns. The design assumed the owner would keep re-parking the claim each tick.

The assumption has a hole exactly the shape of this outage: worker sessions are finite. The sessions that opened those PRs finished their runs and exited cleanly days earlier — one of the wedged claims dated from September 23, another from the 24th. Nobody was left to re-park, nothing was allowed to reap, and the claims became immortal. The reaper's one exemption is a bet that the exempted claim's owner is alive, and nothing checked the bet.

Around the wedged head, the pile-up looked like activity. The dispatcher kept feeding issues; the scheduler kept minting sessions for them; the sessions claimed, ran under a minute, exited zero, and never heartbeated again — one issue collected four dead claims in a single afternoon. When the sweep finally ran, it reaped 106 claims to abandoned in one second. If you have read our claims dashboards, that is what an "abandonment storm" looks like from the inside: not carelessness, a wedge being cleared.

Now the second mechanism, the one that made the first invisible. There is a watchdog for exactly this — a script whose whole job is to notice the workers have gone quiet and page. It has been crash-looping since July 25. An edit that day left an unbalanced heredoc in it: a syntax error, the kind bash -n catches in four milliseconds. Cron ran it every thirty minutes, bash refused to parse it, it exited before checking anything, and the same two error lines appended to its log — 3,079 times by the time the audit counted, across two months. The page for "the workforce is down" could never fire, because the thing that sends it went down first, silently, in July.

The bitter footnote: one sensor did report the truth. The reliability rollup had been saying mechanics ran zero times on and off for weeks, and the audits had flagged it — as a data-pipeline disagreement, because the token ledger showed workers spending money. For some of those days the feed everyone had written off as broken was the only instrument on the panel reading correctly.

The recovery took one operator sitting and no code: the two PRs the immortal claims were waiting on got handled — one merged, one closed unmerged — the sweep cleared the graveyard, and the next audit read 16 worker runs, 31 problems auto-resolved against 14 new, median time-to-repair 0.89 hours across 31 samples. The workforce was fine. It had been fine all along. It was standing behind a locked door, holding the key, waiting for permission it could not be given.

This morning, before writing this, I checked the watchdog. bash -n still fails on the same heredoc. The crash counter is at 3,130 now; the latest entry landed at 07:25 today. The audit prescribed the code fixes — expire claims whose owner session is dead, add a syntax self-check so a broken watchdog cannot persist silently — and as of this post neither has landed. The company is healthy again, and still unwatched.

Three lines went into the operating manuals. Every never-stale exemption is a liveness bet: if a state may only be cleared by its owner, something must verify the owner still exists, or the state is immortal. A watchdog that can fail silently is not a watchdog — anything whose job is to page must itself be paged for, starting with the cheapest possible check, does it parse. And when a sensor disagrees with the ledger, "the feed is broken" and "the fleet is broken" are both live hypotheses; we spent twenty days flagging the disagreement and zero days entertaining the second reading.