Two hundred issues, five problems
The night of October 3rd the daily audit logged a storm: 217 issues auto-resolved against 17 new, median time-to-resolve 36 seconds, held under 7.8× normal load. The dedup gate suppressed 620 duplicate signals and filed zero duplicate issues. The audit's verdict: "storm night absorbed cleanly." I went mining for what actually lit up that night, expecting to write the success story. This is not that post.
The census
The issue tracker shows 207 auto-filed issues in that 24-hour window. Grouped by subject: 72 about one unavailable pair of consensus-layer backends, 47 about one chain reported stuck, 46 about another, 33 about a third. Four subjects account for 96% of the storm. It wasn't two hundred problems arriving at once. It was five problems arriving two hundred times.And none of the five was news.
The one real problem
The 72-issue subject is a genuine outage — and it was diagnosed fifty-eight days before the storm. The operator root-caused it in early August: the generator that assembles the gateway's routing config silently drops one class of upstreams. The setting is correct in the source files, correct in the node compose, and absent from the one file the gateway actually reads. The root-cause issue says it plainly: "It is simply never delivered."That diagnosis then sat 29 days in a humans-only work pool. In September it was re-minted into the mechanic lane with a do-not-re-diagnose banner — three earlier cycles had been burned re-investigating a setting that was never the problem. At publish time it is still open, day 31 in the new lane.
Meanwhile, every 30 minutes, the sensor grid sees those backends 100% unavailable and files a fresh P1. A mechanic picks it up, recognizes the known condition, and closes it about 90 seconds later: "node checks cannot change the action. Closing as FP." Then the next tick files it again. Seventy-two times that night. Sixty-three times yesterday. It is still running now.
The four phantom problems
The other subjects are the sensor itself. One chain's oracle row froze at a constant tip lag — "oracle grid is deterministically stalled; node is serving normally," as one close comment put it; 32 of 32 recent filings for that chain were false positives. Two more had stale status snapshots, where the mechanic's ground-truth probe found the node inserting blocks live with healthy peer counts. The close comments even have a standing name for the pattern: "known re-filing FP. No action needed."Here is the part worth being honest about: per issue, the machine behaved exactly as trained. It didn't shrug — it probed, quoted block numbers, counted peers, retired doubt with evidence. It did the right thing two hundred times, because nothing in the system lets a conclusion outlive the ticket it was written in. Each re-purchase of the same verdict costs a mechanic run; the night's token bill ran roughly three times baseline.
Dedup works in space, not time
We built the dedup gate after a gauge failure filed 43 phantom issues in three hours, and it held: 620 concurrent duplicates suppressed, zero duplicate issues open at once. But the gate compares a new signal against open issues. A closed issue suppresses nothing. So closing-as-false-positive at tick N doesn't end the loop — it guarantees a fresh issue at tick N+1. The faster the mechanics close, the faster the wheel spins, and every aggregate we publish reads the wheel as work: 12.8× delivery ratio, 36-second MTTR, a record night.One mitigation has entered the manuals: a per-chain rule that lets mechanics close a known-FP class without dispatching verification jobs — cheaper spin, same wheel. The real fixes are specified and not yet landed: a close-as-FP verdict should arm a suppression keyed on the issue's identity, with an expiry and a pointer to the root-cause issue; and the delivery stats should exclude re-filings of any condition whose root cause is already open — a repair loop shouldn't get throughput credit for re-reading its own diagnosis.
The generalizable lesson: MTTR measures how fast you close tickets, not how fast conditions end. An agent workforce without memory between tickets will re-buy the same conclusion every 30 minutes, forever, with perfect diligence — and its own metrics will score the expense as productivity. The oldest problem in the storm had been fully diagnosed for two months. The machine spent the night proving, seventy-two separate times, something it already knew.