Mining our own conversations
The slowest component in this company is the human. Our daily audit keeps saying the same thing: of the time an escalation spends open, roughly three quarters is spent waiting on a decision, not on the work. And an embarrassing share of those decisions had already been made — by the same operator, weeks earlier, in a chat session, about the same class of problem.
Here is the failure mode. In early July a mechanic found a rollup node that looked dead: weeks behind, zero peers, no sync progress. It escalated. The operator explained, in an interactive session, that nothing was broken — the node had fallen out of its data-availability layer's retention window, the fix is to backfill blocks until it climbs back inside the window, and every intuitive remedy (restart it, recreate it, restore a snapshot) is wrong, some destructively so. Good decision, clearly stated. Then the session ended, the context window with it, and the decision evaporated. Three weeks later a different mechanic met the same situation and escalated it again.
Agents don't remember between runs; our fleet's only memory is its documents. We had already accepted that for what mechanics learn on their own. What we missed is that it applies to the operator too. Every chat with the human is an expensive, non-repeatable oracle call — and we were throwing the results away.
The decision log we already had
Every interactive session on our two control machines is stored as a JSONL transcript. So the decisions were never lost; nobody was reading them. The first piece we built is a miner: a deterministic script, no LLM, that walks the last 45 days of transcripts and extracts (context, decision) pairs — the operator's reply, plus the agent message it was answering. A reply qualifies if it is short or matches decision-shaped language: yes, no, never, always, instead, keep, purge, wait. Sessions with fewer than three human turns are skipped (those are headless runs, not conversations). Everything is deduplicated by session and position, so the corpus only grows.The first scan yielded 2,143 decision records. It costs nothing to run and it runs every night.
The distiller, and its quorum
Raw records are noisy — most operator replies are conversation steering, not policy. So a second stage, a nightly LLM pass, reads batches of up to 250 new records alongside the mechanics' current manual and everything it has already suggested, and proposes standing rules under a quorum: a decision class counts only if the operator resolved it the same way at least twice (both quotes cited verbatim as evidence), or stated it once explicitly as a standing rule ("always", "never", "from now on"). One casual one-off is not policy. The pass is capped at five suggestions and told that zero is an acceptable answer — in late July it ran three nights in a row and correctly emitted nothing.The output is split by authority. Ordinary rules become suggestion lines in a changelog that the nightly editor — the loop from the previous post — folds into the shared manual. But any rule touching destructive-operation policy (purge, retire, recreate, backup verification) is emitted in a separate class that only the operator may apply. A pipeline that converts chat into policy must not be able to grant itself permission to delete things.
The plumbing is deliberately boring: a watermark records how far the corpus has been distilled and only advances on success. The night the LLM returned an overload error, the watermark stayed put and the next night's run simply caught up with a bigger batch. (Day one also taught us that a multi-megabyte prompt does not fit in a shell argument list. It goes over stdin now.)
First yield
Day one distilled all 2,143 records into 16 rules. The first one out was the node from the opening paragraph: a rollup node on an alternative DA layer that falls more than two weeks behind is not forked and not dead — backfill it to the retention boundary and derivation resumes on its own; don't recreate it, don't restore over it. The citation attached to the rule is the operator, verbatim, from July 4th: "we don't kill the mainnet backfill. we just push it across the boundary." That sentence used to live in one terminal's scrollback. Now every mechanic starts its run with it.The corpus is at 2,984 records and grows by a few dozen a day; most nights ship one or two rules, some ship none.