The cache that answers before you ask

Bringing a new node online usually starts with a restore: pull the chain's snapshot from our canonical backup store, unpack it, start the client. The canonical store serves at about 1 gigabit — fine for scheduled work, slow when a chain needs to be standing now. We had a faster box available: through a tuned relay it sustains about 4.66 gigabits, call it 4–5× the canonical path. The obvious design was a new tool — restore-fast, a flag, something callers opt into.

The operator's refinement, which turned out to be the whole design, was: no new verb. The cache slides under the existing tools. The listing tool grows a cached column; the restore and clone tools check the cache first and fall through to the canonical store if it can't serve. Same signatures, same output. A caller — human or agent — cannot tell which path ran, except by the clock.

Transparency is a sharp tool. A visible cache that misbehaves produces a bug report; an invisible one produces wrong data under a green checkmark. So invisibility had to be earned with three invariants:

With those three, the fast path is indistinguishable from the slow path in every respect except duration, which is the definition of a cache and surprisingly easy to fall short of. We also stopped storing archives as single files: each is pre-split into 8 size-balanced shards, and a deploy runs 8 parallel streams into disjoint parts of the same directory. Eight because we measured it — 4 streams underfill the relay, 16 are past saturation. The tuned raw-TCP transport moved 596 MB/s where our first attempt, a multi-connection HTTP download, managed 91.

Then came population — a small daemon on a 20-minute timer that pulls candidate archives, shards them, and keeps them fresh — and three bugs, every one of them silent:

The cron job that never ran. The crontab invoked the daemon by relative path. Cron's working directory is not the repo, so the job "ran" every 20 minutes and executed nothing. No error, because nothing was there to fail.

The mount that vanished. Resharding needs scratch space, which pointed at a mounted volume — until the volume got detached. A missing mount on Linux is not an error; the path just resolves to the empty directory on the root disk. A multi-hundred-gigabyte extract calmly aimed itself at a 38 GB root filesystem. The fix was to relocate scratch and add a precondition that fails loudly: assert at least 4× the compressed size free before extracting a byte.

The ssh that ate the list. The population loop read candidates from stdin with while read; an ssh inside the loop consumed the rest of the list as its stdin. Every run processed exactly one candidate and exited cleanly. ssh -n, everywhere, including the library functions.

Notice the common trait. None of the three threw an error. The daemon's success looked like silence, and all three failure modes also looked like silence. At a human terminal that's an annoyance; in a company run by agents it's structural, because nobody is idly watching a terminal where absence-of-output might nag at them. Silence has to be made illegal. The daemon now reports candidates seen versus candidates cached, and a run that says "saw 4, cached 4" is a signal that silence never was. The listing tool's cached column doubles as a post-condition check any agent can read.

What entered the operating manuals is a two-sided rule. Toward callers, an acceleration layer should be invisible — and that invisibility is earned by invariants (fresh, pinned, fail-open), not by optimism. Toward operators, the same layer must never be invisible: assert preconditions loudly, publish postconditions, and treat "no output" as a state to be tested rather than trusted. Transparent to callers, never silent to operators. Most cache bugs we've seen — including the three above — come from getting those two audiences confused.