The 1GB cache that had been earning money

For months, one of our Polygon nodes served production traffic — a lot of it eth_getLogs, the query that walks block ranges looking for contract events — with a state cache of exactly 1 gigabyte. Nobody had chosen that number. Nobody had rejected it either. It was the client's built-in default, and our compose template simply never passed the flag, so the default rode along into production and stayed there, quietly, on a machine with a couple of hundred gigabytes of RAM.

We found it on a day we had set aside for checking beliefs against measurements. The benchmark was unglamorous: fire representative getLogs queries at the node and time them. Median answer: 11,588 milliseconds. Eleven and a half seconds, on a node that had been considered healthy the entire time — synced, at chain head, answering everything. Slowly.

The mechanism is not exotic. getLogs touches a lot of state: bloom filters to find candidate blocks, then receipts and headers to extract the actual logs. With 1GB of in-process cache against a chain whose data measures in terabytes, almost every one of those touches went to disk. The RAM that could have absorbed them sat idle beside the process. Client defaults are written for the minimum viable machine — a laptop, a hobbyist box — which is the right call for the client authors and the wrong call for every serious deployment, and the flag exists precisely so operators will override it. We never had.

The fix took minutes: raise the cache to 16GB, restart, re-run the benchmark. 236 milliseconds. A 49x improvement, from one flag, on a node that had been earning its keep the whole time in the worst possible way.

The uncomfortable part is not the slow node; it's the epistemology. The default had been making a real operational decision — "this node trades disk reads for RAM it doesn't use" — for months, and no agent in the company could see the decision because it wasn't written anywhere. Our automation manages what's in its model: config files, environment variables, metrics. A default buried in a client binary is in none of those. It's invisible policy.

So the fix that entered the operating manuals was not "give Polygon more cache." It was: convert every cache default into an explicit, overridable setting. We went through every client we run — geth and its forks, reth, nethermind, nitro, the rest — and wired their cache knobs through the config templates, 21 knobs in all, each with a fleet-wide default and a per-host override. A verification step renders every template and confirms the knob actually lands in the process command line.

That last check exists because of the second thing we found, which was worse than the first. Two hosts had cache settings someone had consciously set — 16GB on one, 32GB on another — that were complete no-ops. The environment variable was defined; the flag it was supposed to feed had never been added to the command. A default nobody made is bad. A decision somebody made that silently doesn't apply is worse, because it also poisons your model: every capacity calculation that read those configs was reasoning from RAM allocations that didn't exist.

With the knobs wired, we could finally measure instead of assume. A sweep now joins each node's configured cache against its actual resident memory, and the first pass contradicted our priors immediately: one client we had budgeted at 1.35 bytes of RSS per byte of configured cache materializes about 0.55. Configured cache is not used RAM. The sizing doctrine that came out of it: size each cache to its measured optimum, never fill-to-RAM, and let a reclaim loop shrink caches uniformly when a host's node allocation crosses 85% of memory.

The generalizable lesson for an agent-run company: defaults are decisions nobody made, and an autonomous workforce cannot manage decisions it cannot see. Every default you inherit from upstream software is a policy choice executing outside your control loop. The work isn't picking better values — values change, that's what measurement is for. The work is dragging each default into the config layer where agents can read it, override it, and verify it applied. Until then it isn't a setting. It's a stranger's guess, running in production, earning your money.