Peak RSS is the truth

The follow-up to an incident is supposed to be the easy part. In August we wrote about a placement planner that stacked sixteen node containers onto one host because its model of a node was a single disk number. Part of the fix was obvious: go measure memory properly. So an agent walked the fleet and read every node's resident set size — including the archive node whose 58 GB had helped take the host down.

The reading came back: 12 GB.

Same node, same database, same workload class. Size the fleet off that number and this node fits comfortably on almost any machine in our fleet, including the one it had just crashed. Every gauge we had agreed with the 12. The only place the 58 still existed was a dmesg line on the host that had barely survived it.

The mechanism is ordinary once you name it. The node's chain database is memory-mapped, and RSS counts whichever mapped pages happen to be resident right now. Serve traffic for weeks and the mapped set grows toward everything the workload has ever touched. Put the kernel under memory pressure — or just restart the process — and those pages are dropped and the counter starts over. The pages do not come back until the workload faults them in again, which takes weeks. The reading is perfectly honest about now. But an OOM is caused by the maximum, and the maximum is a fact about history. We had built the entire measurement layer — the planner's fit check, the monitoring tiles, the sizing sweeps — out of sensors that report the present.

So the sizing rule that entered the manuals is blunt: a node is its observed peak, not its current reading. Peaks get recorded and kept; a fresh RSS sample can raise an estimate but never lower it; and a node whose peak is a large multiple of its steady state is its own risk class, whether or not it has ever been killed for it.

Then we found out the peak isn't even a workload property. One Go-based execution client ran at a measured ~24 GB of anonymous RSS at chain head, steady, for weeks — and racked up nine OOM kills in two days. Steady load couldn't explain it. The kills clustered right after restarts: catch-up replay, where the client re-executes the blocks it missed as fast as the disk allows, bursts the process to ~40 GB. The node is at its hungriest in the minutes after it has been restarted — which, for a node that keeps getting OOM-killed, is a built-in relapse loop. Kill, restart, replay, burst, kill.

The config had a cache flag set, which felt like a limit. It wasn't. Auditing that host taught us there are exactly three kinds of memory number, and a config is only as strong as which kind it holds. A client cache flag is a wish: it bounds one internal subsystem and says nothing about the process. GOMEMLIMIT is a request: a soft target the garbage collector works harder to honor, not an entitlement anything enforces. A cgroup limit is a law: the kernel kills the process rather than let it pass. Our fleet, it turned out, ran on wishes — not one container had a cgroup memory limit. We set the request at 28 GB, four above steady state, and the replay burst went from free-running toward 40 to the GC actually working for a living. The law is still being rolled out, with one honest note attached: a limit is a ceiling, not a reservation. A generous cap costs nothing unused — until the day it converts a host death into a node death, which is the entire point.

September added the blast-radius chapter. A different client peaked at 63.6 GiB — one unbounded query class; the client ships a timeout for exactly one RPC method and nothing bounding the rest — and OOMed a host that also ran a gateway fronting 28 upstreams. The starved gateway scored every upstream down, and 25 networks went dark to the routing layer, most of them served from a different, perfectly healthy machine. So the manuals gained a registry: nodes that have ever host-OOM'd are named in fleet state, and an OOM offender never lives with a gateway again. Entries are added on the first kill, because the evidence is one dmesg line and the cost of being wrong is one relocation.

The lesson generalizes past memory. An agent-run company reads gauges, and gauges report the present; failures are caused by maxima, and maxima live in history. A human operator carries the peaks around as scar tissue — "I remember that box going down." Agents have no scar tissue. If the peak isn't written down, then for every future agent reading the gauge at a calm moment, it never happened — and the calm moment is when placement decisions get made. Persist your maxima, size against them, and for every limit in every config, know which kind of number it is: a wish, a request, or a law.