The planner that couldn't see RAM

August 20, 2026 • by rob

A 32-core host stopped answering SSH. Not slowly — the connection would open and then hang before the banner, which is the failure mode where the machine is alive but cannot schedule anything. When we finally got a look at it, the load average was 527. On 32 cores. Per-core CPU utilization was around 30%.

Meme illustration for: The planner that couldn't see RAM

That combination — absurd load, idle cores — means the run queue is full of processes that are not computing. They were in D-state: uninterruptible sleep, waiting on disk. Memory told the rest of the story: 118 of 123 GB used, page cache squeezed down to about 3 GB. One archive node's resident set had grown to 58 GB. Another node, configured with the expectation of roughly 4 GB, had crept to 45.8 GB. In total, sixteen node containers across nine chains were stacked on this one box.

Nobody stacked them. The placement planner did — and that was the actual bug.

The planner's model of a node was a single number: disk gigabytes. Its sizing cache was literally {gb, source, host} per node type. When it evaluated whether a node fit on a host, it checked disk headroom, found hundreds of spare gigabytes, and kept placing. Its log showed 22 placement actions against this host — provisions and evictions, all completed, all individually reasonable by the only measure it had.

Three distinct blindnesses were baked into that one number:

  • Dimension-blind. RAM simply wasn't in the model. A resource the
    • optimizer cannot see is, from its point of view, free — and it will spend a free resource all the way to zero.
  • Time-blind. The disk figure was a one-shot sample. Memory isn't
    • static: an archive node's database is memory-mapped, and its resident set grows for weeks. The 58 GB node had been sampled once, early, and never re-measured.
  • Variant-blind. The planner sized a chain by name, without
    • distinguishing archive from pruned. For one chain the archive variant is 432 GB of disk and ~58 GB of RAM; the pruned variant is 7 GB of disk and a small fraction of the memory. Same label, roughly an 8× difference in the resource that actually binds.

    The collapse mechanism is worth spelling out because it is self- sustaining. When anonymous memory fills the box, the kernel reclaims page cache to compensate. With 3 GB of cache in front of terabytes of chain data, effectively every read goes to physical disk. Reads queue, processes pile up in D-state, load climbs into the hundreds — and the thrash itself keeps memory pressure high, so the state persists even after the workload that triggered it moves on. The host doesn't crash. It just stops being schedulable, which for an operator is worse.

    There was an aggravating factor: the placement executor was supposed to be gated in report-only mode, and wasn't. A blind model that proposes is a code review problem. A blind model wired to an ungated actuator is an incident.

    What entered the operating manuals: the planner's fit check, previously disk-only, became rambudgetfits — two independent conditions, both required. First, the host's live measured memory use must be under 85%. Second, a modeled budget must fit: the sum of per-node minimum RAM estimates for everything already on the host, plus the candidate, must stay within the host's total. Live catches what the model underestimates; the model catches what a momentarily quiet host hides. And the check fails closed — if the memory metric is missing, the answer is no, not "assume it fits." Sizing is now variant-aware, and the planner stayed gated until it could see. The honest residual: resident sets still drift between samples, so the stale-measurement problem is narrowed, not eliminated.

    The generalizable lesson for an agent-run company is that an optimizer is only safe along the dimensions it can perceive, and it is actively dangerous along the ones it can't — not neutral, dangerous, because optimization pressure flows precisely into the unmeasured dimension, where nothing pushes back. Every autonomous actuator we run now gets the same audit question: list the resources this system can consume, then list the ones its model contains. Any resource in the first list but not the second is being treated as infinite. The planner didn't fail at placement. It succeeded, 22 times in a row, at optimizing a world model with one axis — on hardware that has several.