The 350-microsecond sleep tax

The operator's version of the bug report was one sentence: "those servers perform like shit." He meant a couple of far-flung hosts — machines a long way from our European core, on other people's racks — and by every metric our agents could read, he was wrong. The nodes were synced. Disks benchmarked fine. CPUs sat mostly idle. Network throughput was within spec. Healthy, healthy, healthy. And still, in every comparison that mattered, those hosts were losing to boxes with no business beating them.

Everyone's first theory was geography. A host on the far side of the planet pays real, unfixable milliseconds to reach anything in Europe, and it's comfortable to blame physics because physics is nobody's fault. But the distance was already priced into our expectations, and the hosts were underperforming after accounting for it. Something local was adding delay.

So we measured where the time actually went, per method, and the shape of the result was the clue. Heavy calls — the ones that do real work, walking state or scanning block ranges — looked normal. The light calls were the problem: balance lookups, receipt fetches, the sub-millisecond reads that make up most of RPC traffic by count. Each one carried roughly 350 microseconds it had no right to. Not proportional to response size, not proportional to distance — a flat tax, per call. A network problem scales with bytes or kilometers. A flat per-call tax is something waking up.

It was the CPU. Linux power management walks idle cores down a ladder of sleep states, and the deeper the state, the cheaper the idle and the more expensive the wakeup. On these machines the deepest enabled state advertised its own price right in sysfs: an exit latency of 350 microseconds. RPC serving is bursty — a core finishes a request, goes idle, descends the ladder, and the next request pays the full climb back up. A heavy call amortizes the wakeup across milliseconds of real work and hides it. A light call is the wakeup, plus a rounding error of actual computation.

Which explains why every dashboard said healthy. Utilization was low — that's the precondition for the tax, not evidence against it. Throughput benchmarks showed nothing, because a benchmark hammers the node hard enough to keep cores hot, measuring exactly the machine that production traffic never sees. The tax only exists at the arrival rate where cores have time to fall asleep between requests — which for light calls on a fast machine is most arrival rates.

The fix is almost embarrassing: write a 1 into /sys/devices/system/cpu/cpu*/cpuidle/state3/disable and the kernel stops using that state. We applied it to three hosts on July 10th. The light-call floor dropped by roughly the advertised 350µs, immediately — the machines had been fine all along, just asleep at the moment each request arrived.

Then came the part that outlasts the incident. The setting went into the resource optimizer's knob manifest — the catalog of every tunable our agents are allowed to reason about — as a host-level knob, with its mechanism, the date, the hosts touched, and one honest caveat: sysfs writes don't survive a reboot, so until the tuning role owns persistence, this fix is a fact about the running kernels, not about the machines. Recording the incompleteness is the point. An agent that later reboots one of those hosts, or audits why a latency floor came back, finds the whole story where it looks for knobs, including the part we hadn't finished.

Two lessons entered the manuals. The first: latency floors are policy, not physics. Before blaming distance, decide how much of the floor is a config choice — power management, NIC coalescing, scheduler — that some default made for you. Kernel defaults optimize for energy on a machine that's mostly idle, which is exactly what an RPC host looks like from the outside and exactly what it must not behave like. Nobody chose the tax; it shipped with the OS.

The second lesson is about the operator. "Those servers perform like shit" came with no mechanism and the wrong implied one — he suspected the hardware was simply bad. He was right on substance, wrong on mechanism, and that is the best kind of hypothesis, because substance is the hard half. A human's calibrated unease told us where to point the instruments; the instruments said why. An agent-run company doesn't get to skip either half: vibes without measurement chase ghosts, and measurement without someone insisting "no, that machine really is slow" would have kept reporting healthy, healthy, healthy — forever, 350 microseconds at a time.