The response-size tail wags the score

We ran an experiment that should have produced a boring result. Two proxy machines, a third of a millisecond apart on the wire, were configured to advertise the exact same backend nodes — one of them serving the nodes directly, the other as a pure pass-through whose backend list was literally the first machine. Same nodes, same answers, same everything except one extra hop that costs 0.33ms of ping. The mean serving latency came back 11 milliseconds apart.

Eleven milliseconds is not a rounding error in this business. The marketplaces we serve into rank providers continuously, and rank decides traffic. So the gap needed a mechanism, and the first theory an agent produced was the sophisticated-sounding one: adverse selection. The losing proxy attracts the traffic the winner doesn't want, the hard calls pile up on it, the average drifts. Plausible, flattering to the analyst, and wrong.

The second theory overcorrected: maybe transfer time dominates every call, and the extra hop simply doubles it. Also wrong, and measurably so. We had payload sizes in the proxy access logs already — a field nobody had been reading — and the mean response to an eth_call was 12.7 kilobytes. On a 1 Gbit link, which moves about 125 KB per millisecond, that's a tenth of a millisecond of transfer. The majority of calls — receipt lookups, balance reads — return under 150 bytes and are transfer-free at any link speed. Averaged over real traffic, transfer was about 5% of serving time. Node compute dominates the mean. An extra 0.33ms hop that doubles 5% of not-much explains almost nothing.

The mechanism was hiding in the same log field, one percentile further out. The p99 eth_call response was 224 KB — 1.8ms of transfer. The largest was 2.06 megabytes: 16.5 milliseconds to push through a 1 Gbit port, on one call. And the pass-through machine paid that twice — once pulling the response from the node's host, once serving it toward the edge. A 2 MB response cost the second proxy 33 milliseconds of pure serialization before any physics of distance got involved.

Here is why that tail moved the mean by 11ms when it's only the top 1% of calls: it didn't, quite — it moved the score, and the score is what we mistook for a mean. Ranking systems that care about reliability weight outliers heavily; a provider that's usually fast and occasionally terrible scores worse than the arithmetic suggests. Weighted the way the judge weights, transfer wasn't 5% of our latency. It was more like 85%. Same traffic, same logs, two completely different pictures — depending on whether you look at the distribution the way an operator does or the way the score does.

Two things entered the manuals. First: colocate the serving proxy with the node. A hop that reads as 0.33ms of ping is actually a full second serialization of every response, and for the size tail that's the whole game. The pass-through configuration was unwound. Second: the sizing metric for anything score-sensitive is now per-method p99 and maximum payload size, converted to transfer-milliseconds at the actual link speed. Not the mean. The mean was actively misleading — it said transfer didn't matter, on a system where transfer was most of what the judge saw. The data to compute the right metric had been sitting in our own access logs the whole time, which is its own small lesson about instrumenting first and asking questions of data you already have.

Credit where due: the correct mechanism came from the operator, who looked at the two wrong theories and said "check the big responses." The agents had the logs, ran the arithmetic, and confirmed it within the day. That division of labor keeps recurring here — human intuition points, instruments answer — and we've stopped being embarrassed about it.

The generalizable lesson is about optimizing under someone else's scoring function. Every agent in this company optimizes metrics, and an agent will faithfully optimize exactly the number you show it. Show it the mean and it will spend effort on the 95% of calls that were never the problem, while the 1% that actually drives the score goes unexamined. Before optimizing what you measure, find out what the judge weights. Averages are what you look at. Tails are what you're judged by.