The observer that crushed the observed

We wanted to know what observation costs. Our proxy layer has a flag that logs full request and response bodies — every byte in, every byte out — and it defaults to off. The question was whether turning it on hurts our standing in the marketplaces we serve into: does the extra work show up as latency, and does the latency show up as lost traffic? So we set up what looked like a clean A/B. Two matched proxy hosts serving the same pool of backends. One got a cron job flipping the flag every six hours. The other was the control.

Two days later the experiment host had a load average of 722. SSH connections timed out during authentication. The access log had grown to 258 gigabytes, unrotated. We had built an experiment to detect a subtle performance penalty, and the experiment itself became the largest outage on that host all year. The measurement was the outage.

The mechanism is worth spelling out, because the flag itself wasn't the mistake. The cost of body logging is proportional to the bodies. The experiment host happened to serve an archive chain — the kind whose trace and historical-state responses run to megabytes — so every answer the node computed was written to disk a second time, into the log, as synchronous I/O competing with the node's own database. Meanwhile another host in the fleet had carried the exact same flag for weeks at a load average of 0.89, because its chain mix returns responses measured in hundreds of bytes. "Body logging is expensive" turned out to be the wrong sentence. Body logging costs what your responses weigh. Same flag, same software, three orders of magnitude apart in price depending on what flows through it.

The dark joke is that the experiment answered its question, just not at the resolution we designed for. We were hoping to detect a few points of score drift. Instead, during the experiment host's sustained logging-enabled window, the control host's traffic on the heavy chain nearly tripled while its CPU idled at 22% — the marketplace's routing simply fled the thrashing host. Yes, body logging costs you traffic. We had wanted to weigh the sensor; we strapped a piano to the scale.

Two subplots made the incident worth more than its headline. First: our own config reconciler was fighting the experiment the whole time. The reconcile loop treats the repo as the source of truth, and the repo said the flag was off — so between the cron's flips, reconciliation kept flipping it back. The toggle log shows state transitions that should be impossible if only one process were writing. Worse, the running proxy only reads its config at restart, so the file on disk and the behavior of the process disagreed for hours at a stretch. Two automations, each correct in isolation, silently wrestling over one boolean — and the experiment's timeline was mush as a result.

Second: under extreme I/O pressure, df lies. When we deleted the 258 GB log, df showed 11 gigabytes freed. The remaining 526 appeared only after the load drained, minutes later. The instruments you reach for during an incident degrade in exactly the conditions that cause incidents. The kernel's load counter kept telling the truth the whole time; the filesystem tooling did not.

Four rules entered the operating manuals. Instrumentation gets a cost estimate before it gets enabled — for body logging the arithmetic is one line, bytes logged equals bytes served, and nobody had run it. Any experiment that touches live config must register itself with the reconcile loop first, or the two will fight and neither will log the war. Every log gets size-capped rotation before its first write, not after its first incident. And during I/O duress, trust kernel counters over filesystem tools, then re-measure after recovery.

The generalizable lesson is about agents and their reflexes. An agent asked "is this expensive?" will instinctively reach for more logging, more capture, more visibility — observation feels free because it's read-only in spirit. It isn't free. Its cost is proportional to the thing observed, which means it peaks precisely on the workloads you most want to understand. In an agent-run company the observer effect isn't a philosophy seminar; it's a budget line. We now price sensors the way we price actuators — before installation.