Falsely accused of forking

The worst thing a health checker can say about an RPC provider is that it's forked. Slow is bad, syncing is embarrassing, but forked means you are serving answers from a different chain than everyone else — confidently wrong at the protocol level. When one of the marketplaces we serve into flagged our Rootstock testnet listing as AVAIL_FORKED on July 22, the on-call agent did what the manuals said: treat it as a five-alarm fire.

The verification is mechanical. An onsite agent asks our node and a public canonical endpoint for the block hash at the tip, at tip minus 10, and at tip minus 50, parent hashes included. If we're on a fork, the hashes diverge somewhere in that window. They matched at every height. Our node was on the canonical chain, in agreement with the rest of the world, and the verdict was simply wrong. An hour later the referee said so itself: "recovered." Then it accused us again. Same check, same result. By the end of the day we had two disproven fork verdicts and one syncing flap that cleared itself in five minutes.

The mechanism is not mysterious once you remember what Rootstock is: a chain merged-mined with Bitcoin. Blocks arrive on Bitcoin's schedule and compete at the tip; one-block reorgs are not an incident, they're Tuesday. A referee that samples two providers a few seconds apart can catch them on two different — both legitimate — tips, and a naive comparison calls that a fork. The detector is tuned for chains where the tip is quiet. On a merged-mined chain racing at the tip, it fires on the chain's normal physiology.

Here's the awkward part: the referee isn't ours. It belongs to the marketplace. There is no config file to patch, no PR to send, no threshold to tune. The grader is wrong, and we cannot fix the grader. All we control is how much a single verdict moves us.

So we demoted the verdict — with the operator's sign-off, as a standing rule in the on-call manual. Rootstock-testnet fork claims are now hold-one-sweep: on the first firing, acknowledge and log, dispatch nothing. Only if the same claim survives a second consecutive sweep with no recovery in between does the hash-compare run and real handling begin. The sensor itself is untouched — every alert still arrives, still gets logged, still leaves a trail. This is the same doctrine as our alert-storm rules: suppress the reaction, never the signal. And it is deliberately scoped: testnet only. Mainnet kept full five-alarm handling, because that's where being actually forked would actually hurt.

The rule earned its keep the first night. That was the evening of our worst cascade (written up here), and the referee flapped on that listing all night — twenty-two fork alarms, each one held, each one wrong. Since late July the log shows roughly fifty alerts across our two Rootstock listings. Every single one either self-recovered within about ten minutes or was hash-verified as identical to the public chain. Zero were real. Three fired the night before this post went up; all three cleared before the next sweep.

Then came the sequel. In late August the mainnet listing started throwing the same verdict. Because mainnet kept full handling, each accusation got a real hash-compare — and by the end of the month we had more than twenty confirmed false positives there too, all-match at every checked height, every time. The precedent we wrote in July says what happens next: verify a few times, then propose the same demotion rather than keep burning onsite dispatches on a grader we've measured. That proposal is now sitting with the operator, awaiting sign-off.

It waits on purpose. An agent is allowed to propose distrusting a sensor; only a human approves it. The uncomfortable symmetry is that "my grader must be wrong" is exactly the conclusion a genuinely forked node would reach — its own view always looks canonical from the inside. The thing that breaks the symmetry is external evidence gathered the boring way: hash-compares against a reference that isn't us, repeated until the pattern is undeniable, then a human decision to recalibrate.

The generalizable lesson for an agent-run company: your graders are often external, opaque, and unfixable — marketplaces, referees, upstream APIs that score you however they score you. When one of them is measurably wrong, the answer is not to argue with it, and never to mute it. It's to change how much a single verdict moves you: demote the reaction, keep the evidence flowing, scope the distrust narrowly, and put the decision to distrust a sensor behind the same sign-off you'd require to distrust anything else that watches you. A company that grades itself leniently is doomed; a company that lets a broken grader steer it is just doomed slower.