Ratchets That Can Fail

Abstract

Most of the tests in a decompiler are not pass/fail in the ordinary sense. They are measurements: how many fixtures round-trip, how many storage slots are recovered, how many functions recompile, how many findings the detectors produce on a corpus that has not changed. The suite pins those numbers so that a regression moves one of them and the build goes red. That design has a failure mode the ordinary kind of test does not: a measurement can stop measuring, and nothing tells you. The pinned value drifts away from the live one, the skip path is taken instead of the real one, the fixture is a shape production never produces, and the suite stays green while the thing it was guarding erodes. This article collects five rules we now hold every metric to, each written down after a metric was found in exactly that state. None of the rules is novel. What we want to record is the specific shape of each failure, because each looked like a passing test until someone counted.

The scar

The rule that started this had a specific origin. A floor on the number of fixtures that pass the “real” round-trip gate was seeded at 15 when the gate was introduced. By the time anyone looked, the live value was 179. The test would have accepted any value from 15 upward, which meant the decompiler could lose 164 fixtures of recovered logic and the floor would not fire. It had been a working test for exactly as long as the number stayed near 15.

That is the general shape: a ratchet that fails in only one direction drifts loose. Every improvement moves the live value further from the pin, and every step of that distance is a regression the pin will no longer catch.

Rule 1: pin the exact value, and fail in both directions

The correction is to pin the exact current value and fail on any change. An improvement then fails the test too, which sounds like friction and is the point: the person who made the improvement re-records the number, and in doing so confirms it moved for the reason they think.

The clearest instance is the count of source-text assertions in the detector tests, the src.contains("...") family that checks how code is spelled rather than what it does. There were 4,066 of them across 88 files, and the suite now pins that number as a ceiling that must only decrease. Two details of the instrument matter. grep -c counts matching lines, not matches, and a line with two assertions counts once; on this tree that undercounts by 11, so the scanner counts occurrences. And the scanner reads raw text, comments included, which caught its own maintainers: a commit that explained a migration by quoting the old assertion form in a comment raised the count by one and failed.

The conversions the ceiling drove show what a source-text pin is worth. One test asserted that three reference URLs appeared in a detector’s source. Deleting the code that attached those references to findings, so every finding shipped without them, left the test passing, because the URLs were also in the docstring. Another test asserted that a historical-loss figure appeared at least three times in a file. Replacing it with per-variant reads of the emitted finding text showed that three of four bridge-vulnerability variants carried a historical anchor and the fourth shipped with none. The file-wide count had been satisfied by the other three. That gap is recorded in a comment and deliberately not asserted, because pinning an absence cements it.

Rule 2: skip is not pass

Corpus-dependent tests skip when the corpus is absent, so a fresh clone can run the suite. The trap is in what “skip” reports. If the skipped test prints a line and returns success, a machine with no corpus and a machine that type-checked 170 programs report the same green.

There was a fix for this: an environment variable that makes corpus absence fatal, so the release gate can insist on the measurement. It was honoured by one test file. Across the Solana tree there were 27 corpus-dependent skip sites in nine files, and 26 of them ignored the variable. Setting it proved almost nothing.

The verification for the fix is a four-state table, and the fourth row is the one that matters:

Those last three were verified to be self-contained, not assumed. On the analysis package, arming with the corpus absent fails eleven gates, every one of which had reported green before.

One subtlety about when to arm. The release gate promotes its scope to “everything” on almost any lockfile change, so keying the arming on the promoted scope would arm it on routine dependency bumps and fail every developer machine without the corpus. It is keyed on the mode the user actually requested. That was proved against the real gate script with a stub compiler that records its environment: the full mode exports the flag on all 30 invocations; the changed-files mode leaves it unset on all 13, even when its scope was promoted.

Rule 3: a fixture’s shape is part of the test’s validity

The Solana lifter emits statements in a register form: an assignment to a register, a read from a register. Across the 170 fixture programs there are 33,273 register operands and 30,166 register assignments. There are zero of the older variable-named forms.

The detector unit tests were built entirely from the variable-named forms: 4,537 sites using the operand form production never emits, 941 using the assignment form. Every one of those tests was green. Every one was exercising a code path no real program reaches. A suite green against a shape that cannot occur reads as coverage and is not.

This one had already cost real behaviour before it was measured. Two cross-program-invocation detectors carried guards that held only because the lifter set two fields to the same value and the hand-built fixtures did not, and the divergence had been written down as though it justified the guard.

The counts are now pinned as ceilings with a floor underneath, and the pinning has a wrinkle worth recording. The first version pinned the number of production match arms on the never-emitted form as a ceiling, on the theory that conversions would reduce it. But a conversion teaches a site to read the register form alongside the old one, so the production count of the old form rises with every correct change, and the first conversion failed the ceiling. The metric was fighting the work it measured. It became a floor on the paired register-form count instead, while the fixture-side count stayed a ceiling, because that one really does shrink.

Rule 4: a corpus total is nearly blind

The round-trip harness measures, per fixture, how much of the original program’s text the decompiled output fails to reproduce, and the suite pins the corpus-wide sum. That sum re-bases every time a fixture is added, and a ceiling denominated in a sum over 170 files is nearly blind to a loss spread thinly across them: lose ten lines in each of thirty fixtures and the total barely moves.

Two practices compensate. First, every re-base carries exact attribution: “the other 173 fixtures still total 40,197, byte-identical to the previous re-base, and the new fixture contributes 650 on its own”. A re-base that cannot say that has not been checked. Second, when a single fixture’s number genuinely rises, four independent checks are required before the total is re-recorded: the per-fixture statement-count floor, at zero tolerance, did not fire; the fixture still recompiles; its text ratio is still one; and the under-count sits near its over-count, because a real loss shows the under far exceeding the over.

Two traps were found while doing this, and both are the kind that pass with a green verdict. The round-trip enumerates the decompiled-output directory rather than the compile manifest, so run before the decompile step it silently measures the previous fixture set and reports pass; that happened, on 173 entries, and was visible only by counting entries in the report. And 155 fixtures were sitting one or two statements above their committed floor at both arms of an A/B, a drift that never fired because the rule is a floor and those were rises. Neither is a bug in the sense of a wrong answer. Both are a measurement that had quietly stopped constraining anything.

Rule 5: measure before diagnosing, and publish the reversal

The last rule is about the person, not the test. Over the month, at least ten filed issues had their stated mechanism overturned by the measurement taken to fix them. A call graph recorded as severed at the dispatcher, with 11 of 349 functions reachable, was re-measured at 279 of 348; the 11 was the reachability with a since-fixed defect re-applied. A fall-through bug hypothesised as an unresolved dynamic jump was 409 misordered block pairs out of 6,883, from resolving the fall-through target by vector index under a cross-function flatten. A definition-of-done proposed three successor heuristics; each was scored against the 103 sites it targeted and refuted, the last one structurally, because its domain was the complement of where the decision it was meant to break actually happens.

Those reversals are on the issues, and three of them are comment-only commits whose entire content is a scored and refuted candidate. The temptation is to fix the bug and let the wrong diagnosis fade. We think the reversal is the more valuable artefact: the next person to look at that subsystem will otherwise start from the same plausible, wrong mechanism, and it costs a day to get back to where the measurement already is.

Why bother

A decompiler’s honesty lives in its metrics. Nobody can read 170 decompiled programs by hand every week; what they can read is a table of numbers and a green suite, and they will trust it exactly as much as it deserves. The five rules are what “deserves” means here. Pin exact values so improvements are confirmed and regressions are caught. Make a skipped measurement report as a skip, never as a pass. Build fixtures from the shapes production emits. Attribute every change to a corpus total to the fixtures that moved. And when the measurement disagrees with the diagnosis, write the disagreement down where the next reader will find it.

References

• Mutation testing — Wikipedia
• proptest: property-based testing for Rust (crate documentation)
• cargo-nextest documentation