On this page
Abstract
Every static-analysis finding carries a confidence score, and almost nobody agrees on what the number means. In a detector suite written by several people over two years, a 0.75 in one detector and a 0.75 in another were unrelated facts: each author had picked a value that felt right for the pattern in front of them. That would be harmless if the number were decorative. It is not. Confidence is multiplied by context modifiers when a finding lands in library code or behind a proxy, compared against floors that decide whether a finding is shown at all, and read by users as a promise about how much evidence sits behind a claim. This article describes the change we made to the EVM detector suite over the summer: confidence is no longer a float a detector types, but one of a small set of named evidence classes, each with a fixed base value and a narrow modifier. It walks through what the migration required, which was reading every detector’s predicates rather than rounding its numbers, and what it found along the way, including a finding that was surviving suppression by six thousandths and a detector whose findings had shipped at zero confidence for as long as it had existed.
What a confidence value is actually used for
It is worth being concrete about where the number goes, because the answer is what makes hand-typed values dangerous.
First, a finding’s confidence is scaled by context. A finding inside code recognised as a vendored library gets multiplied by a small factor, on the reasoning that a widely deployed library is more likely to have been reviewed than a bespoke contract. Proxy shells and compiler idioms get similar treatment. Second, the scaled value is compared against a floor, and a finding below the floor is dropped before the user sees it. Third, findings are ranked, and the rank is what a user works through when there are two hundred of them.
So a detector that types 0.78 has made three decisions it did not know it was making: how the finding behaves after a ×0.2 library multiplier, whether it lands above or below a 0.15 floor, and where it sorts against every other detector’s findings. The author was thinking about none of those. They were thinking “this is fairly likely to be right”, and encoded that thought as two decimal digits.
The arithmetic shows how thin the margin can be. A 0.78 finding in library code becomes 0.156. The floor is 0.15. The finding survived, but by 0.006, and any reasonable adjustment to either constant would have made it vanish without a test noticing. Several findings in the suite were in that position. None of their authors knew.
The ladder
The replacement is a ladder with five bases and one modifier step. The bases are named for the kind of evidence a detector has when it fires:
| Evidence class | Base | What the detector can show |
|---|---|---|
| Confirmed data flow | 0.90 | An attacker-controlled input reaches the dangerous operation along a path the analysis followed end to end |
| Guard proven absent | 0.75 | The protective check that would make the pattern safe is demonstrably not present in the lifted code |
| Pattern in context | 0.60 | The dangerous shape is present at a site the analysis cannot prove dead, but reachability is not established |
| Structural match | 0.45 | The shape matches, with no context supporting or undermining it |
| Heuristic | 0.30 | An indicator the detector treats as suggestive, not conclusive |
A detector may strengthen or weaken its class by one step of 0.05. Combined, that gives exactly fifteen legal values. 0.78 is not one of them, and neither is 0.82, and that is the point: a value off the grid is a value nobody has justified.
Two properties matter more than the specific numbers. The class travels with the finding, so downstream logic can reason about evidence rather than about a float. And the floor logic reads the class first: a finding whose guard is proven absent is admitted regardless of what the multiplier did to its number, because “the check is not there” does not become less true inside a library. The untiered fallback, comparing a raw float against the floor, is what produced the 0.156 case.
Migration is not rounding
The tempting way to do this migration is mechanical: map every 0.78 to 0.75, every 0.82 to 0.80, run the tests, done. We did not do that, because it would have preserved the original problem in a new form. The number would be on the grid, and still nobody would know why.
Instead, each off-grid value was replaced by reading the detector’s predicates and asking which evidence class they actually establish. Three examples show the range.
UUPS initialization. The detector emits three findings about upgradeable proxies whose implementation can be initialized by anyone. Two of them fire when the analysis has walked the initializer and shown that the ownership guard is not in the lifted function body. Those are “guard proven absent” at 0.75, and the old 0.78 was a rounding of the right intuition. The third fires when the raw delegatecall opcode appears at a site the analysis cannot prove unreachable, but nothing shows that an attacker reaches it. That is “pattern in context”, strengthened one step to 0.65 because the opcode is specific. Same detector, same old number, two different amounts of evidence, and the migration made the difference visible.
Diamond storage collision. The detector reasons about facets, but on stripped bytecode it recovers facets by heuristic decomposition rather than from the diamond’s own registry. Its own comments admitted the decomposition was a stand-in. So the ceiling is set by the weakest link: selector collisions land at “pattern in context”, and storage collisions one step weaker at 0.55, because the grouping is what is uncertain, not the duplication the grouping revealed. Before the migration both arms were 0.78.
ERC-165 spoofing. Two findings about supportsInterface implementations that claim interfaces the contract does not implement. These were the 0.156 case. They were also the site of a small comedy: one test pinned the absence of 0.82 in the source, protecting an earlier partial migration, while a second test asserted that 0.78 was present. One pin protected the migration and the other blocked it, in the same file. Neither told anyone what the finding’s evidence was.
What the migration found
Reading every detector’s confidence path, rather than its constants, turned up things that a rounding pass would have preserved.
A value that was dead. One detector’s census entry said it emitted 0.78. Reading the builder chain showed that the 0.78 was set and then overwritten by a later call before the finding was ever emitted. It had never reached a user. It was deleted, not migrated. The three values that survived in that detector, 0.35, 0.90 and 0.75, were left as literals on purpose, because they are context multipliers applied to identical evidence in different situations, and the ladder is for evidence, not for context.
A detector shipping at zero. The dead-code detector never set a confidence. The builder’s default was 0.0, so every finding it had ever produced carried zero confidence, and its library mitigation, which multiplied confidence by 0.3, had never lowered anything because there was nothing to lower. This was found by a behavioural check rather than by reading: a test that walks every finding builder in the suite and asserts the emitted confidence is on the grid. The population it had to cover was smaller than feared. Twenty-three builder chains, nineteen of which set confidence, and all four that did not were in one file.
Pins that protected the wrong thing. Several tests asserted the presence of a specific .with_confidence(0.78) call in the source text. Those tests do not check behaviour; they check spelling. A detector could emit the wrong confidence, or no confidence, and pass. The migration replaced them with assertions on the emitted finding, and the number of source-text assertions in the suite is now itself pinned as a ceiling, so it can only go down.
Why this is worth an article
A confidence ladder is not a novel idea; risk frameworks have used ordinal evidence scales for decades. What we want to record is the specific way an unstructured score fails in a static analyser, and why the fix is a reading exercise rather than a refactor.
The failure is that a confidence value does work far from where it is written. The detector author sees a local judgement. The floor, the multipliers and the ranking see a number with no provenance and act on it. Once a suite has more than a handful of detectors, nobody can hold the interactions in their head, and the only defence is to make the number mean something the downstream logic can read. A class does that. A float does not.
The reason the fix is a reading exercise is that the right class is a property of what the detector proves, and only the detector’s code says what it proves. Two 0.78s in one detector turned out to be 0.75 and 0.65. A 0.78 in another was dead. A missing value was 0.0. Mapping numbers to nearby numbers would have kept all of that hidden.
There is a limit worth stating. The ladder describes the quality of evidence a detector had when it fired. It says nothing about how often that evidence turns out to be right in practice; that is calibration, and it is tracked separately from observed true and false positives. The two should agree over time, and where they disagree, the detector whose evidence class is consistently too optimistic is the next one to read.