The Vacuous Pass: Why Verification Cannot Tell Silence From Clearance

Every verification system, however sophisticated, ultimately has to render its judgment through a channel with limited bandwidth. Somewhere between the depth of what was actually checked and the report a human reads, that depth gets compressed into something close to a single bit: flagged, or not flagged. This compression is not a failure of implementation. It is close to unavoidable, because a human reviewing thousands of outputs a day cannot absorb the full internal state of a verifier for each one. But the compression has a specific and underexamined cost, and that cost does not shrink as verification systems get better. It grows, because better systems get trusted more, and trust is exactly the resource a compressed signal cannot actually justify.

The claim precisely stated

Here is the structural claim this piece makes, stated as precisely as it can be stated without reference to any particular system: a verification architecture that reports its result through a channel expressive enough only to distinguish pass from fail cannot, by construction, distinguish pass from not evaluated. Both collapse to the same observable state. No amount of care downstream, no auditor rereading the output, no dashboard redesign that keeps the same channel, can recover a distinction that was already destroyed at the point of compression. The fix, if there is one, has to happen at the point where the result is generated, not at any point after.

This sounds almost too simple to be worth stating formally, and that is exactly why it keeps recurring across fields that have nothing else in common.

Vacuous truth as the formal seed of the problem

Start with a piece of standard formal logic that has nothing to do with software or law. A universally quantified statement, of the form for every element of this set, property P holds, is trivially true when the set is empty. There is no element of the empty set that fails to have property P, because there is no element at all. Logicians call this a vacuous truth, and it is not a paradox or a trick. It is simply what the quantifier means. All citations in this filing have been verified as current is a claim of exactly that shape. If the filing contains zero citations, the claim is vacuously true, and it will render, in any system that checks claims of this shape mechanically, identically to a filing whose forty citations were individually checked and all found current.

The reason this matters beyond a logic class is that verification systems are, almost without exception, built around claims of this shape. All flagged items were reviewed. All transactions in scope were tested. All model outputs were checked for the property in question. Each of these is a universal claim over some working set, and each one is vulnerable to exactly the same collapse the moment that working set can be empty, whether by honest absence, by a scoping decision, or by anything upstream that shrinks the set to nothing before the check ever runs. The verification logic itself can be flawless. The vacuous case still passes, because passing is what the logic is designed to do when there is nothing left to fail on.

The channel capacity argument

Push the same idea into information theoretic language, because that framing makes the second half of the argument, why this cannot be patched downstream, considerably sharper.

Treat the output of a verification step as a channel. The channel has some finite capacity, meaning it can distinguish some finite number of distinct underlying states. A system that reports only pass or fail has a channel of exactly one bit. But the actual space of states that could have produced that report is larger than two. At minimum there is: the check ran, and the target satisfied it, the check ran, and the target failed it, the check did not run, because there was nothing to check, the check did not run, because something upstream suppressed the input it needed, and the check ran in a degraded mode that could not actually test the property it claims to test. That is at least four or five genuinely distinct states, compressed into a channel with room for two.

A basic result from information theory is unforgiving here: a channel cannot carry information it did not have the capacity to carry in the first place, and nothing done after transmission can recover what the channel itself was too narrow to represent. It does not matter how carefully a human reads the one bit that survives, or how experienced the reviewer is, or how many times the same report gets reread. The distinctions that mattered were lost before the bit was ever produced. Any fix that operates on the output side, more careful reading, better training, a second reviewer, is trying to reconstruct a signal from a channel that structurally cannot carry it. The only fix that can work is widening the channel itself, at the point where the compression happens, so that not evaluated becomes a state the channel can actually express and distinguish from evaluated and clean.

This is worth dwelling on, because it is the part most operational fixes get wrong. The instinct, when someone points out that a clean result might mean nothing was checked, is to add a note, a caveat, a footnote explaining the limitation. But a footnote is still downstream of the compression. If the channel that actually drives behavior, the dashboard color, the automated escalation, the field that downstream systems key off of, still only carries one bit, the footnote is decoration. The states are already merged by the time anyone reads the footnote.

Where this recurs, and why it recurs identically each time

The pattern shows up wherever a system reports coverage or compliance through a binary or near binary signal, and it is worth walking through several instances precisely, because the mechanism is identical every time even though the domain vocabulary changes completely.

Consider automated code testing. A test suite reports a pass rate, some number of tests ran, and some fraction of them succeeded. A pass rate of one hundred percent is treated, correctly in isolation, as good news. But a test suite with three trivial assertions covering a sliver of the actual logic reports the identical top line number as a test suite with exhaustive coverage of every branch and edge case, one hundred percent passing. The difference between them lives entirely in a second number, coverage, that measures how much of the system the tests actually touched. A pass rate without a coverage figure sitting next to it, with equal visual weight, is a vacuous truth wearing a confidence inspiring outfit. This is precisely why serious engineering organizations insist on tracking coverage as a first class metric rather than an optional diagnostic: without it, the pass rate alone cannot distinguish thorough validation from an absence of anything to fail.

Consider clinical laboratory testing. A diagnostic assay returns negative. That negative could mean the sample genuinely lacked the analyte being tested for, which is the result everyone assumes when they read the report. It could also mean the sample degraded in transit, or was collected incorrectly, or the reagent had failed, in ways that make the assay unable to detect the analyte even if it were abundantly present. A well designed laboratory process includes an internal control, a second measurement whose only job is to confirm the assay was actually capable of detecting a positive if one were present, precisely so that a negative result can be distinguished from a non result wearing a negative result's clothing. A laboratory process without that internal control has, structurally, the exact same one bit channel problem as the code testing example, just with mortal stakes attached.

Consider financial audit. An auditor issues an opinion on a set of financial statements, and an unqualified opinion is read by nearly everyone outside the audit profession as these numbers are fine. What that opinion actually certifies is considerably narrower: that within a defined scope, using a defined sampling methodology, and subject to a materiality threshold, nothing tested rose to the level of a reportable exception. A transaction class excluded from scope, a population defined narrowly enough to exclude the transactions that actually mattered, or a sample that happened not to include the problematic instances, all produce the identical unqualified opinion that a genuinely clean set of books would produce. The scope paragraph exists specifically to prevent this collapse, and much of the recurring criticism of audit quality across decades of financial history amounts, at bottom, to the observation that readers treat the headline opinion as the channel and ignore the scope paragraph that actually carries the missing bits.

Each of these examples comes from a field with no shared vocabulary, no shared regulator, and no shared history. The mechanism recurring identically across all three is the strongest evidence that this is a structural property of binary verification channels in general, not an idiosyncrasy of any one of them.

Where this recurs in our own domain

The same structure appears in the specific problem of verifying that a legal claim's cited authorities actually exist, remain current, and support what is being claimed for them. A verification pass over a document's citations produces, at the level a reader actually consumes, something close to a single signal: citations checked, none broken. That signal is exactly as vulnerable to the vacuous case as every example above. A document that makes no citations requiring verification, and a document whose citation apparatus is present in substance but has been rendered invisible to the mechanical check, whether through reformatting, through paraphrase that drops the formal markers a parser looks for, or through any other transformation that removes the surface features the verification step keys on, both produce the identical downstream state: nothing to flag. A reader looking at that state cannot tell, from the state alone, whether they are looking at a document making no claims that need grounding, or a document making claims that are ungrounded in a form the checker cannot see.

The distinction matters more here than in most of the other examples, because the two cases carry very different consequences and are easy to conflate under pressure. A short procedural note that genuinely cites nothing is not a risk. A substantive assertion of controlling authority that has been stripped of its verifiable form, deliberately or by accident, is exactly the risk the verification step exists to catch, and it is precisely the case the vacuous collapse hides.

The only durable fix follows directly from the channel capacity argument above: the verification output has to carry a third, explicit, equally weighted state, separate from both pass and fail, for exactly this situation, and that third state has to be treated as at least as significant as a failure, not as a quieter cousin of success. A document with substantive unverifiable assertions of authority should not read the same as a document with nothing to verify. Building that distinction into the signal itself, rather than into a caveat attached after the signal has already been generated and consumed, is the only place in the pipeline where the missing bits can still be recovered, because it is the only point before the compression has already happened.

The general design principle

Generalizing across every instance above yields a design rule simple enough to state in one sentence and demanding enough that almost no verification architecture actually satisfies it by default: any system that can produce a clean result must also be able to produce, as a distinct and equally visible output, the statement that it did not evaluate the thing it is being read as having evaluated.

Three consequences follow from taking this rule seriously, and none of them are optional extras.

First, not evaluated cannot be allowed to default silently into the same bucket as evaluated and clean anywhere in the pipeline, from the point the check runs to the point a human reads a summary of it. Every place a default value gets assigned when data is missing is a place this collapse can reintroduce itself even after the underlying logic has been fixed once.

Second, the population or working set that a check operates over needs to be logged and surfaced with the same prominence as the result. A verification step's result is only interpretable alongside an honest statement of what it actually had to work with. Reporting a result without reporting the size and composition of what was checked is reporting half a measurement and presenting it as a whole one.

Third, and hardest to internalize organizationally, an unusually clean result deserves more scrutiny, not less, when the working set has shrunk unexpectedly. A verification pass with fewer things to check than the same class of document usually contains is a signal in its own right, and treating it as neutral or reassuring, rather than as a prompt to check why the working set shrank, is how the vacuous case ends up doing the most damage in practice. Not because anyone designed it in, but because a system that only ever notices problems has, by construction, nothing to say about the absence of a problem it never got the chance to look for.

Why this gets more urgent, not less, as verification is automated

The final piece worth stating explicitly is why this problem grows rather than shrinks as more of the verification work in any field moves from a skeptical human doing it by hand to an automated system doing it at scale. A human reviewer doing the work manually tends to notice when there was unusually little to review, because doing the work manually means directly experiencing the size of the working set. Automation removes exactly that incidental safeguard. A pipeline that runs the same check a million times a day produces a million single bits, and nobody is in a position to notice, case by case, that some of those million results came from an empty working set rather than a genuinely clean one. The very thing that makes automated verification valuable, its ability to produce a clean signal without a human ever looking at the underlying material, is the same thing that removes the incidental check a human reviewer used to provide for free.

This is not an argument against automating verification. It is an argument that automating verification without separately and deliberately engineering the third state described above does not just fail to fix the vacuous pass problem. It removes the last safeguard that was quietly containing it, and does so at exactly the moment the volume of results being trusted without being read goes up by orders of magnitude. A blind spot that a careful human absorbed by accident, one document at a time, becomes a blind spot operating at the scale of everything the automated system touches, the moment nobody rebuilds the missing distinction on purpose.