Everyone Is Verifying Legal AI. Almost Nobody Is Verifying It Against Anything Real.

This week we ended up in three unrelated conversations that all used the same word, verification, to mean three different things. One was with a platform that scores AI outputs against a rubric. One started from a paper out of Stanford, Berkeley, and NVIDIA, broken down sharply by a legaltech newsletter we follow. One was about the EU AI Act's next compliance deadline, eleven days out at the time of writing. Put side by side, they map almost perfectly onto three separate layers that the industry keeps collapsing into one word, and legal is the domain where collapsing them costs the most, because the price of being confidently wrong is a citation that doesn't exist, or a holding that isn't what a memo says it is.

Here is the taxonomy, and then the technical part: what it actually takes to build the third layer, the one almost nobody builds, because it is the least generalizable and the most expensive of the three.

Three things people call "verification"

Disclosure verification. Did you tell the user they are talking to a system, not a person. Did you label the synthetic content. Did you train your staff on the risks of the tools they use. This maps onto Article 4 (AI literacy, which has applied since February 2, 2025) and Article 50 (transparency, enforceable from August 2, 2026) of the AI Act. It is real and it is not optional, and it is auditable at market scale precisely because it does not require anyone to check whether the underlying output is correct. A regulator can confirm a training program happened. A regulator cannot, across an entire market, confirm that a specific legal analysis was right.Sources: EU AI Act (Reg. (EU) 2024/1689)

Confidence verification. An LLM grades another LLM's output against a rubric and returns a score. This is the fastest moving layer right now, and it is genuinely improving. The interesting recent development is a paper called LLM-as-a-Verifier (Kwok et al., Stanford, Berkeley, and NVIDIA, arxiv.org/abs/2607.05391), which found that a standard judge model, asked to score a candidate on a five point scale, throws away almost all of its own signal the moment it emits a single integer. Underneath that integer sits a full probability distribution across every possible score, and reading that distribution instead of the rounded output turns a coarse, tie-prone judge into a much sharper one. On a hard coding benchmark, a discrete judge ties two candidates 26.7 percent of the time. Reading the continuous expectation instead produces zero ties on the same benchmark, and on one adversarial example the authors dissect, a subtle validation cheat that a discrete judge missed 88 times out of 100, the continuous version catches it correctly the majority of the time, more so as the scoring scale gets finer.Sources: LLM-as-a-Verifier (Kwok et al., arXiv:2607.05391)

That is a real, useful advance, and it costs nothing to add: no training, just a different read on logits a running model already computes. But the paper's own benchmarks share one property its authors are upfront about: each comes with a hard answer key, a hidden test that resolves cleanly, a robot that either grasps the object or it does not. Marco Rossi, who writes the newsletter The LegalTech Overlap, took that boundary and built his own four question test for where the technique earns a place on a lawyer's desk, starting with whether the task has a checkable notion of correct at all. Where it does not, a smooth, confident score is not an improvement on an honest tie. In Rossi's framing, a number like that dresses contested judgment up in "the costume of a measurement" and invites a reader to trust something that was never actually measured. The one case he names as clearing every one of his four gates without hesitation is checking that a citation in a legal memo actually exists and actually backs up what it is cited for.

Source verification. Does the specific factual or legal claim in front of you match a specific, real, checkable, current, jurisdiction-correct source. This is the narrowest of the three, the hardest to build, and the one that generalizes the least, because it needs an actual domain corpus, not a general purpose judge. It is also the layer that neither disclosure nor confidence scoring touches at all.

What source verification actually requires

This is the part that is easy to gesture at and hard to build, so here is what it actually takes, mechanically.

A real, current, jurisdiction-correct corpus. You cannot verify a citation against what a model remembers, because what a model remembers is exactly the thing that produced the hallucination in the first place. You need a retrievable body of actual primary source text, statutes and case law, that is current (an article repealed last year cannot ground a claim about the law today, so the corpus needs a temporal dimension, not just a static snapshot) and jurisdiction-correct (the same doctrine means something different across jurisdictions, and silently defaulting to one jurisdiction's rule as if it were universal is its own quiet failure mode, easy to miss when test data happens to skew toward wherever the system was built first).

Holding match, not vector proximity. Retrieval gets you to the right neighborhood. Embedding similarity will happily hand you a real, existing case that is topically adjacent to the claim in front of you, and topical adjacency is not the same thing as the case actually saying what it is cited for. A citation can point to a genuine ruling and still misstate its holding, and that failure mode passes an existence check cleanly, which makes it more dangerous than a fabricated case, since a fabricated case is at least sometimes catchable by a naive existence search. A second, independent check has to bind the specific claim to the specific source text, not just confirm that something plausible turned up nearby.

Independent audit, not self grading. The process that generates an answer is structurally the wrong process to ask whether that answer is correct. A model confident enough to invent a citation is not, by virtue of being asked a follow up question, suddenly more skeptical of its own output. This is the actual argument for splitting generation and verification into two independent passes, not one model marking its own homework, but two processes built to genuinely disagree with each other, where the second one's entire job is to try to break the first one's claim rather than confirm it.

Decomposition, tested adversarially, not just designed once. A single vague question, is this citation correct, hides a handful of narrower, genuinely checkable ones: does the cited authority exist, does the date match, does the amount match, does the holding actually say what is claimed. Splitting the question is where most of the accuracy lives, and it is also where the risk hides, because a hand built rubric only protects against the failure modes its author thought of. The dangerous gap in a decomposition is never the sub check that is obviously missing, it is the one nobody wrote because nobody thought of that particular way of being subtly wrong. In practice that means adversarial testing against specific categories of mutation that preserve surface plausibility while flipping the actual meaning: a negation flipped, can into cannot, a quantifier quietly weakened, all into some, a magnitude changed by an order of ten, five hundred milligrams into five thousand, a sum written out in words that no longer matches the digits two sentences later, a date shifted by exactly the kind of small edit a careless paraphrase introduces. Every one of these needs its own explicit, executable test, in every language the system runs in, because a mutation a verifier catches in one language can pass silently in a language nobody built a test for, and a guarantee that holds in one language while quietly failing in another is worse than no guarantee at all, since it looks like coverage from the outside.

Abstention as a first class output, not a fallback. A verification system needs a genuine third answer, not just verified or not verified, but an explicit statement that a given claim simply is not the kind of thing that can be checked, together with the reason why. Most legal AI products are built to never come back empty handed, so a question with no real answer gets forced into one anyway, and once it is expressed as a score, nothing downstream can tell it was ever unanswerable to begin with. Building a system that can say, plainly, this one is outside what I can verify, is a harder product decision than it sounds, mainly because that answer is the one that makes the product go quiet exactly when a demo wants it to talk.

Why regulation does not close this, and was never going to

None of this is a criticism of the compliance layer. Disclosure obligations exist because they are the thing a regulator can actually check at the scale of a market, cheaply and repeatedly, without reading the substance of every output a company's AI ever produces. That is a reasonable design choice for a regulation that has to apply to every company using AI, not only the ones building legal tools. But it means full compliance and full correctness sit on genuinely independent axes. A company can have every disclosure in order, every employee trained, every synthetic output labeled exactly as the law requires, and still be shipping a legal assistant that invents a citation or misstates a holding, because no article in the AI Act asks that specific question, and no confidence score from a general purpose judge is built to answer it either, not for the contested majority of legal work.

Naming the gap

Line the three layers up and the shape of the industry's effort becomes obvious. Enormous energy is going into disclosure, because regulation demands it on a deadline. Real, fast moving energy is going into confidence scoring, because it is general purpose, training free, and it transfers across every domain a judge model has ever seen text from. Very little is going into source verification, because it is the opposite of general purpose. It needs a real corpus for the specific domain, adversarial coverage built and maintained for that domain's specific ways of being subtly wrong, and an architecture that treats staying silent as cheaper than being confidently incorrect. It does not transfer. It has to be built once, honestly, for the one domain you actually hold the corpus for.

That is the layer we have spent the better part of the last year building for law specifically, and it is the least flashy of the three by a wide margin. It does not produce a clean single number the way a confidence score does, and it does not come with a certificate the way disclosure compliance does. What it produces, when it is built honestly, is something narrower and harder to fake: a claim that either matches a real source or is flagged as not checkable at all, with nothing smoothed over in between.