What is verdict-preserving unfaithfulness?

An incorrect formal encoding of a problem that still executes successfully and returns the verdict the correct encoding would have returned. Because the verdict is right, every check downstream of the verdict reports success. The paper abbreviates it **VPU**.

Why can't the solver catch it?

The solver only receives the encoding, never the original problem, so it has nothing to compare against. It correctly answers the question it was handed. The paper proves the stronger version of this: any heuristic built on the verdict is **bounded to chance-level detection**, because a VPU trace is defined by having the expected verdict.

What does GenV score instead?

**Reference-equivalence** — whether the encoding means the same thing as a designated reference formalization. GenV distills an offline Z3-equivalence oracle into a continuous, reference-free score emitted through the model's vocabulary space, reporting **0.961 AUROC** for the oracle-mined GenV+HN variant.

Does this only matter for formal math?

The proof is about autoformalization, but the shape is general: whenever an agent translates a task into something a checker can execute — a SQL query, a test, a policy rule — the checker validates the translation, not the intent. Any check computed from a passing result inherits the same blind spot.

What does the 11.3-point gain refer to?

Downstream accuracy in agentic **test-time compute allocation**: using GenV's score to decide which traces deserve additional reasoning, rather than spreading the budget uniformly. The score is useful because it ranks, not because it decides.

Agent· 2026-09-17

GenV catches formal encodings that pass the solver but changed the question — Verdict-preserving unfaithfulness — What does it mean?

TL;DR

What is it: Verdict-preserving unfaithfulness (VPU) is the failure mode named by Beyond Solver Verdicts, posted on September 10, 2026: a formal encoding of a problem that executes cleanly and returns the verdict everyone expected, while no longer standing for the problem it was supposed to encode.
Why it’s needed: Neurosymbolic agents bolt a solver onto a language model precisely because the solver cannot bluff, so any error that survives the solver turns the pipeline's one trustworthy signal into a rubber stamp — and the paper proves that checks built on the verdict cannot do better than a coin flip at catching this class.
vs previous: The previous approach was verdict-only verification: did the encoding execute, is it structurally well formed, did the answer match. GenV replaces that pass/fail bit with a continuous reference-equivalence score — asking not whether the solver was happy but whether the encoding still means what the original said.

Jargon

Autoformalization: Turning a problem written in informal language or ordinary math into a formal statement a solver can execute. It is a translation step, and like any translation it can be fluent and wrong at the same time.
Neurosymbolic system: A pairing of a language model with a symbolic engine — a theorem prover, an SMT solver, a constraint engine — so that the reasoning is checked mechanically instead of trusted.
Z3 / SMT solver: Z3 is a widely used SMT (Satisfiability Modulo Theories) solver: it decides whether a formal statement can be satisfied. It reasons only about the statement it is handed, never about where that statement came from.
Reference-equivalence: Whether the model's encoding means the same thing as a designated reference formalization. A solver verdict cannot establish this, because the solver only ever sees one of the two.
VPU: Verdict-Preserving-Unfaithfulness. The paper's name for an encoding that is wrong and returns the expected verdict — the two properties that usually rule each other out.
AUROC: Area Under the Receiver Operating Characteristic curve. For a score that ranks suspects, it is exactly the probability that a randomly chosen bad item outranks a randomly chosen good one: 0.5 is a coin flip, 1.0 is perfect.
Test-time compute allocation: Spending extra reasoning at inference on the cases that need it, rather than uniformly. A ranking score is what tells you which cases those are — see when to spend more tokens.

The news. On September 10, 2026, researchers posted Beyond Solver Verdicts: Generative Reward Models for Autoformalization (arXiv 2609.11085, revised September 11). It names a failure mode in neurosymbolic systems — verdict-preserving unfaithfulness — proves that verdict-only verification heuristics are mathematically bounded to chance-level detection on it, and introduces GenV, which distills an offline Z3-equivalence oracle into a score the language model simply emits, as an ordinary token, instead of a verdict. The oracle-mined variant (GenV+HN) reports 0.961 AUROC and an 11.3-point downstream accuracy gain when an agent uses the score to decide where to spend more reasoning. Read the paper →

Picture the notary. You hand over a contract in English and ask for a French copy to file. The translator works. The notary reads the French copy, confirms it is a coherent, properly formed contract, and stamps it. The stamp is genuine, the filing is valid, and the file is now closed. And the clause you cared about says something else, because nobody in that room ever put the two documents side by side.

That is the shape of a neurosymbolic pipeline. A language model autoformalizes your problem into a formal statement; Z3 or another solver executes that statement and returns a verdict; everything downstream treats the verdict as ground truth, because the entire reason to bolt a solver onto a language model is that the solver cannot bluff. It genuinely cannot bluff — and it also cannot see your original, so it answers exactly the question it was handed, correctly, whether or not that was your question. The paper's name for an encoding that earns the right verdict while meaning the wrong thing is verdict-preserving unfaithfulness.

Your probleminformal, what you meant

Autoformalizethe model writes the encoding

Solver runssees the encoding only

Verdict matchesthe answer you expected

Acceptednothing compared step 1 to step 2

The obvious defence is to look harder at the run: did it execute, is the structure well formed, did the verdict match the expected answer? The paper's theoretical result is that this whole class — verdict-only verification heuristics — is bounded to chance-level detection on these traces. That is not a complaint about weak implementations. A verdict-preserving failure is, by construction, one that produces the right verdict, so any signal computed from the verdict carries no information about it. You cannot catch a forged document by checking that it is internally consistent. It is the same trap as grading an agent on whether it finished instead of on what it did: a bit tells you nothing, and a score is what exposes the difference.

GenV changes what is being compared — not the encoding against the solver, but the encoding against what it was supposed to mean. That property is reference-equivalence, and GenV learns to judge it by distilling an offline Z3-equivalence oracle into the language model itself. The score comes out of the model's native vocabulary space: the verifier emits a token, and the probability mass on that readout is a continuous suspicion score rather than a verdict. It is reference-free at inference — you do not need the reference formalization in hand to score a fresh trace — which is what lets it sit in a live pipeline. The paper also reports that this readout localizes where the encoding went wrong without ever being trained to localize anything, and that it transfers zero-shot to translators and formal styles it never saw.

Check	What it compares	Catches VPU?
Solver verdict	the encoding against itself	No — by construction, the verdict is the thing VPU preserves
Verdict-only heuristic	structure, execution, answer match — all downstream of the verdict	No — proven bounded to chance-level detection
Re-reading the encoding yourself	encoding against your intent	Yes — but it is the manual step the solver was added to avoid
GenV reference-equivalence score	encoding against a reference formalization	0.961 AUROC for GenV+HN, the oracle-mined variant (paper)

Worked example: one budget, two orderings

Hold three things fixed (illustrative): 1,000 finished traces, every one of them solver-valid; 50 of those encodings are verdict-preserving-unfaithful; and you can afford to re-examine 100 of them, by hand or by spending extra reasoning. The only thing that varies is the order you walk the list in.

With a verdict-only heuristic, the paper's bound pins you at AUROC 0.5 — your list is in effectively random order. Re-examining 100 out of 1,000 then surfaces about 5 of the 50 bad traces, and the other 45 ship. GenV+HN, the oracle-mined variant, reports 0.961 AUROC, which is exactly this statement: draw one unfaithful trace and one faithful one at random, and the score ranks them correctly 96.1% of the time. The odds that you look at the bad one first move from even money to roughly 25 to 1 (96.1 ÷ 3.9). Nothing about the solver, the encodings, or the budget changed between those two worlds — only the order — and that ordering is what the paper's 11.3-point downstream accuracy gain is bought with.

Goes deeper in: AI Agents → Evals & Diagnostics → Pass/Fail vs Score

Related explainers

MaxProof's defense-in-depth generative verifier — what a generative verifier looks like when it is stacked in layers rather than scored once.
Verification as a scaling axis — the case that more verification buys more accuracy; GenV is the footnote about what it must verify.
Agent leaderboards and predictive validity — the same disease one level up, where the benchmark passes but the deployment does not.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based