Magenta closes informal math reasoning with Lean-guided repair — Error-attribution routing — What does it mean?
The news. On September 10, 2026, a paper introduced Magenta, a training-free pipeline that starts from a maths problem written in ordinary language and ends at a machine-checked Lean 4 proof. Nothing is fine-tuned; the whole system is orchestration around an existing reasoner. The paper reports 100% accuracy across its evaluated AIME 2025, AIME 2026 and HMMT February 2026 sets (high-school olympiad qualifiers), and says all six IMO 2026 problems were solved when the pipeline was paired with the open-weight K2-Horizon-7B model. The mechanism that carries beyond mathematics is what Magenta does when the checker says no. Read the paper →
Picture the triage desk in a busy clinic. A patient comes back still unwell. The cheap, thoughtless move is to hand them a new ticket and point at the end of the queue — same intake, same questions, same wait, and a coin flip on whether this round goes better. The triage nurse does something else: reads the chart, names what actually went wrong, and sends the patient to the one place that can fix that one thing. A dressing that came loose is retaped at the desk. A diagnosis that was wrong goes back to the specialist.
Most agent retry loops are the ticket machine. A verifier — a test suite, a compiler, a proof checker — says no, and the harness responds by sampling a fresh attempt from scratch. That is best-of-N dressed up as a retry policy: every attempt is independent, so the run costs N full attempts and the (N+1)-th knows nothing the first N learned. It works, in the sense that enough lottery tickets eventually win, and it is the reason retry budgets get large.
Magenta puts a nurse at the desk. When the Lean proof fails to compile, an error-attribution judge reads the failure and classifies its cause before anything is retried, and the class decides where the retry goes. A step that does not follow is a local repair: patch that step, leave the rest of the proof standing. A derivation that was wrong in the first place is a referral: throw the proof away and redo the mathematics. The two repairs cost wildly different amounts, which is the whole point of telling them apart.
There is a second judge in front of the first, and it guards a failure the checker cannot see. Formalizing a problem into Lean can quietly change what is being asked — drop a constraint, widen a bound — and Lean will happily certify the altered claim. A proof that compiles is only evidence if the statement it proves is still your problem, so a statement judge checks the translation before the proof is trusted at all. That is the same instinct as asking what a passing eval is actually measuring.
The reason a verifier can support this at all is that it returns more than a verdict. A proof checker does not say "no" — it says no at line 41, naming the step it could not close. A compiler names the type it expected. A test suite names the case that failed. That detail is a diagnosis, and a retry policy that only reads the pass/fail bit is throwing it away, which is exactly why choosing a tool with a legible failure signal matters more than choosing a strict one.
| What broke | Which check catches it | Where the retry goes |
|---|---|---|
| The statement | The statement judge — the formalization no longer means the original problem | Re-formalize. The proof, right or wrong, is answering a different question |
| The mathematics | The error-attribution judge — the informal answer was wrong before Lean ever saw it | Re-derive. A full redo, the expensive branch |
| One proof step | The error-attribution judge — the maths holds but a step does not compile | Local repair. Patch that step, keep the rest |
Work the cost through, holding two numbers fixed. Say a full attempt — answer, formalize, prove — costs 12,000 tokens, and a local patch to one rejected step costs 1,500 tokens (illustrative; the paper publishes no per-stage costs, and the four-in-five split below is illustrative too). Now take five rejections, and say four of them are broken proof steps rather than broken mathematics. Blind resampling cannot tell those apart, so it pays the full price every time: 5 × 12,000 = 60,000 tokens. Routing pays 4 × 1,500 for the patches and 12,000 for the one real re-derivation: 6,000 + 12,000 = 18,000 tokens, or 70% less for the same five repairs — and note where the saving came from. Not a better prover, not a bigger model: a classifier. Which is also where it breaks. Misroute a mathematics failure into the patch branch and you spend the 1,500, still fail, and pay the 12,000 anyway — 13,500 for an attempt blind resampling would have closed for 12,000. Routing is only cheaper while the judge is right, so the judge's accuracy, not the prover's, is the number to watch.
Mathematics is the convenient case, because Lean gives a failure signal that is both unambiguous and richly located. Most production agents have a weaker one — a test name, an HTTP status, a reviewer's comment — and the classification gets correspondingly harder. But the shape transfers. If your harness already distinguishes a transient tool error from a bad plan, you are doing a two-class version of this, and every class you can name reliably converts some full retries into cheap ones. The discipline is to stop treating "it failed" as one event, which is the same move as naming the cause before the fix in an incident. The honest caveat is that a misclassification is worse than no classification, so a judge you cannot measure is not an upgrade.
Goes deeper in: AI Agents → Planning & Reflection → When to Retry
Related explainers
- ToolFailBench separates skip, ignore, and fabricate failures — the taxonomy side of the same idea: you can only route a failure you can name.
- MaxProof clears IMO/USAMO gold — what happens when you layer the verifier itself rather than the repair.