SWE-Critix — Verifier that returns a label and a rationale — What does it mean?

The news. On October 9, 2026, researchers from Huawei and Monash University Malaysia posted SWE-Critix (arXiv 2610.13356), a recipe for training verifiers of coding-agent runs. They trained 30B-parameter models on trajectories from the CodeForge-Preview dataset and report a preliminary result on a held-out test set: 57% accuracy for the base Qwen3-30B-A3B model, 66% after SFT and 78% after reinforcement learning. Models, datasets and training code are released. Read the paper →

Picture a referee asked to rule on a match played over hundreds of turns, with no scoreboard in the stadium. That is the position of anyone who has to grade a coding agent: the agent read files, ran commands and edited code for hundreds of steps, and often there is no test suite that says whether the task is done. SWE-Critix trains a model to be that referee: it reads the whole trajectory and returns a pass/fail label together with a written reason.

The recipe has two stages, and the scoreboard comes back in the second one. In stage one, a stronger teacher model (GPT-5.5 in the paper) is shown two trajectories for the same issue, one that passed and one that failed. The teacher compares them and explains what separated them: a summary of each procedure, the mistake patterns from a fixed list, and the causes of success or failure. Comparing a pair is easier than judging one run alone, so the paper uses pairs and then rewrites the notes so each trajectory can be judged by itself. These notes become 10,390 SFT examples, only 8.7% of the training data.

In stage two, the other 91.3% of the data carries no written notes, only the unit-test outcome. The model writes its own reasoning and a verdict. The reward is +1 if the verdict matches the unit tests and −1 if it does not, and the scoreboard supplies only the RL training reward: when the verifier is deployed, no tests are needed. The paper optimizes this with GRPO, sampling 8 answers per prompt.

The accuracy numbers need the class counts to mean anything. The test set holds 43,445 trajectories: 28,741 that passed their unit tests and 14,704 that failed. A lazy referee who always says pass would score 28,741 ÷ 43,445 = 66.2% (my arithmetic from the paper's Table II counts; the authors do not report this baseline). The base model caught only 51 of the 14,704 failures. The SFT model catches 9,181 of them (62%) but accepts only 19,340 of the 28,741 passing runs (67%), so its 66% accuracy merely ties the lazy referee while finally doing the job of flagging failures. After RL it accepts 24,499 passing runs (85%) and flags 9,426 failures (64%), giving 33,925 correct rulings out of 43,445, or 78%, about 12 points above always saying pass.

Three limits keep this a preliminary result. The paper scores only the label, not the written reason: the RL reward checks the verdict alone, and a qualitative review of the rationales is listed as future work. Second, failures are still the weak side, with 64% of failed runs caught; the authors suggest a possible cause is the variety of failure patterns and their subtle signals. Third, the gain is uneven across data sources: on the R2E-Gym subset the SFT model scored 52%, below the base model's 60%, and RL lifted it to 62%. Whether RL alone would work is also open, since the cold-start baseline without annotation and SFT is listed as future work. Each model was evaluated with one pass per trajectory.

This is the same trade the curriculum covers in pass/fail versus a score: one number tells you whether a run worked but not why. A written reason is what lets a team do error analysis without reading every trace by hand, and a verifier that runs on live traffic is the kind of check discussed under online versus offline evals. A verifier is only as trustworthy as its measured miss rate, which is why the negative-recall figures matter more than the headline accuracy.

Goes deeper in: AI Agents → Evals & Diagnostics → Pass/Fail vs Score