LongZero trains long-context reasoning from zero data — answer-first synthesis — What does it mean?

The news. On October 21, 2026, a Tsinghua University team posted LongZero (arXiv 2610.30132), a method that improves a model's long-context reasoning with reinforcement learning on tasks the model writes for itself, with no external corpus and no stronger teacher model. It was tested on two Qwen3 models across five long-context benchmarks. Read the paper →

Picture a puzzle designer making a maze. A designer who builds the maze first and hides the treasure afterwards must walk the whole maze to prove the treasure can be reached; a designer who buries the treasure first and builds corridors around it already knows the way. LongZero uses the second order. Before any long text exists, the model being trained writes a short JSON specification of a fictional world: named entities, atomic facts that chain into multi-hop questions, the answer to each question (a name, place, product, title or number that appears verbatim in a fact), and, for every question chain, a few distractor facts that involve the same bridge entity but differ in role, time or attribute. The spec also forces at least two people to share a surname and at least two organizations to have confusably similar names.

Only then does the model expand the spec. It assigns every fact to one of eight documents in different genres, spreads each question's supporting facts across different documents, and places each decoy away from the fact it imitates. Each document is written independently and states each assigned fact exactly once. A plain program shuffles the documents into one context and applies a faithfulness gate: a question survives only if its answer, or an alias, appears verbatim in the expanded text. Because the answer was fixed before the long text existed, the main risk moves to the expansion, and a string match catches the commonest failure, a dropped or altered answer fact. The model that cannot yet reason over long text reliably is never asked to verify a long text it just wrote. The check is not perfect: the paper's appendix shows a defective question, with contradictory documents, that still passed filtering. The finished question then asks the trained model to attend to the few relevant tokens among tens of thousands and to reject near-duplicates, the kind of trouble described in the failure modes of a crowded context.

The second task type is retrieval, in the style of MRCR. The model writes several responses to each request such as "write a poem about tigers", using only a small set of formats and common nouns taken from the public OpenAI MRCR dataset. A program inserts N of those responses, the needles, at random positions among responses to other requests until the conversation reaches the target length, then asks for the k-th needle prefixed by a random string s. The program, not a model, decides which response is the k-th, so the reference answer is correct by construction. The reward is 0 unless the answer starts with s; otherwise it is a similarity ratio (Python difflib SequenceMatcher) between the rest of the answer and the needle, which gives partial credit for a nearly correct copy.

A correct question is still not a useful one. LongZero trains with GRPO, which scores each of G = 8 answers to a prompt against the group: the advantage is the reward minus the group mean, divided by the group standard deviation. If all eight answers are right, or all eight are wrong, every advantage is zero and the question contributes no gradient. So the same model answers each question in small waves and keeps it as soon as it has produced both a correct and an incorrect answer. The paper also reports that consistently failed questions were often defective, for example two documents giving conflicting values for one attribute, so the filter doubles as a quality check. Training then mixes 2,304 reasoning questions with 1,152 retrieval conversations (a 2:1 mix) for 72 steps on Qwen3-4B-Thinking-2507 and 108 steps on Qwen3-8B, with 64 prompts and 8 responses per step.

Worked example with the paper's synthesis table for Qwen3-4B-Thinking-2507: it generated about 14.4K questions, of which 69.3% were always answered correctly, 13.0% always incorrectly and 17.7% unstable, about 2,550 questions, which is the 2,560 it needed. Qwen3-8B kept only 13.1% of about 19.5K, so it had to synthesize more raw data for the same 2,560. A GRPO check (illustrative): if 3 of 8 answers are right, the mean reward is 0.375 and the standard deviation about 0.48, so each correct answer gets an advantage near +1.3 and each wrong one near −0.8; with 0 or 8 right, every advantage is exactly 0.

The gains are real but modest, and the setup limits how far to read them. Averaged over five benchmarks, LongZero adds 2.74 points to the Qwen3-4B-Thinking-2507 base and 1.03 points to Qwen3-8B, and it beats the teacher-written baseline by 1.43 and 0.83 points respectively, although that teacher is a far larger model. On the 4B model it is best on four of five benchmarks (AA-LCR is the exception); on the 8B model it is best on three, ties on AA-LCR and trails on LongBench v2. Qwen3-8B natively supports only 32K tokens, so the paper evaluates it with YaRN at a scale factor of 4 and no extra training, and its AA-LCR base score is 2.50, close to the floor where a half-point change is hard to interpret. All methods share one RL configuration and checkpoint step, and scores are the mean of two runs.

Two design choices explain why the comparison is clean. Fictional worlds remove the shortcut of answering from parametric knowledge and the risk that public text overlaps with a benchmark. And because each model writes and filters its own data, the kept questions adapt to that model. What the evidence does not show: only two small models were tested, so whether the recipe holds for larger ones is not established by this paper.

Goes deeper in: AI Agents → Context Engineering → The 4 Failure Modes