Constrained decoding is not the culprit — Format, validity, and reasoning constraints — What does it mean?

The news. On October 26, 2026, Nguyen and colleagues at Télécom Paris and Nokia Bell Labs posted a study of constrained decoding across four model families and eleven open-source models, from 1.7B to 32B parameters. They compare matched prompts and output formats and report that the reported accuracy loss from JSON-style output mostly disappears once those are held fixed. Read the paper →

Picture the exam answer sheet from the card above. A constrained decoder is a sheet where some bubbles are struck out before the student picks one. The student still chooses by their own judgment; the sheet only removes choices that were never valid. In a language model, the same thing happens inside the step that turns logits into probabilities: the decoder sets every disallowed token to zero and rescales the rest so they sum to 1.

The study separates three kinds of struck-out bubble, because they remove different things. What a constraint removes decides whether it can hurt. A format constraint removes text outside the structure, a validity constraint removes answers the task already excludes, and a reasoning constraint removes some of the model's own thinking paths.

The older finding that JSON mode lowers accuracy is real, and the paper reproduces it. The cause is that JSON mode usually changes the prompt and the output space at the same time. Asking a model to reason inside a schema is a different task from asking it to reason freely and then answer, so the two runs are not equivalent. When the paper fixes the prompt and the expected output format, constrained decoding matches or beats unconstrained decoding. The authors also give a short argument for why: if the mask removes only invalid continuations, the correct answer keeps its probability and divides by a total of at most 1, so it cannot lose. This is also why structured outputs work well in practice when the schema asks for the answer and nothing more.

Now run the sheet with numbers (illustrative). A model picks one answer letter when sampled. It gives B, the correct letter, 0.40; wrong letters A, C and D get 0.20, 0.05 and 0.05; and chatty openers such as "Sure" or "The" take the remaining 0.30. Unconstrained sampling is correct 40% of the time. A format constraint that bans the openers keeps 0.70 of the mass, so B rises to 0.40 / 0.70, about 57%. A validity constraint that also rules out A and D, because the question excludes them, keeps 0.45, so B rises to 0.40 / 0.45, about 89%. The model never learned anything new; the sheet only stopped it from spending probability on answers that could not be right.

The margin rule is the risky one. Forcing reasoning to begin with "Let's think step by step" generally helped on GSM8K and AIME2026 in the study, yet it cost 6.6% on AIME2026 for Qwen3.5-4B. A reasoning constraint only helps if the path it forces is a good path, and finding that path in advance usually needs knowledge you do not have. That is the same trade as the reasoning-budget step: more structure on the thinking is useful only when it matches the problem.

There is also a price on the compute side. The paper notes that constrained decoding must work out the allowed tokens and apply a mask at every step, and that for JSON this overhead starts at the first token and grows with grammar complexity. InWriting avoids most of it by constraining only the final answer. In the authors' runs it added fewer than 15 tokens while keeping the reasoning free. The study does not report serving-level latency, so treat the cost as a design pressure and not a measured number.

Goes deeper in: LLM Internals → Text Generation → From Logits to Probabilities