What is Adaptive Entropy-Sparing (AES)?

AES is an RL post-training method for reasoning models, from a November 2026 paper. During GRPO training it lowers the advantage of **high-entropy tokens** (hesitation points like "wait") only when they appear in a correct answer and come after the length of the shortest correct answer in the same sampling group.

Why does a uniform length penalty hurt accuracy?

It docks every token of a long answer equally, so it also punishes the reasoning steps the answer needed. The paper calls this **credit confounding**; on R1-Distill-Qwen-1.5B a uniform-penalty baseline halved answer length but lost 2.0 accuracy points, while AES halved it and gained 3.8.

Why use token entropy to find wasted reasoning?

Entropy measures how unsure the model was when it picked a token. The paper found that the highest-entropy tokens are mostly hesitation words such as "maybe" and "wait", which start new checking branches, and that cutting the last of those branches saved 26.8% of tokens for a 0.2-point accuracy drop.

How does AES differ from difficulty-aware methods like When2Think?

Difficulty-aware methods decide how much to think for each problem. AES works inside one answer: it keeps the full reward for most tokens and only pushes down the late hesitation tokens, using a correct sibling answer from the same group to decide where "late" begins.

LLM· 2026-11-11

AES paper — Entropy-sparing vs uniform length penalty — What does it mean?

Jargon

Large reasoning model: A language model trained to write a long chain of thought before its final answer, such as DeepSeek-R1 and its distilled versions. The paper tests the 1.5B and 7B R1-Distill-Qwen models.
Overthinking: Writing many more reasoning tokens than the answer needs, usually by re-checking a result that was already correct.
RLVR: Reinforcement learning with verifiable rewards: the model is rewarded when a checker confirms its final answer is right. Full explainer
GRPO: Group Relative Policy Optimization. The trainer samples a group of answers to the same question (8 in this paper) and scores each one against the group's average, so no separate value model is needed.
Advantage: How much better one sampled answer did than its group. A positive advantage makes every token in that answer more likely next time; a negative one makes them less likely.
Token entropy: A measure of how unsure the model was when it picked one token, computed from its probability distribution over the vocabulary. High entropy means several next tokens looked plausible.
Credit confounding: The paper's name for what a uniform length penalty does: it cannot tell the tokens the answer needed from the ones that only added length, so it punishes both.
T_ref: The length of the shortest correct answer in a sampling group. AES uses it only as a boundary, not as a target to copy.

The news. On November 9, 2026, Cong Jiang, Xiaofeng Zhang, Tom Ko and Zheng Zhang (Harbin Institute of Technology, Shenzhen, and Shenzhen Loop Area Institute) posted Adaptive Entropy-Sparing for Efficient Reasoning; the arXiv listing notes it was accepted at NeurIPS 2026. They fine-tuned DeepSeek-R1-Distill-Qwen-1.5B and -7B with GRPO on the DAPO-Math-17k dataset. Across eight benchmarks, AES cut average answer length by up to 52.9% while raising mean accuracy by up to 3.8 points over the base models. Read the paper →

Picture eight hikers sent up the same mountain, where the goal is to reach the summit by the shortest route. Most of them get there. For illustration, the fastest arrives after two kilometres; another arrives after four, because at every fork it stopped, doubted itself and walked a side trail to make sure. A judge who simply docks points per kilometre punishes that slow hiker for the whole route, including the first two kilometres that every successful hiker also had to walk. That is what a uniform length penalty does to a reasoning trace: the trainer adds a term like reward minus a constant times length, and the constant applies to every token.

The paper calls this credit confounding, and it measured why it matters. With R1-Distill-Qwen-7B on MATH-500, the authors cut traces at the high-entropy "transition points" and let the model answer from what was left. Dropping only the final segment saved 26.8% of reasoning tokens and moved accuracy from 92.8% to 92.6%. Most of the waste sits at the end of the trace, not spread across it, so a penalty spread across it is aimed badly.

How do you find the forks without reading the trace? The model leaves a signal. Before each token, the model produces a probability distribution over every possible next token, and the entropy of that distribution says how unsure it was. When the authors marked the top 5% highest-entropy tokens in each trace, the most frequent words were not maths terms but hesitation markers: "maybe", "perhaps", "wait", "check". A high-entropy token is where the hiker stood at a fork. Under a simple branching model with stated assumptions, the paper derives that the expected remaining length scales with the exponential of that entropy, which motivates entropy as a proxy for where long continuations start; it does not prove that every spike leads to waste.

AES then needs a line that separates a useful fork from a wasted one, and it takes that line from the group itself. GRPO already samples 8 answers per question; AES finds the shortest correct one and calls its length T_ref. Only tokens that are high-entropy, sit in a correct answer, and come after position T_ref lose advantage. Forks before T_ref are spared, because a correct answer of that length proved some exploring was needed. Wrong answers are not touched, so the model is never pushed to be brief before it can be right. If no answer in the group is correct, AES switches off for that question.

Sample 8 answersGRPO group, same question

Find T_reflength of shortest correct answer

Mark forkstop 10% entropy in each answer

Dock late forkscorrect answers, past T_ref only

Here is the arithmetic for one correct but long answer (illustrative numbers; the formula and the coefficient 2.0 are the paper's). Say T_ref is 2,000 tokens and this answer runs to 4,000. Its GRPO advantage is +0.58, the value you get when 6 of 8 answers are correct (population standard deviation). AES subtracts 2.0 × η × λ(t) from a high-entropy token at position t, where η is this answer's excess length scaled to between 0 and 1 within the group (take the longest, η = 1) and λ(t) = ln(1 + (t − 2,000) / 2,000). A fork at token 2,100 gets λ = 0.05, so its advantage falls only to +0.48: still reinforced. A fork at token 3,000 gets λ = 0.41, so its advantage becomes 0.58 − 0.81 = −0.23: now discouraged. At least 90% of the tokens in the same answer keep the full +0.58, so the reasoning that found the answer is still rewarded while the late "wait, let me check" is trained away.

Back on the mountain, the comparison the paper cares about is two judges cutting the same amount of walking. On the R1-1.5B maths benchmarks, the uniform-penalty baseline (Efficient Reason) and AES shorten answers by almost the same share, about half, but the uniform penalty loses 2.0 accuracy points while AES gains 3.8. The ablation test, which removes one part of the method at a time, points the same way: on MATH-500 with R1-7B, penalizing a random set of tokens of the same size gave 88.5% accuracy against 91.7% for full AES, even though the random version produced shorter answers. Which tokens you dock matters more than how many.

Method (R1-Distill-Qwen-1.5B)	What the penalty targets	Mean accuracy, 4 maths benchmarks	Mean answer length (tokens)
Base model (Table 1)	Nothing	64.9%	6,467
Efficient Reason (Table 1)	Every token, through a uniform length term in the reward	62.9%	3,123
LC-R1 (Table 1)	Length plus a compression reward	63.7%	3,130
ThinkPrune (Table 1)	Training under a hard token budget	66.7%	3,242
Laser (Table 1)	Length, scaled by problem difficulty	68.2%	3,794
AES (Table 1)	High-entropy tokens past T_ref in correct answers	68.7%	3,048

Two limits are worth keeping in view. The paper tests one model family, the R1-distilled Qwen models, mostly on maths, with LSAT and GPQA as the out-of-domain checks; it describes its results as an average improvement in accuracy and length, not a full analysis of every accuracy-for-length trade-off. And the entropy-length link is a modelling argument with stated assumptions, not a proof that every late fork is waste. What carries over is the design rule: when you want shorter reasoning, decide which tokens are waste before you penalize length, and use a correct sibling answer to say where waste begins. For a serving team that matters directly, because every output token is a decode step the server pays for, and a model that knows when to stop spends its reasoning budget where it changes the answer.

Goes deeper in: AI Agents → Planning & Reflection → When to Spend More Tokens

Related explainers

When2Think: difficulty-aware Think vs NoThink routing — decides how much to think per problem; AES decides which tokens inside one answer were wasted.
RATIO: overthinking after 3-bit quantization — the same "wait" and "hmm" tokens, fixed at decoding time instead of in training.
PKTP: pass@k-optimal thought pruning — cutting reasoning paths at inference time rather than teaching the model to write fewer.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based