LLM·

Late Entry speeds up looped language models — Late-entry decoding over a deep KV cache — What does it mean?

The news. On November 18, 2026, Ziqi Jin, Jiaxi Li, Weiwen Xu and Aixin Sun posted Late Entry in Looped Language Models (arXiv 2611.24576). On Ouro-2.6B with four passes, two-pass late entry plus Wake verification decodes GSM8K 5.2× faster than sequential full-depth decoding while staying greedy-exact, 26% above their own lossless block-decoding baseline. On Huginn-3.5B at 64 passes the exact speedup reaches 8.4×, and the lossy mode reaches 10.6× to 12.5×. All numbers are single-request runs on one H100. Read the paper →

Picture the tower. Every token has to climb four floors, and on each floor it reads the notice board, where every earlier visitor left a note from that same floor. That is a looped model: one shared transformer stack run T = 4 times, with a separate cache for each pass. Climbing all four floors for every token means four serial forward calls per token. The obvious shortcut, early exit, stops the newcomer on floor 2, and then floors 3 and 4 never get its note, so every later visitor reads boards that only hold shallow notes. Running the remaining passes in the background does not fix it either: under the paper's pass-aligned routing, passes 1 and 2 only ever read boards 1 and 2, so the early-exit prediction stays bit-for-bit unchanged.

Late Entry flips the choice. The newcomer takes the express elevator to floor 3, runs passes 3 and 4, and reads boards that earlier tokens filled at full depth. The catch is the lobby clothes. A raw token embedding has a median size (ℓ2 norm) of 0.76, while pass 4 expects layer-normalized states with norms of 37 to 51. Fed straight in, the unadapted Ouro-2.6B agrees with full depth on only 75.1% of next tokens, worse than truncating the whole model to one pass (94.9%). A short fine-tune that shows the late passes raw embeddings over full-depth caches lifts one-pass late entry to 96.2%. And the paper's key comparison follows: one late pass over a four-pass context (96.2%) beats two early passes over a two-pass context (95.4%), because the deep cache already holds the reasoning that the skipped passes would have redone. The keys and values must come from the same pass, though: pairing pass-1 keys with pass-4 values drops agreement to zero.

Someone still has to climb the stairs, or the boards on floors 1 and 2 stay empty for the next visitor. Late Entry has each accepted token rerun its true passes 1 to 4 as extra rows inside the next steps' forward calls, so finishing the deep cache adds no serial call, only arithmetic. Each step has K = 2 calls and the same weights serve every pass, so one call can carry the new candidates at pass 3 and an older token at pass 1 side by side; a token's full climb finishes ⌈T/K⌉ = 2 steps later, and causal order is kept because earlier tokens always write a pass before later tokens read it. Those extra rows are cheap because single-request decode is memory-bound: once the weights stream in from HBM (the GPU's main memory) for a call, more rows ride along. On an H100, forwarding 32 tokens took only 9% longer than forwarding one.

Prefill vs Decode on the Roofline

Decode~1 FLOP/bytePrefill~100 FLOP/byte← memory-boundcompute-bound →

Same GPU, fundamentally different bottlenecks

The stair walk also produces the answer key. When a token reaches the roof, its finished four-pass state is projected through the same output matrix multiply the step already runs, and the model compares that full-depth prediction with what the express rider guessed two steps earlier. This roof check, Wake verification, keeps a token only if both agree, and otherwise rolls back from that token, the same accept-or-roll-back rule speculative decoding uses. The tolerance τ sets how strict the inspector is: τ = 0 gives full-depth greedy output, τ = ∞ never rolls back. Strictness costs little when the model is sure: on the 68% of GSM8K tokens where full depth is at least 95% confident, two-pass late entry already agrees 100% of the time, and most of the remaining misses sit in the 6% of tokens where full depth is below 50% confident.

Strategy (Ouro-2.6B, T = 4, GSM8K)Passes the new token runsDepth of past tokens' cacheResultSource
Sequential full depth1 → 44baseline (1×), 87.3% accuracypaper, Table 3
Block decoding, K = 41 → 4 per step44.05×, losslesspaper, Table 3
Early exit, K = 2, one token at a time1 → 2274.9% accuracypaper, Fig. 2c
Late entry, K = 2, one token at a time3 → 4485.3% accuracy (+10.4 points)paper, Fig. 2c
Late entry K = 2 + Wake, τ = 03 → 4, then 1 → 4 in later steps45.2×, greedy-exactpaper, abstract
Late entry K = 2 + Wake, τ = ∞3 → 4, then 1 → 4 in later steps46.8×, lossy; +16.9 points over trained early exit at matched speedpaper, abstract

Worked example (paper's settings: Ouro-2.6B, T = 4, block size B = 8, one H100). Hold the model, the block size and the GPU fixed. Full-depth block decoding runs 4 serial calls per step and reaches 4.05× over sequential decoding on GSM8K. Late entry with K = 2 runs 2 serial calls per step. Each call carries the active candidate block, which the paper sizes at (1 + 8)² = 81 rows, plus at most two catch-up groups of up to 9 tokens each, so at most about 99 rows per pass. The H100 stays memory-bound up to roughly 348 rows per pass, so the extra rows barely move the call time. Halving the serial calls is a ceiling, not a result: Wake measures 5.2× greedy-exact, which the paper reports as 26% above K = 4 block decoding in the same frontier run (the 4.05× above comes from a separate table run, so the two do not divide to exactly 26%). Deeper loops leave more to skip: on Huginn-3.5B at T = 64 with K = 16, the same recipe reaches 8.4×, 153% above 64-pass block decoding.

The speedup depends on spare memory bandwidth, so it shrinks under load. With 1 to 32 concurrent requests in continuous batching, full-depth B = 8 block decoding falls from 3.90× to 0.77×, which the paper traces mainly to the 81-row candidate block rather than to late entry itself; late entry's gain over block decoding at the same block size holds steadier, from 1.65× to 1.45×, and shrinking the block with load keeps 2.31× decode at 32 requests. Two more limits: the exactness proof covers greedy decoding only, and shallow loops roll back often, so at T = 8 with K = 2 the exact mode (1.96×) is slower than plain block decoding (2.86×). For T of 16 and above, rollbacks fall to 0.26 to 2.2 per 100 tokens. The method also needs the short fine-tune and runs all three τ modes in one static CUDA graph. The general lesson reaches beyond these two models: in a looped model, a deep cache can stand in for token-local depth, so the place to save passes is at the start of a token's climb, not at the end.

Goes deeper in: LLM Internals → KV Cache → The KV Cache Solution

Related explainers

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based