Parallel Language Models paper — Latent auxiliaries for multi-token decoding — What does it mean?

The news. On October 19, 2026, a paper by authors at UC Irvine, ETH Zurich and other institutions introduced Parallel Language Models. The authors report that models from 80M to 3B parameters came close to autoregressive pretraining loss, and that a post-trained 1.3B model matched or beat an identically post-trained autoregressive model on math, coding and instruction-following benchmarks, at roughly 2× to 6× the decoding speed in their setup. Read the paper →

Picture a relay race. Every leg must wait for the baton, so four legs take four legs of time however many runners you hire. A standard autoregressive model is that relay: each forward pass hands over exactly one token, and the next pass cannot start until the baton arrives. The pass is expensive because producing even one token means streaming the model's weights from memory, the bandwidth-bound regime described in Prefill vs Decode and in the KV cache module.

The obvious shortcut is to send four runners at once. It fails for a specific reason: if each position predicts its own token independently, nothing makes the positions agree. After a prompt that could continue as "New York City" or "San Francisco Bay", independent positions can produce "New Francisco City" (illustrative). Each position looked plausible alone, and the pair is broken. The dependencies between tokens are exactly what left-to-right order was carrying.

Now swap the relay for synchronized swimmers. Before the routine starts, every swimmer is handed a cue card. Given its card, a swimmer has nothing left to decide, so all of them can perform together. A PLM moves the randomness out of the output and into a small latent card per position, so the cards, not the left-to-right order, carry the agreement.

How the cue cards are made

During training the coach, who has already seen the finished routine, picks each card: the encoder samples a card from the region belonging to the token that actually came next. In the paper the encoder is an ordinary autoregressive model: at each position it produces the usual next-token probabilities. Each token is then tied to a share of a fixed set of random unit vectors in the embedding space, split by a weighted Voronoi partition so that a token's share of cards equals its probability. A likely token owns a large region and a rare token a sliver, so drawing a card uniformly at random draws a token in proportion to the model's own belief. In the paper's construction only tokens among the autoregressive model's top 128 predictions get their own region, and any other token gets a mask card.

The swimmers are the decoder. It reuses the encoder's weights, differing only in the embedding layer that reads cards instead of tokens, and it must rebuild a block of tokens from the cards alone, attending to the encoder's KV cache for everything before the block. Training minimizes a variational bound that splits into the usual autoregressive loss plus a parallel-decoding loss. Because the encoder is a full autoregressive model, a PLM can still be sampled left to right, so the autoregressive path is kept rather than replaced. The block length is capped, since the paper finds reconstruction gets harder as blocks grow.

At inference: draw cards, decode once, let the judge keep a prefix

At inference there is no coach to write cards. The model draws a random card for each position of a block, runs the decoder once, and reads out a block of proposed tokens. Finite capacity makes proposals worse the farther they sit from the context, so the encoder checks them in the same call using Quadratic Decoding: the call is widened with a custom attention mask so several proposals are verified together and one is kept. The accepted prefix joins the context and the loop repeats.

Two acceptance rules appear in the paper. Exact acceptance keeps only tokens identical to a random rollout of the autoregressive model, the same guarantee speculative decoding gives. Relaxed acceptance also admits tokens in the encoder's high-probability nucleus, which raises the accepted tokens per call at what the authors describe as comparable quality. Compare this with the classic draft-then-verify loop below: the PLM keeps the verify half but replaces the separate draft model with its own parallel decoder.

Hold three things fixed: the post-trained 1.3B model, the same post-training recipe for both models, and batch size 1 on one NVIDIA RTX 6000 Ada under vLLM, which is how the paper's appendix describes its speed measurements. On GSM8K the autoregressive baseline scores 27.9 at 0.9 questions per second. The PLM with relaxed acceptance scores 32.6 at 5.2 questions per second, which is 5.7× faster with a higher score. The paper's token-level table shows the baseline at 291 tokens per second and 1.0 token per call; the PLM with exact acceptance at 1112 tokens per second and 4.2 tokens per call; and the relaxed PLM at 1337 and 6.1.

Divide those and the speedup is 1112 / 291 ≈ 3.8× for exact acceptance and 1337 / 291 ≈ 4.6× for relaxed, below the 4.2 and 6.1 tokens per call. Tokens per call overstates the speedup, because a widened verification call is not free. The 5.7× figure divides questions per second, not tokens per second, so the two ratios need not match. Other benchmarks in the same table land lower or higher depending on task and stride: relaxed acceptance on IFEval is about 2.4× faster, but its score (45.1) is below the baseline (48.2), so the quality claim is not uniform.

What the paper does not show

The speed numbers come from batch size 1 on one GPU. The paper does not report what happens on a busy serving batch. Our inference, not a measured result, drawn from the trade-off in When to Use It: extra proposals spend compute, and a heavily loaded batch may leave less compute to spend, so the speedup could differ. The models are also small: 80M to 3B parameters. The encouraging sign is scale: in the pretraining runs, accepted tokens per call under exact decoding rose from 1.95 at 350M to 2.26 at 3B, and the authors report that speedup follows a power law in model size. Whether that holds at frontier size is open.

Goes deeper in: LLM Serving → Speculative Decoding → The Draft-Verify Idea

Related explainers

• K2 Horizon: a LoRA diffusion adapter for block-parallel decoding — another way to write a whole block of tokens per pass, via diffusion
• Masked diffusion serving: denoising-step batching — what batching looks like for a model that decodes in rounds rather than one token at a time