dQwen3.5 — Adapting a hybrid AR backbone into a diffusion LM — What does it mean?

The news. On September 17, 2026, a paper introduced dQwen3.5, a family of diffusion language models adapted from Qwen3.5 at 0.8B, 2B, 4B and 9B parameters. Qwen3.5 is a hybrid: three Gated DeltaNet recurrent layers for every one attention layer. That is a problem for diffusion adaptation, because a diffusion model needs to read context in both directions and a recurrent layer is one-way by construction. The paper's answer is to not fight it — only the attention layers are un-masked. Against a closely size-matched full-attention control, the hybrid reaches a given training loss in about half the tokens. Read the paper →

Picture the line again. A blinkered station still does real work: it sees every part that already passed it, so it knows the whole history of the object in front of it. What it cannot do is look ahead. For writing text left to right that limitation costs nothing, because there is nothing ahead yet. For a diffusion model that is the whole difficulty, since the point is to fill in a blank in the middle of a sentence using the words on both sides of it.

The obvious fix is to give every station a window. That is what previous adaptations did: start from a full-attention transformer, where every layer is attention, and delete the causal mask everywhere. It works, but it only works on that one shape of backbone — and the newest models are not that shape. dQwen3.5 never makes the recurrent layers bidirectional at all. It opens the windows it can open cheaply, and leaves the rest of the line blinkered.

Concretely: a Qwen3.5 hybrid block is three Gated DeltaNet layers followed by one attention layer. Only the attention layers are un-masked — 6 of 24 sequence-processing layers in the two smaller models and 8 of 32 in the two larger ones, a quarter of the stack either way — while every Gated DeltaNet layer is left exactly as it was. A Gated DeltaNet is one-way because it scans: it walks the sequence from left to right, compressing everything it has read into a fixed-size state. There is no mask sitting on top of it to delete, which is why the paper describes bidirectionalizing one as a non-trivial architectural change rather than a configuration flag.

One other change carries over from standard adaptation practice. The model prepends a beginning-of-sequence token and shifts the readout by one position, so the hidden state at position k still predicts the token at position k+1 — the same alignment the model learned during autoregressive pretraining — while the bidirectional attention layers are free to pull in context from the right. The stack structure itself is otherwise untouched.

Why would a quarter of the stack be enough? The paper offers an interpretation rather than a proof, and it is worth stating as such. Suppose language is mostly causal — most tokens really are predictable from their prefix — except for a small set of anchor positions that carry the future context that matters, such as a later noun that disambiguates an earlier phrase. If the number of those anchors is far smaller than the length of the sequence, then the causal recurrent layers can carry the prefix dependence on their own, and the bidirectional attention layers only need to provide a path to a few positions on the right. Under that reading, a small number of bidirectional layers is enough because the non-causal part of language is itself small — the windows do not need to be everywhere, only where the anchors are.

That leaves the obvious worry: a model with causal recurrence in most of its layers might simply keep writing left to right, and look like a diffusion model without behaving like one. The paper tests this by recording, on HumanEval, which decoding step filled each position, then scoring how left-to-right the resulting order was — locally, between neighbouring positions, and globally, across all pairs.

The answer splits. dQwen3.5-9B scores 0.636 on local AR-ness, squarely inside the 0.631–0.652 band of the three full-attention diffusion models it is compared against, so the causal recurrent layers do not stop it from filling nearby positions out of order. Globally all of the models score high, 0.884 to 0.967, meaning they sweep the canvas roughly left to right — but that is true of full-attention diffusion models too, including ones trained from scratch, so on this evidence the global sweep reads as a tendency these diffusion models share, not a symptom of the hybrid backbone. The flexibility also survives being pushed: under fixed decoding budgets, dQwen3.5-9B leads its comparison group on HumanEval at every speedup beyond one token per step — which is the point of the exercise, because committing several positions per forward pass is how a diffusion model attacks the decode bottleneck that makes one-token-at-a-time generation slow.

Put two rows of that table side by side and the size of the saving becomes concrete. Hold three things fixed: the trunk size class (6.5–7B), the decoding scheme (full-canvas, one token per step), and the fact that these are base models with no instruction tuning or RL on top.

LLaDA-8B has a 6.98B trunk and was trained on 2.3T tokens. dQwen3.5-9B has a 6.92B trunk — within 1% — and was adapted with 50B tokens. That is 2,300B ÷ 50B = 46× fewer tokens. On MMLU the cheaper model scores 74.57 against 65.88, a gap of 8.69 points in its favour; on HumanEval it is 64.02 against 33.54, a gap of 30.48 points.

Read that ratio carefully, because it is not a like-for-like training bill. The 46× counts diffusion-training tokens only, and excludes the autoregressive pretraining that dQwen3.5 inherits from Qwen3.5. LLaDA was trained as a diffusion model from the start; dQwen3.5 was adapted on top of a model someone had already paid to train. What the ratio sizes is the adaptation step, not the total cost of owning the model.

The same pattern shows up in the controlled comparison, where it can be read more carefully. Against dQwen3-1.7B — a full-attention control with a 1.41B trunk, closely matched to dQwen3.5-2B's 1.37B — the control needed a median 2.21× as many content tokens to first reach the same training loss — counting only the real text each run consumed, and measured over the settled part of training rather than the noisy warm-up at the start. That is the paper's cleanest architectural comparison, and it is the number to trust over the cross-model one above.

Three limits the authors state, and they matter for how far the result generalizes. The architectural comparison is one pair at one scale, and although the trunks are closely matched the two models have different parents and different tokenizers — on the paper's own mixture Qwen3 averages 4.18 bytes per token against Qwen3.5's 4.05, which makes per-token losses not strictly comparable. Downstream benchmark differences therefore cannot be attributed to the backbone alone, since the autoregressive parents already differed and the adaptation mixture was code-heavy. And only base models were evaluated, so how any of this survives instruction tuning is untested. The authors' own proposed follow-up is the clean version of the experiment: pretrain matched hybrid and full-attention models from scratch, then adapt both under the same tokenizer, data and optimizer.

Goes deeper in: LLM Internals → Text Generation → One Token at a Time

Related explainers

• DiffusionGemma: parallel block decoding — what a diffusion language model is, and how filling a canvas differs from writing left to right
• PSD: parallel speculative decoding for diffusion LLMs — how these models commit several positions per forward pass
• SMELT: looped depth reuse — another case of changing what a stack does without changing its parameter count