Qwen-Image-2.1 runs two mask rules in one sequence — Mixed-granularity attention — What does it mean?
The news. On September 20, 2026, Qwen released Qwen-Image-2.1 as open source, unifying text-to-image generation, image editing and native RGBA transparency in one model. The visual generator carries 7B parameters across 32 Single-Stream DiT layers and accepts up to 10 reference images per generation or editing task. Two of its choices are inference-side rather than architectural fashion: mixed-granularity attention — a token-level causal mask for text, a chunk-level mask for image generation — and computing the key-value states of the static context in the first step, then reusing them. Read the release →
Picture the pass in a working kitchen. On one side sits the recipe card, and the cook reads it the only way a recipe can be read: one line at a time, in order, never peeking ahead. Step four cannot be performed before step three, because step three is what makes step four mean anything. On the other side sits the plate, and it is not built that way at all — a whole section goes down in one motion, three elements placed together, because none of them is the reason for any of the others.
That is the shape of Qwen-Image-2.1's sequence. The text half — the system prefix, the prompt, the editing instructions — runs under a token-level causal mask, the same left-to-right rule that governs an ordinary language model reading its own output. Instruction order carries meaning, so the model is held to reading it in order. The image half runs under a chunk-level mask, which handles a group of tokens as one unit. The mask is not one rule applied to the whole sequence; it changes where the modality changes.
This only becomes a decision worth making because Qwen-Image-2.1 is a Single-Stream DiT — text and image tokens travel through the same 32 layers, in the same sequence, rather than in separate towers that meet at a cross-attention seam. One kitchen, one pass, two rules. A two-tower design would not need the choice, because each tower could simply have its own fixed rule; sharing the stack is what forces the mask to become position-dependent.
Now the photo pinned above the pass. When you are editing rather than generating from nothing, the reference images and the editing instructions are fixed for the whole job — they are what the cook keeps glancing up at, not something being rewritten between plates. Qwen computes their key and value vectors in the first step and caches them, then reuses that cache on every step after. Anything whose keys and values cannot change is work you only have to do once.
If that sounds familiar, it should: it is the same observation that drives prefix caching in a serving stack, where a shared system prompt is projected once and reused across requests instead of being recomputed for each one. The saving here runs along a different axis — across the denoising steps of a single job rather than across separate requests — but the precondition is identical, and it is the only one that matters: the span has to be genuinely frozen. The diagram below counts what a single cached token costs in a transformer — the general shape of the bill, not Qwen-Image-2.1's published numbers.
Put numbers on the pinned photo. Qwen publishes the parts of this that are fixed — 32 Single-Stream DiT layers and up to 10 reference images — but not how many tokens a reference image costs or how many denoising steps a generation runs, so take those two as (illustrative): say each reference image contributes about 1,000 tokens, for 10,000 static-context tokens, and say the generation runs 30 denoising steps.
Without first-step caching, those static tokens are re-projected into keys and values at every layer on every step: 10,000 × 32 × 30 = 9,600,000 key/value projections. With the cache, they are projected at every layer exactly once: 10,000 × 32 × 1 = 320,000. That is 30× fewer projections — exactly the number of denoising steps. Notice what dropped out of the division: the token count and the layer count appear identically on both sides and cancel, so the factor you save is the step count and nothing else — passing five reference images instead of ten halves both numbers and leaves the ratio at 30×.
Read that figure narrowly. It counts the static context's projection work only. The image tokens being generated do change at every step and are re-projected at every step, and they are the bulk of a generation, so a 30× cut in one component is not a 30× faster image — Qwen frames it as reducing repeated work and memory use, and claims no end-to-end speedup for it.
| Model | Mask rule | What it costs you |
|---|---|---|
| Autoregressive LLM | Token-level causal, everywhere | One token per forward pass; a token can never be revised using later context |
| Block-diffusion LM, e.g. DiffusionGemma | Bidirectional within a block, everywhere | A whole block is refined at once, but strict left-to-right instruction order is given up |
| Qwen-Image-2.1 | Token-level causal for text, chunk-level for image generation | Instruction order is preserved while image tokens are handled in groups — at the price of a mask that depends on position |
The durable idea here is not the chunk size, which Qwen does not publish anyway. It is that a sequence is not obliged to be homogeneous. Once text and images share one stack, two questions stop having a single global answer and start having a per-span one: at what granularity does this span need to be masked, and is this span frozen for the duration of the job? What generalizes is the pair of questions, not Qwen's particular answers — and both are worth asking of any sequence you are about to pay for token by token.
Goes deeper in: LLM Internals → Self-Attention → Causal Masking
Related explainers
- DiffusionGemma — Parallel block decoding — the other way to drop the causal mask: refine a whole block at once instead of varying the rule by span.
- Kev — Question-isolation masks — a second custom mask, cutting the sequence apart by question rather than by modality.
- ReCache — Composition-invariant KV blocks — the cache half of this story, pushed across agent calls instead of across denoising steps.