What is N-gram Embedding in Qwen3.8-Flash-Next?

It is a second embedding table, separate from the token embedding table, that is looked up using the local run of tokens rather than a single token id. Qwen reports 51B parameters in it, on top of a 125B main model. Because reading it is a lookup rather than a matrix multiply, it adds model capacity with very little extra computation.

Why can the N-gram embedding table be offloaded to host memory?

A lookup does not need to happen where the arithmetic happens, and the key — the tokens already emitted — is known a step before the row is needed. That makes the fetch predictable, which is what lets the transfer start early and overlap with computation the GPU is already doing. Qwen describes the result as asynchronous prefetching and does not publish the scheduling details.

How is this different from a normal token embedding table?

A token embedding table is indexed by one token id and lives in GPU HBM with the rest of the weights, so every gigabyte it uses is a gigabyte the KV cache cannot have. N-gram Embedding is indexed by several consecutive tokens, so the same word can pull different rows in different contexts, and it is designed to live in host DRAM instead of HBM.

Does offloading the table slow inference down?

Only if the prefetch does not finish inside the work it overlaps. The transfer crosses PCIe, which is the slowest link between host and GPU, so the scheme depends on issuing the copy early enough and on each row being small enough. Qwen has not published the row width, hit rates, or measured overlap, so the efficiency claim is reported rather than independently verified.

LLM· 2026-09-14

Qwen3.8-Flash-Next adds 51B of embeddings that can live off the GPU — N-gram embedding offload — What does it mean?

TL;DR

What is it: Qwen3.8-Flash-Next is the architecture preview Qwen says the full Qwen4 family will be built on, and one of its four changes is N-gram Embedding: a second embedding table that is looked up with the last few tokens together instead of one token id, adding 51B parameters on top of a 125B main model.
Why it’s needed: Those 51B parameters are a table lookup, not a matrix multiply, so they buy model capacity at almost no extra arithmetic — and because a lookup does not need to sit next to the compute, Qwen says the table can be offloaded to host memory and prefetched asynchronously while the GPU is still working.
vs previous: A standard token embedding table is indexed by a single token id and lives in GPU HBM alongside the weights, where every gigabyte it takes is a gigabyte the KV cache cannot have. N-gram Embedding is indexed by local context and is explicitly designed to leave the accelerator, so extra capacity stops competing for the scarcest memory in the machine.

Jargon

Embedding table: A big lookup table with one learned vector per entry. The model reads a row out of it instead of computing anything, which is why an embedding table costs memory but almost no arithmetic. Covered in LLM Internals → Embeddings.
N-gram: A run of N consecutive tokens treated as one unit — a bigram is two, a trigram is three. N-gram Embedding keys its lookup on the local run of tokens rather than on one token id, so the same word can pull a different row depending on what came just before it.
HBM: High-Bandwidth Memory — the fast memory soldered next to the GPU die. It is where model weights and the KV cache live during serving, and it is the scarcest resource on an inference node. See GPU & CUDA → Memory Hierarchy.
Host DRAM: Ordinary system RAM attached to the CPU, often hundreds of gigabytes or more per node. It is far larger and far cheaper than HBM, and far slower for the GPU to reach because every byte crosses PCIe — the bus connecting the CPU and the GPU, and the narrowest pipe between them.
Asynchronous prefetch: Starting a memory transfer early, on a separate stream, so the data arrives before the computation that needs it. The transfer overlaps with work the GPU is already doing instead of stalling it. See GPU & CUDA → Memory Hierarchy → NVLink & PCIe.
Activated parameters: In a Mixture-of-Experts model, the subset of parameters that actually take part in computing one token. Qwen3.8-Flash-Next reports 6B activated per token against a 125B main model, so total parameter count and per-token compute are two different budgets.

The news. On August 26, 2026, Qwen published Qwen3.8-Flash-Next, an early release of the architecture it says the full Qwen4 family will be built on — the same move it made with Qwen3-Next before the Qwen3.5 through Qwen3.8 series. Four changes ship together: a GDN + QSA hybrid attention stack, a Gated Residual stream, the N-gram Embedding described here, and a refined Muon optimizer. The model is 125B main parameters plus 51B N-gram embedding parameters, with 6B activated per token, and Qwen reports training cost around one ninth of Qwen3.7-Plus. Read the release →

Picture the chef's counter. Everything the kitchen can reach right now has to fit on it, and the counter is already crowded with the pans in use. The walk-in pantry down the hall holds twenty times as much, but walking there costs you seconds you do not have mid-service. The usual compromise is to keep the counter small and the menu small with it. The interesting move is not making the counter bigger — it is noticing that some ingredients never need to be on the counter until the moment they are used, and sending a runner ahead for them. That is the whole shape of what Qwen did to the embedding table.

Start from the ordinary version. A token embedding table holds one learned vector per vocabulary entry, and the model reads a row out of it — no multiply, no activation, just an indexed fetch. An embedding table is the cheapest capacity a transformer can buy, because a lookup costs memory instead of arithmetic. N-gram Embedding keeps that property and changes the key: instead of indexing on one token id, it indexes on the local run of tokens, so bank after river and bank after savings pull different rows. Qwen's description is one sentence long — the table is looked up "using the local context to scale model capacity with very little extra computation" — and the paper-level detail of how the n-gram is hashed into the table is not published yet.

One-hot (12 dims shown)

cat

dog

king

Every token looks the same distance apart

→

Embedding (8 dims shown)

cat

0.5

0.8

-0.1

0.3

-0.6

0.2

0.7

-0.9

dog

0.4

0.8

-0.1

0.3

-0.5

0.3

0.6

-0.8

king

-0.7

0.1

0.9

-0.5

0.8

-0.3

-0.2

0.6

cat & dog look similar — king looks different

Now the part that makes the jars worth walking for. Because the lookup is not matrix math, the table never has to be co-located with the compute, and Qwen says the N-gram embedding table "can be offloaded to host memory and overlapped with model computation through asynchronous prefetching." What makes that schedulable is the key itself: it is the tokens the model has already emitted, so the row needed next is knowable before the layer that reads it runs — the runner can leave for the pantry before the chef asks. Qwen states the overlap, not the schedule. The standard way to build it is to issue the host-to-device copy on a separate stream during the current layer's work, so the row lands in HBM before it is read; the release does not say whether that is what Qwen does. Either way the transfer is hidden rather than removed.

CPU (Host)

GPU (Device)

float *a, *b

(input data)

float *c

(results)

copy→

←copy

1. cudaMalloc

reserve memory

▼

3. kernel<<<>>>()

1000s of threads

▼

done

1. Allocate2. Copy →3. Launch4. Copy ←

Data crosses the PCIe bus twice — GPUs need large workloads to pay off

Where the memory actually goes

Hold the three numbers Qwen published fixed — 125B main parameters, 51B N-gram embedding parameters, 6B activated per token — and pick a weight format to make them concrete. Take BF16 at two bytes per parameter (illustrative; Qwen does not state a serving dtype). The main model is then 125 × 2 = 250 GB and the N-gram table is 51 × 2 = 102 GB, so keeping everything resident costs 352 GB of weights before a single byte of KV cache is allocated. On an eight-way 80 GB node — 640 GB of HBM in total — that is 55% of the machine gone to weights. Move the table to host DRAM and the resident bill drops to 250 GB, or 39%, and in this simplified calculation the 102 GB you freed is aggregate HBM capacity the KV cache was competing for — sharding and runtime overhead are held outside the example. The parameters did not get cheaper; they moved to a shelf where floor space is not rationed.

The boundary is the runner's walking speed. Offloading only pays while the prefetch finishes inside the compute it overlaps — miss that window and the GPU stalls on a PCIe transfer, which is the slowest link in the box and the one every other offload scheme also queues on. What makes this particular table a candidate at all is that the key is known a step in advance, so the fetch is predictable rather than reactive. Offload works here because the lookup is predictable, not because host memory got fast. The scheme also requires each row to be small enough that the transfer is a short hop rather than a bulk weight load — but Qwen has not published the row width, the hit rate, or the measured overlap, so that condition is assumed rather than confirmed. Treat the mechanism as documented and the efficiency as reported rather than verified.

Scheme	Indexed by	Where the table lives	What it trades
Token embedding	one token id	GPU HBM, beside the weights	the baseline — capacity competes with the KV cache
N-gram Embedding (Qwen3.8-Flash-Next)	the last few tokens together	offloadable to host DRAM	+51B parameters at very little extra computation, paid for in PCIe traffic
Hash-signature tokens	hashes of the token string	GPU HBM, but far smaller	drops the vocabulary-sized table entirely

Goes deeper in: LLM Internals → Embeddings → From Token IDs to Vectors

Related explainers

Qwen3.8-Flash-Next widens the residual stream into four gated branches — the second architecture change from the same release, and the one that pays for depth instead of capacity
MultiHashFormer drops the vocab-sized embedding table — the opposite bet: shrink the table rather than grow it and move it

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based