What does 74B-A9B mean?

The model stores 74 billion parameters in total but uses about 9 billion of them for each token. It is a mixture-of-experts model: a router picks a few experts per token and the rest stay idle for that token.

Why does a 74B model fit in 24GB?

At 2 bits per weight, 74B parameters need about 74 × 10⁹ × 2 ÷ 8 = 18.5 GB for the weights alone, which leaves a few GB for the KV cache and other memory. NVIDIA says the model uses native 2-bit quantization to fit in 24GB.

Does a mixture-of-experts model need less memory than a dense model?

No. Every expert must stay in memory because the router can choose any of them for the next token. The saving is in compute and in the bytes read per token, which follow the active parameters.

Does the announcement say how the 2-bit model was trained?

No. The announcement says only that the model uses native 2-bit quantization. It does not say whether quantization-aware training was used or which layers are quantized.

LLM· 2026-10-20

NVIDIA Nemotron 3.5 Bolt — Total vs active parameters in a 2-bit MoE — What does it mean?

Jargon

Mixture-of-experts (MoE): A model whose feed-forward layers are split into many small expert networks; a router picks a few experts for each token, so most weights sit idle on any single token.
Total parameters: Every weight stored in the model, including all experts. This number sets the memory footprint.
Active parameters: The weights actually used for one token (the "A9B" in 74B-A9B). This number sets the work and memory reads per token.
2-bit quantization: Storing each weight in 2 bits instead of 16, which cuts weight memory by 8×. See how numbers shrink.
KV cache: The per-token attention memory a model keeps while it generates. It lives in memory next to the weights and grows with context length.
Memory-bandwidth-bound: A workload whose speed is limited by how fast weights can be read from memory, not by arithmetic. Token-by-token decoding usually is.

The news. On October 15, 2026, NVIDIA announced Nemotron 3.5 Bolt 74B-A9B, which it says uses "a native 2-bit quantization to fit in 24GB of memory and run alongside other applications." The announcement does not say whether the model was trained at 2-bit, whether quantization-aware training was used, or which layers are quantized, so this article explains only the memory arithmetic the name implies, reading "74B-A9B" by the usual mixture-of-experts convention (74B total, about 9B active). Read the announcement →

Picture a warehouse. The building must hold the stock of every aisle, even though a picker fulfilling one order walks through only a few of them. A mixture-of-experts model works the same way. The building is the memory. The aisles are the experts. The router is the dispatcher who sends each token's picker to a few aisles. The name "74B-A9B" says both numbers: 74B parameters are stored, but only about 9B are used for any one token.

The two numbers answer two different questions. Memory asks "how much stock is in the building?", and the answer is the total, 74B. Speed asks "how far does the picker walk?", and the answer is the active 9B. Because the router can pick a different pair of aisles for the next token, none of the aisles can be moved out of the building. That is why an MoE does not get a smaller memory footprint from its sparsity. It gets cheaper reads. (See why the size problem comes first.)

The 2-bit part attacks the other factor in the memory formula. Weight memory is parameters × bits ÷ 8, so cutting 16 bits to 2 bits shrinks the stock room by 8×, as in how numbers shrink. It also matters which tensors are quantized, which the module covers under what to quantize.

Layout	Weight memory (lower bound)	Bytes read per token (illustrative, assumes weights dominate reads)
74B at 16-bit, all weights	~148 GB	~148 GB if dense
74B at 2-bit, dense	~18.5 GB	~18.5 GB
74B-A9B MoE at 2-bit	~18.5 GB (all experts stored)	~2.25 GB (9B active)

Worked example (illustrative; the announcement gives only the 74B, 9B and 24GB figures). Hold three things fixed: 74B total parameters, 9B active parameters, 2 bits per weight. Stored weights: 74 × 10⁹ × 2 bits ÷ 8 = 18.5 GB, compared with about 148 GB at 16-bit. That leaves roughly 24 − 18.5 = 5.5 GB of the 24GB for everything else: the KV cache, activations, quantization scales, any tensors kept at higher precision, and the other applications NVIDIA mentions. In practice that margin will be smaller, so treat 18.5 GB as a floor. Per token, a decode step reads about 9 × 10⁹ × 2 ÷ 8 = 2.25 GB. Suppose (illustrative, not an NVIDIA figure) the machine reads 200 GB/s and decoding is memory-bandwidth-bound with weight reads dominating: the ceiling is about 200 ÷ 2.25 ≈ 89 tokens per second, against 200 ÷ 18.5 ≈ 11 tokens per second for a dense 74B model at the same 2 bits. Real speeds are lower, since attention and the router also cost time, and the same 5.5 GB margin shrinks as the context grows.

Goes deeper in: LLM Internals → Quantization → The Size Problem

Related explainers

Nemotron 3 Nano Omni — 30B-A3B multimodal MoE — the same total-versus-active split at a smaller size
vLLM v0.20 — TurboQuant 2-bit KV cache — the other memory consumer that shares the 24GB with the weights

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based