What is capacity-first inference memory?

It is designing an accelerator around how much memory it holds rather than around how fast that memory can be read. The mainstream approach uses High Bandwidth Memory, which wins bandwidth through a very wide bus but is expensive, hot and limited in capacity. A capacity-first design like Intel's Crescent Island uses ordinary LPDDR5X instead, accepting lower bandwidth per byte in exchange for a much larger pool — 480GB on one 350-watt air-cooled card, by Intel's figures.

Why does memory capacity matter for inference if decode is bandwidth-bound?

Because capacity and bandwidth fail differently. Insufficient bandwidth makes a workload slower; insufficient capacity stops it running on that device at all, forcing you to split the model across several cards or spill to host memory over a much slower link. Capacity is a threshold and bandwidth is a rate. The threshold has become the binding one for a growing class of workloads because the KV cache grows with both context length and the number of concurrent requests, while the weights do not.

How does it relate to the roofline model?

Directly, and as a corrective. The roofline says decode is limited by bytes moved rather than math performed, which is the case for prioritising bandwidth. Extra capacity does raise throughput, by letting more sequences run at once and amortising the weight read across them — but the added KV cache must also be read every decode step, so the gain flattens quickly. Working the arithmetic for a 70B-class model at 100,000-token context, four times the concurrency yields roughly 1.45 times the tokens per second at equal bandwidth, which sets a break-even the slower memory has to clear.

Intel reveals a 480GB air-cooled inference GPU — Capacity-first inference memory

TL;DR

What is it: The news anchor is Crescent Island, the inference GPU Intel described at Hot Chips 2026; the concept it makes concrete is capacity-first inference memory — building an accelerator around 480GB of LPDDR5X rather than around a smaller pool of High Bandwidth Memory.
Why it’s needed: A card's memory has two separate numbers — how much fits, and how fast it can be read — and inference has quietly moved the pressure from the second to the first, because KV cache grows with context length and with every concurrent request while the weights stay the same size.
vs previous: Where an HBM-based accelerator stacks DRAM dies beside the compute die to win bandwidth, and pays for it in capacity, cost, power and liquid cooling, Crescent Island takes ordinary mobile-class DRAM and takes the opposite side of that trade: far more room, less speed per byte, and a 350-watt air-cooled PCIe card that fits servers that already exist.

Think of it as

a depot with a wide floor but one loading door

floor space = memory capacity, how many gigabytes of weights and KV cache the card holds at once
the loading door = memory bandwidth, how many bytes per second the compute units can actually pull in
stock that will not fit in the building at all = a model plus its KV cache larger than the card, which does not run slowly, it does not run
trading a small depot with a loading bay for a big shed with a side door = choosing LPDDR5X capacity over HBM bandwidth, which is Crescent Island's bet
every single order re-walking the whole floor = decode re-reading the full weight set for each token, which is why the door and not the floor sets the pace
a shed that fits in the yard you already have = the 350-watt air-cooled PCIe envelope, against liquid-cooled racks you would have to rebuild for

Jargon

HBM (High Bandwidth Memory): DRAM built as a vertical stack of dies sitting next to the compute die on a shared silicon substrate, connected by a bus thousands of bits wide. The width is where the bandwidth comes from. It is also where the cost, the heat and the capacity ceiling come from, which is why it is the rung of the memory ladder your model actually lives on and the rung that is hardest to make bigger.
LPDDR5X: Low-Power Double Data Rate memory, the mobile-class DRAM in phones and laptops. It attaches over an ordinary, much narrower bus, so it delivers less bandwidth than HBM — but it is cheap enough and low-power enough to attach a great deal of it. Crescent Island is built around up to 480GB of it.
Memory capacity vs memory bandwidth: Two independent properties of the same pool. Capacity is a cliff and bandwidth is a slope: below the capacity you need, the workload does not run at all; above it, bandwidth decides how fast it runs. Most accelerator design is an argument about which of the two is the binding constraint.
Model residency: Whether the weights and the working state fit in one device's memory. If they do not, you either split the model across several devices or move data across a link an order of magnitude slower than local DRAM. Residency is the thing extra capacity is really bought for.
KV cache: The keys and values every previously generated token leaves behind, kept so attention does not recompute them. It is the part of the memory bill that grows — linearly with context length and linearly again with the number of sequences in flight — which is what turns "longer contexts and more concurrent agents" into a capacity problem.
Memory-bound (decode): Limited by bytes moved rather than by math performed. Generating one token reads the entire weight set to do a small amount of arithmetic per byte, so decode sits on the bandwidth-limited side of the roofline. This is the reason bandwidth has been the number accelerator vendors advertise.
XMX engines: Intel's matrix-multiply units, its equivalent of tensor cores — dedicated hardware that multiplies small matrix tiles in one instruction. Intel says Crescent Island carries 256 of them across 32 Xe3P cores, so the capacity-first choice is not paired with giving up matrix acceleration.
TDP and cooling envelope: The power a part is designed to dissipate, and therefore what it takes to cool it. At 350 watts on a PCIe card, Crescent Island is meant for air-cooled servers; the HBM-class accelerators it is positioned against draw enough more to need liquid cooling and, usually, a different rack.

The news. On August 24, 2026, at Hot Chips 2026, Intel outlined Crescent Island, a 350-watt, air-cooled PCIe GPU for real-time AI inference, alongside its Diamond Rapids and Wildcat Lake parts. The design pairs an Xe3P compute architecture — Intel cites 32 Xe cores and 256 XMX matrix engines — with up to 480GB of LPDDR5X, rather than an HBM-only capacity strategy. Intel frames the target as larger models, longer contexts and more concurrent agents inside existing data-center cooling envelopes. Read the announcement →

Picture a distribution depot. It has two numbers that matter and they have nothing to do with each other: how much floor space there is, and how wide the loading door is. Floor space decides what you are able to stock at all. The door decides how fast any of it moves. You can be defeated by either one, and being defeated by them feels completely different. Too small a door and the business is slow. Too small a floor and there is no business — the pallet arrives, will not fit, and goes somewhere else.

A GPU's memory is that depot, and the two numbers are capacity and bandwidth. For about a decade the industry has optimised the door. High Bandwidth Memory is DRAM stacked into towers and set down next to the compute die on a shared piece of silicon, wired with a bus thousands of bits wide; the width is the whole point. But stacking is expensive, it runs hot, and it does not go very tall, so an HBM part gets a magnificent door attached to a fairly modest floor. Crescent Island takes the other side of that trade: ordinary LPDDR5X, the mobile-class memory in a laptop, on an ordinary narrow bus — less bandwidth per byte, but cheap and cool enough to bolt on 480GB of it.

The reason that trade is even arguable now is that what consumes memory during inference has changed shape. The weights are a fixed cost: a 70B model at 8-bit is 70GB whether one user is connected or a thousand. The KV cache is not fixed — every token in every live conversation leaves keys and values behind, so it grows with context length and grows again with each concurrent request. Intel's own framing of the target names exactly those two axes. A workload of short prompts and few users is a bandwidth problem. A workload of hundred-thousand-token contexts and many concurrent agents is a capacity problem, and no amount of bandwidth rescues a pool that is too small to hold the state.

Memory choice	What it optimises	Typical capacity per card	Cooling	Fails when
HBM stacks	bandwidth, via a very wide bus on a shared substrate	~100–300GB, generation-dependent, illustrative	liquid, at the high end	the working set outgrows the pool
LPDDR5X (Crescent Island)	capacity and power, via ordinary mobile-class DRAM	up to 480GB, per Intel	air, at 350W per Intel	decode is bandwidth-limited and the batch is small
Unified CPU–GPU pool	capacity plus removing the host copy	~128GB on the desktop parts, illustrative	air	one pool must serve two very different consumers
Split across several HBM cards	capacity, by adding devices rather than memory	scales with card count	liquid, usually	the interconnect, and the cost per served token

Here is where the bet earns its keep, and where it stops earning it — in one calculation. Hold a 70B-class model fixed: 80 layers, 8 KV heads, head dimension 128, weights at 8-bit, KV in 16-bit. Each token's KV entry costs 2 x 8 x 128 x 2 = 4,096 bytes per layer, or 327,680 bytes in total, so 320 KiB per token across 80 layers, and a single 100,000-token conversation therefore parks 32.8GB of cache. The weights take 70GB before any user arrives. On a 192GB HBM card that leaves 122GB of headroom, which is 3 concurrent 100K-token sequences. On 480GB it leaves 410GB, which is 12 — four times the concurrency, on one card, from the capacity alone. Now price the speed. A decode step must re-read the weights and every live sequence's cache, so it moves 70 + 3 x 32.8 = 168.4GB on the small card and 70 + 12 x 32.8 = 463.6GB on the large one, for 3 and 12 tokens respectively. Per byte of bandwidth, that is 0.0178 versus 0.0259 tokens — so four times the concurrency buys only about 1.45x the token throughput, and only if the bandwidth were equal. It is not equal. Crescent Island's LPDDR5X has to deliver roughly 69% of the HBM part's bandwidth just to break even on throughput — and that is the one number Intel did not publish. (The model geometry, the 192GB comparison point and the 8-bit weights are illustrative, built from the curriculum's own KV arithmetic; Intel discloses the 480GB, 350W, 32-core and 256-engine figures and does not publish a bandwidth figure or a throughput comparison.)

That result is worth sitting with, because it is the roofline reasserting itself against an intuition that sounds obviously right. "More memory means more concurrent users means more throughput" is true, and it is also sharply diminishing, for a reason that is easy to miss: the KV cache you just made room for is not inert. It has to be read on every single decode step, by the same narrow door. Past the point where cache traffic dominates weight traffic, adding capacity adds bytes to move at nearly the same rate it adds tokens to produce, and the ratio flattens out. Extra floor space converts a cliff into a slope; it does not widen the door.

Which is why the honest reading of Crescent Island is not "capacity beats bandwidth." It is that the two failures are not symmetric, and the asymmetry is the argument. Being, say, 30% slower per token — a hypothetical, since no per-token figure for this card exists — is a bad quarter. Not fitting is a different category of problem: you split the model across cards and pay an interconnect tax on every layer, or you fall back to host memory over PCIe and pay far more than any DRAM would have cost you. A card that removes the cliff entirely for a class of workload — long contexts, many concurrent agents, models that would otherwise need two or four devices — has done something a faster card cannot do at all, and it has done it at 350 watts in an air-cooled server that already exists, which is its own kind of capacity. What Intel disclosed at Hot Chips is the 480GB, the 350W, the 32 cores and the 256 XMX engines; the announcement carries no memory-bandwidth figure and no benchmark, so every performance statement here — including the break-even above — is arithmetic rather than measurement. The tradeoff it names is real and durable regardless of whether this particular card is any good — but the number that decides whether this card is any good is precisely the one still missing.

Goes deeper in: GPU & CUDA → Memory Hierarchy → HBM: Where Your Model Lives

Related explainers

TokenStack — attention inside HBM-PIM — the opposite response to the same pressure: instead of swapping HBM for something roomier, put compute inside the HBM so the hot KV never crosses the door at all.
NVIDIA RTX Spark — unified CPU–GPU memory — capacity bought a different way, by merging two pools rather than by choosing a cheaper kind of DRAM.
Jalapeño — inference ASIC vs GPU — specialisation on the compute axis rather than the memory axis, and the same question about what flexibility is worth giving up.
MobileMoE — DRAM-aware MoE scaling — the identical capacity-versus-bandwidth argument settled at the other end of the scale, under 3GB on a phone.

Continue in trackMemory Hierarchy: what HBM buys and what it costs

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based