What is a CUDA locality domain?

A locality domain (LD) is a CUDA 13.4 control that names one compute die of a multi-die GPU. You can allocate memory on a chosen LD and limit kernels to that LD's SMs, with the two controls working independently.

Does pinning data to one die always make a kernel faster?

No. In the preprint, memory placement sped up a 1 MiB remote read by 2.08×, but putting a 256 MiB streaming buffer on one die made it 47% slower because only one die's memory channels were available.

Does this change how LLM decode attention runs?

Not by itself. The study found that neither memory placement nor compute binding gave a consistent gain to a bandwidth-bound decode-attention kernel once each configuration chose its own tile size.

GPU· 2026-11-07

Blackwell locality domains — Die-aware memory placement — What does it mean?

Q: Why is a localized allocation slower to create?

Beyond 32 MiB the driver maps memory one 2 MiB page at a time, and the paper attributes 77–87% of the cost to full-chip barriers and TLB invalidations. Coalescing mappings and deferring synchronization cut a 1 GiB `cuMemCreate` by 7.02×.

Jargon

Die: A separate piece of silicon inside one chip package. A B300 GPU holds two compute dies joined by an on-package fabric.
Locality domain (LD): A CUDA 13.4 control that names one die as a place, so memory can be allocated on it and kernels can be limited to its SMs.
Striping (interleaving): Spreading an allocation in chunks across the memory of both dies. The paper's measurements show stock cudaMalloc memory is laid out this way.
L2 cache: The on-chip cache between the SMs and HBM. Each die has its own slices, about 63.2 MiB per die on a B300. See the memory ladder.
TLB: Translation lookaside buffer: a small cache of virtual-to-physical address translations. When a mapping changes, stale entries must be invalidated.
Green context: A CUDA mechanism that limits a stream's kernels to a chosen group of SMs. The paper uses it to bind compute to one die.
cuMemCreate: A CUDA driver call, part of the virtual memory management API, that allocates physical memory. It accepts a locality domain as the target location.

The news. On November 5, 2026, a team from Scitix AI posted an arXiv preprint that measures CUDA 13.4 locality domains on eight B300 GPUs in one node, comparing stock execution, memory placement alone, compute binding alone, and both together. The results are from that one platform and have not been peer reviewed. Read the paper →

A locality domain is CUDA's way of treating one die of a two-die Blackwell GPU as its own place, so data and the threads that read it can sit on the same die. Each B300 exposes two LDs of 70 SMs each, out of 148 SMs in total.

Picture the GPU as one floor with two offices. Each office has its own filing room (HBM) and its own desks with papers on them (L2). A hallway (the on-package fabric) joins the offices. Stock cudaMalloc deals files out across both filing rooms like cards, so about half of any request has to walk the hallway. Locality domains let you say "file this here" (memory placement) and "work from this office" (compute binding). The two rules are independent, and the paper varies them separately and together.

The hallway is expensive only when the trip ends at a filing room. For a 256 MB dependent-load chain, a same-die load took 295.3 ns and a cross-die load took 488.8 ns, a gap of 193.4 ns. For an 8 MB working set, which fits in L2, the gap was 3.0 ns. The reason the paper gives is that data read from the other die can be cached in the reading die's own L2, so the hallway is walked once and the paper stays on the desk. This is the memory ladder with one new rung: which die's L2 and HBM.

Pinning data to one die can lose more bandwidth than it saves in latency. Striped memory lets a die's SMs pull from the memory channels of both dies. Pinning removes that: in the paper's 256 MB streaming read, the stock striped layout reached 5229.8 GB/s, while memory localized on the accessor's own die reached 3628.1 GB/s. Putting a 256 MiB streaming buffer on one die made the workload 47% slower. Streaming workloads are limited by HBM bandwidth, and pinning leaves them only one die's channels.

Measurement (accessor on die 0)	Same die	Cross die	Striped (stock `cudaMalloc`)
Read latency, 8 MB working set, ns (Table 4)	135.8	138.8	137.3
Read latency, 256 MB working set, ns (Table 4)	295.3	488.8	466.1
Read bandwidth, 256 MB, GB/s (Table 4)	3628.1	2645.7	5229.8

Localized allocation has its own price, and it jumps once an allocation passes 32 MiB. Allocate through cuMemCreate with an LD and the driver maps memory one 2 MiB page at a time; the paper attributes 77–87% of the cost to full-chip barriers and TLB invalidations. Its allocation table shows the jump: at 32 MiB, creation took 135.9 µs stock and 158.3 µs with LDs (+22.4 µs). At 34 MiB it took 134.9 µs stock and 492.0 µs with LDs, a 3.65× slowdown. At 1 GiB the stock call took 145.6 µs against 5745.3 µs with LDs.

A worked example at 1 GiB shows what the fix buys and what it leaves. The allocation is 512 pages of 2 MiB, so 5745.3 µs is roughly 11 µs per page (derived). The authors coalesce physically contiguous mappings and defer the synchronization; that cut the 1 GiB call to 818.1 µs, a 7.02× reduction; from 64 MiB to 1 GiB the same fix gave between 7.0× and 10.2×. Even after the fix, 818.1 µs is still more than five times the 145.6 µs stock call, so a localized allocation needs to be reused many times to repay its cost.

Which control helps depends on who pays the cross-die cost, not on the name of the workload. In the paper, compute binding alone sped up collectives issued by SMs by 1.495×. Resident-data grouped GEMMs (DeepGEMM) needed both memory placement and compute binding to reach 1.37×. A 1 MiB read of a peer's data through a copy engine gained 2.08× from memory placement alone, against 1.15× from compute binding alone. A bandwidth-bound decode-attention kernel got no consistent gain from either once each configuration was allowed to pick its own tile size. The practical rule from the authors is to verify where an allocation really landed (cudaMalloc buffers read back an LD ordinal of −1 and produced a bimodal 74–101 GB/s baseline) and then benchmark the real workload. For how links between whole GPUs compare, see NVLink and PCIe.

Goes deeper in: GPU & CUDA → Memory Hierarchy → HBM: Where Your Model Lives

Related explainers

Cerebras CS-5 and wafer-scale memory locality — the same "distance to memory sets the speed" idea, taken to a whole wafer.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based