What is FastTPS's reloading-free KV-cache concatenation?

FastTPS is a system for running the token-generation (decode) phase of decoder-only LLMs on general AI accelerators, such as an AMD Ryzen AI NPU. Its core trick is reloading-free KV-cache concatenation: instead of reloading and recopying the whole KV cache to append each new token's key and value, it writes the entry into a preallocated slot in place, removing a major source of redundant memory traffic in the token phase. It pairs this with a tiling-optimized FLAT kernel for RoPE attention and a separately fused, pipelined feed-forward network, reaching 93% peak memory-bandwidth utilization and a 6× speedup over the non-fusion baseline (arXiv:2607.11211).

Why is the token phase memory-bandwidth-bound?

In decode, the model generates one token at a time, and each step must read the entire KV cache — every key and value from earlier tokens — to compute attention. The arithmetic per step is small, but the amount of data moved grows with the context, so the accelerator finishes the math quickly and then waits on memory. That inherent read is what makes the phase limited by memory bandwidth, not compute. On top of it, the naive path recopies the whole cache just to concatenate each new token — and it is that redundant recopy, not the unavoidable read, that FastTPS's reloading-free concatenation removes.

How does FastTPS relate to FlashAttention and operator fusion?

They share the same principle: keep work on-chip and stop bouncing intermediate data to memory. FlashAttention tiles the attention computation so it never materializes the full score matrix in memory; operator fusion runs several steps as one kernel so results stay in fast on-chip memory. FastTPS applies both ideas to the token phase on an accelerator — a tiling-optimized FLAT kernel for RoPE attention plus a fused feed-forward network — and adds reloading-free KV-cache concatenation so even growing the cache doesn't trigger a memory reload. The result is 93% peak memory-bandwidth utilization and 6× over non-fusion (arXiv:2607.11211).

FastTPS fuses token-phase LLM inference on AI accelerators — Reloading-free KV-cache concatenation

TL;DR

What is it: The FastTPS paper introduces reloading-free KV-cache concatenation — a way to run the token-generation (decode) phase of an LLM on a general AI accelerator without reloading and recopying the whole KV cache just to grow it each step.
Why it’s needed: Token generation is memory-bandwidth-bound — each step is dominated by moving the growing KV cache, not by the math — so cutting that repeated memory traffic is what turns an idle accelerator into real speedup.
vs previous: The naive (non-fusion) path reloads and recopies the entire KV cache just to append each new token's key and value; FastTPS concatenates in place, tiles the attention and separately fuses the feed-forward, so the chip stops recopying the cache to add each entry — 6× over non-fusion at 93% of peak memory bandwidth.

Jargon

KV cache: The store of keys and values an LLM has already computed for every earlier token, so it doesn't recompute them each step. It grows by one entry per generated token. Full primer →
Token phase (decode): The stage that generates the answer one token at a time, as opposed to prefill, which reads the prompt in one parallel pass. Decode is the slow, repetitive phase FastTPS targets.
Memory-bandwidth-bound: When a step is limited by how fast data moves from memory, not by how fast the chip computes — so the compute units sit idle waiting on reads. Decode is the classic example. Roofline primer →
Concatenation: Appending the new token's key and value onto the end of the KV cache before attention runs. Done naively, it can force a reload of the whole cache into a fresh buffer.
RoPE: Rotary Position Embedding — the common way to encode where a token sits in the sequence by rotating its query and key vectors. FastTPS gives it a tiling-optimized attention kernel.
Operator fusion: Running several math steps as one kernel so intermediate results stay on-chip instead of bouncing to memory between steps. Fusion primer →
NPU: Neural Processing Unit — an on-device AI accelerator (here, the AMD Ryzen AI 300 series) whose memory bandwidth is the binding constraint, which is why avoiding wasted memory traffic matters so much.

The news. On July 13, 2026, the authors of FastTPS posted their paper (arXiv:2607.11211). It targets the token phase — the decode step that writes one token at a time — of decoder-only LLMs on general AI accelerators. FastTPS combines three pieces: reloading-free KV-cache concatenation, a tiling-optimized FLAT kernel for RoPE attention, and a separately fused feed-forward network scheduled with fine-grain pipelining. On an AMD Ryzen AI 300-series NPU running Phi-3-mini-4k in BF16, it reports a 6× speedup over the non-fusion baseline and 93% peak memory-bandwidth utilization. Read the paper →

Picture a busy bar. The tab is the running list of everything the table has ordered — in an LLM, that tab is the KV cache, the keys and values the model saved for every token so far. Now the table orders one more drink: the model writes one more token. The trouble is how the waiter adds it. In the naive setup, before writing the new drink he walks to the back office, fetches the entire tab, and recopies the whole thing onto a fresh slip just to append one line. Writing each new token means reading that entire growing store back from memory — and on a small accelerator, that repeated read, not the math, is what sets the pace. The decode phase is memory-bandwidth-bound: the arithmetic finishes quickly, then the chip waits.

FastTPS keeps the same drinks but stops the recopying. FastTPS's reloading-free KV-cache concatenation appends the new key and value in place, so the accelerator stops recopying the cache to add one entry. Instead of fetching the tab and rewriting it, the waiter simply writes the new drink on the tab already on the table — the existing entries are never read back from memory just to grow the list by one. That single change removes the largest source of repeated memory traffic in the token phase.

The other two pieces make sure the freed-up bandwidth isn't wasted elsewhere. FastTPS gives attention a tiling-optimized FLAT kernel so the RoPE attention math runs in on-chip tiles — keeping the softmax and intermediates on-chip instead of writing the full score matrix out to memory — and it separately fuses the feed-forward network into a pipelined pass. With the attention tiled and the feed-forward fused and pipelined, the work stays on-chip, and the NPU runs at 93% of its peak memory bandwidth instead of stalling on memory traffic.

Where it earns its keep

Decode is memory-bound because attention reads the whole growing KV cache every step — that read is inherent. The naive path stacks a second cost on top: to append the new token's key and value, it reloads and recopies the entire cache into a fresh buffer. Take a 4,000-token context — the KV cache is on the order of a few hundred MB in BF16 (illustrative), so recopying it to add one token moves hundreds of MB of redundant traffic per step (illustrative). On an NPU whose bandwidth is the binding constraint, that redundant traffic is pure latency. Reloading-free concatenation writes the new entry into a preallocated slot instead, dropping the recopy, while the tiled FLAT attention runs the unavoidable cache read as on-chip tiles rather than extra memory round-trips: FastTPS reports 6× faster token generation than the same model without fusion, at 93% of peak memory bandwidth on an AMD Ryzen AI NPU with Phi-3-mini-4k in BF16 (arXiv:2607.11211).

Token-phase strategy	How the KV cache is grown each step	Concat memory cost (illustrative)	What the accelerator does
Non-fusion baseline	reload + recopy the whole cache to concatenate	~all prior entries (illustrative)	redundant traffic on top of the attention read
Reloading-free concatenation	write the new key/value into a preallocated slot	~one written entry (illustrative)	removes the redundant recopy
+ FLAT RoPE attention + fused FFN (FastTPS)	append in place; tiled attention, fused MLP	~one written entry (illustrative)	~93% peak bandwidth, ~6× vs non-fusion (AMD Ryzen AI NPU, Phi-3-mini-4k BF16, arXiv:2607.11211)

Because the change lives in how the token phase touches memory rather than in the model's weights, it is a systems win, not an accuracy trade — the reported gains are relative to the same model run without fusion. The model didn't get smaller and the math didn't change — the accelerator just stopped recopying a cache it already held.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Continue in trackRoofline — why token generation is memory-bandwidth-bound

Related explainers

FastTPS attacks the KV-cache cost of the token phase by not recopying it to append; these siblings attack the same cost from other angles — shrinking it, compressing it, or splitting where it lives.

WorldKV — Evict-and-reinsert KV — bounds the cache by warehousing old entries and fetching them back, where FastTPS instead avoids recopying the entries it keeps
InfoKV — Entropy-aware KV compression — makes each KV entry smaller so there is less to move; complementary to reloading-free concatenation
AMD Atom — Prefill/decode disaggregation — another AMD-accelerator serving win, splitting the phases instead of fusing the token phase's memory access

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based