Rimuru unifies serving memory — Shared page pool for KV cache and MoE experts — What does it mean?

The news. On October 29, 2026, a team including researchers at Sun Yat-sen University posted the Rimuru paper on arXiv. It reports up to 19.03× the throughput of one offloading baseline, KTransformers, but only on a two-RTX-4090D server under high workload, where that baseline falls back to slower CPU instructions because the CPU lacks AMX (special CPU matrix instructions). It also reports a 98.1% average cut in time to first token. Read the paper →

Picture a warehouse floor with a fence painted down the middle. One side is reserved for expert shelves, the other for order slips, order slips that pile up for each customer, a few more with every item added. The fence was drawn on day one, and it never moves. On a busy afternoon the order side overflows and customers wait outside, while shelves on the other side hold experts nobody is asking for. On a quiet morning the order side is half empty, and it still cannot lend a single tile to hold one more popular expert.

That is the situation the paper describes for MoE serving. Engines built on PagedAttention carve the KV cache into small blocks that a table points to, which removes waste inside the KV region, but the model weights live outside that managed pool. Expert offloading moves some experts to CPU memory to free room, yet the expert cache is still a fixed size. The paper's motivating example is DeepSeek-V4-Flash on eight A100 GPUs, which it says leaves only about 7 GB per GPU for attention state, enough for just two concurrent 1M-token requests. Because the KV cache grows with every token, a rigid fence turns directly into a cap on how many requests run at once. (The paper also covers other per-request states, such as state-space-model states, but the KV cache is the clearest example.)

Moving the fence is harder than it sounds, because the two kinds of goods are not the same size. KV blocks are small, usually 16 tokens each, while an expert's weights are large. If every page had one size, small KV blocks would waste space or large experts would need huge page tables. Rimuru's answer has three parts. First, each tensor type gets its own memory view: its own page size and its own page table laid over the same physical pool. Second, the pool is cut into huge pages whose size is the least common multiple of the expert and state-space-model page sizes, so a boundary never splits another type's page. Third, a multi-level allocator hands out pages by splitting: a huge page becomes expert-sized pages, and an expert-sized page can in turn become KV-sized pages. Reclaiming runs the same path in reverse. The paper says it avoids cutting a huge page straight into KV blocks to limit fragmentation, and it tries to cut one request's smaller pages from the same parent page so freeing them is simple. This is the same block-table idea from PagedAttention, stretched across two tensor types, and the memory manager inside a serving engine is the part that has to change.

Two more pieces make the moving fence usable. Once an expert lives on scattered pages, an ordinary fused MoE kernel would need them copied into one contiguous buffer first. Rimuru's page-table-aware fused MoE kernel reads the expert pages where they sit, so no copy is needed. And a workload-aware adjuster decides when the fence moves: it grows the KV side when concurrency climbs, and returns space to experts when requests finish and KV pages are freed, while coordinating with the prefetching pipeline so expert loads and evictions do not collide. The shared pool by itself buys nothing; the paper's ablation says it only provides the mechanism for sharing.

Here is that table as a worked example, using the paper's own case study: Qwen1.5-MoE-14B on one RTX 5090, ShareGPT requests arriving at 10 per second, GPU memory capped at 0.9. Hold the model, the memory budget and the request stream fixed, and change only the memory design. Offloading alone lifts peak concurrency from 34 to 74 requests and cuts the steps needed from about 2,100 to about 1,300. Letting the boundary move lifts it to 90 requests and about 1,000 steps. By simple arithmetic on those figures, the step count drops about 2.1× against plain vLLM (2,100 / 1,000) and about 1.3× against offloading alone (1,300 / 1,000). The step count falls by less than the 2.6× jump in peak concurrency (34 to 90), which is expected: peak concurrency is a ceiling reached at times, not an average, so the two numbers need not move in lockstep.

The headline numbers need their conditions attached. The paper's ablation on Qwen3.5-35B and Qwen3-30B with two RTX 5090 GPUs starts from the Fate offloading system and adds pieces one at a time: unified memory alone stays near the baseline, adding the adjuster reaches about 3.85× throughput, and adding the page-table-aware kernel reaches about 4.19×. Against vLLM and SGLang the reported gain is a more modest 1.1–2.88× throughput. The tests use Qwen models, RTX 4090D and 5090 GPUs and ShareGPT traffic, so the article cannot say how the gains carry over to datacenter GPUs or other workloads. Treat 19.03× as a best case against one baseline, and treat the 1.1–2.88× range against vLLM and SGLang, the engines people actually run, as the more useful signal.

Goes deeper in: LLM Internals → PagedAttention → Block Tables & Allocation