What is just-in-time state residency?

It is the idea that a runtime should decide **when** each piece of execution state occupies memory, not just how small that state can be made. JustFit reconstructs compressed KV while a step runs, swaps which components are resident per serving phase, and preserves state across phase changes — so the peak amount resident at any one moment is far below the total the request touches.

Why does it matter on a laptop?

A 24 GiB machine has no headroom to grow into. Model weights are a fixed cost, but the KV cache grows with every token, so context length is what exhausts the budget first. Scheduling residency raised completed single-request context from **30,720 to 212,992 positions** on the same hardware — a 6.93× increase with no change to the model.

How is this different from KV-cache compression?

Compression asks how few bytes a cached token can occupy; residency scheduling asks how long those bytes need to be in memory at all. The two compose: JustFit does keep the cache compressed, but the capacity win comes from reconstructing and releasing it around each phase rather than from a higher compression ratio.

Does it require quantizing the model weights?

No. The paper reports that its mechanisms coordinate just-in-time materialization and release **independently of model-weight quantization**. The measurements happen to use an MXFP4 model, but the residency scheduling is applied to execution state, so it stacks on whatever weight precision is already in use rather than replacing it.

LLM· 2026-09-18

JustFit serves a 212,992-token context on a 24 GiB laptop — Just-in-time state residency — What does it mean?

TL;DR

What is it: JustFit (arXiv 2609.17475) is an MLX inference runtime for laptops, and the concept it makes concrete is just-in-time state residency — deciding when each piece of execution state has to be in memory, instead of only how small it can be squeezed.
Why it’s needed: On a fixed-memory machine the model weights and the KV cache compete for the same bytes, and only one of them grows with every token — so on a laptop it is the context length, not the parameter count, that runs you out of room first.
vs previous: A conventional local runtime materialises state once and keeps it resident for the whole request; JustFit reconstructs and releases it around each phase, lifting completed single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 on the same hardware.

Jargon

KV cache: The stored keys and values from every token the model has already processed, so each new token can look back without recomputing the past. Covered in LLM Internals.
Residency: Whether a piece of data is actually occupying memory right now. A value can exist logically (on disk, or compressed) without being resident, and only resident bytes count against the budget.
KVExec: JustFit's first mechanism: it keeps the KV cache in compressed form and reconstructs what a step needs while that step runs, so the decompressed cache never has to sit in memory waiting for its turn.
PhaseSwap: JustFit's second mechanism: it changes which components are resident as the runtime moves between serving phases, rather than holding every component for the whole request.
StateTrans: JustFit's third mechanism: state-preserving serving transitions. It carries live state across a phase change so the runtime does not have to rebuild what it just released.
MLX: Apple's array and machine-learning framework for Apple silicon. It is the runtime layer JustFit is built on, which is why the paper's numbers come from a MacBook rather than a datacenter GPU. Its baseline, mlx-vlm, is the existing MLX serving stack JustFit is measured against.
MXFP4: A 4-bit floating-point format used to store model weights compactly. JustFit's mechanisms work on execution state and are reported to operate independently of it — see load-time NVFP4-to-MXFP4 requantization and what to quantize.

The news. On September 15, 2026, a paper describing JustFit appeared on arXiv (2609.17475): an MLX-based inference runtime that serves long contexts on consumer Apple silicon. On a 24 GiB M4 Pro MacBook running Qwen3.8-27B at MXFP4, three independent runs completed 196,608 input and 16,384 output tokens — 212,992 positions in a single request, against 30,720 for the mlx-vlm baseline. The runtime also answered 29 of 30 AIME 2026 problems correctly, so the extra capacity did not come at the cost of the reasoning it was holding. Read the paper →

Picture the stage again. The pink stage is not smaller than the green one — both are 24 GiB — but on the left every set the show will ever need is standing on it from the first minute, so the last one topples off the edge before the show reaches its final scene. On the right the same stage carries a much longer show, because only the set for the current scene is standing: the rest wait flat-packed at the side and are assembled on cue.

Translated back, the flats are execution state, and the one that grows without limit is the KV cache — the stored keys and values that let each new token look back over everything before it. Model weights are a fixed cost: you pay for them once and they sit there. The KV cache is not. It grows with every token the model reads or writes, which is why on a fixed-memory machine it is the context length, not the parameter count, that decides when you run out. Multiply the KV cache out per token and the slope is steep.

JustFit's answer is not to make the state smaller but to make it resident for less of the time. Three named mechanisms split that job between them — one for the cache itself, one for which components are up at all, one for what survives a phase change — and the table below sets out what each controls. What matters more than the names is the claim underneath them: the paper is explicit that all three operate independently of model-weight quantization. This is a scheduling idea, not a compression idea, and it composes with whatever precision the weights are already stored in. Where paged attention asks where a block of cache lives, this asks when it has to exist at all; the component that has to answer is the runtime's memory manager.

Mechanism	What it manages	Why it frees memory
KVExec	Compressed KV execution	The cache is reconstructed as a step runs, so a decompressed copy never has to be resident
PhaseSwap	Component residency	Only the components the current serving phase needs are resident at all
StateTrans	Serving transitions	State survives a phase change instead of being released and rebuilt

Hold three things fixed and the size of the win is easy to read. Same machine — a 24 GiB M4 Pro MacBook. Same model at the same precision — Qwen3.8-27B at MXFP4, so the weights occupy the same share of memory in both runs; the paper states its mechanisms work independently of model-weight quantization, so nothing about the weights changed between them. That leaves exactly one variable: how the execution state is held.

Under the mlx-vlm baseline the machine completed 30,720 positions in a single request — one position per token held in the context. Under JustFit it completed 196,608 input plus 16,384 output tokens — 212,992 positions, or 212,992 ÷ 30,720 = 6.93× the context on the same hardware with the same weights. Stated as a difference rather than a ratio: 182,272 additional positions came out of scheduling alone. A separate two-request run retained 229,376 positions in aggregate, more in total across two requests than the single-request ceiling — consistent with residency being scheduled across requests rather than reserved per request, though the paper reports the figure without breaking down the cause.

The performance probe is a different workload and should not be folded into that ratio: a 32K-input, 64-output run reached 19.11 tokens per second, and a repeated 32K+6K workload had a median peak process footprint of 16,374 MiB — roughly two-thirds of the 24 GiB the machine has, on a workload far shorter than the capacity test.

Goes deeper in: LLM Internals → KV Cache → Memory Cost

Related explainers

Matryoshka memory budgets — the other lever on the same problem: how far to compress the context, rather than when to hold it
Query-cluster KV residency prediction — which cached entries to keep when you cannot keep them all
Partial loads from a lower cache tier — what changes when the state lives on a tier below the accelerator

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based