What is ForkAttention?

ForkAttention is the attention scheme inside the Flide inference engine. It puts a shared state and all the questions about it into one flat token stream, then makes each question see only the shared state and itself. In Gated DeltaNet layers it starts each question from the shared state's recurrent memory instead of using a mask.

Why can't a normal attention mask separate questions in a hybrid model?

Gated DeltaNet layers fold every earlier token into a single fixed-size state, so there is no attention matrix to mask. A question placed after its siblings would inherit them through that state. Flide runs the shared state first and starts every question from the saved state of its own request.

How is this different from prefix caching?

Prefix caching computes the shared state once but still runs every question as a separate sequence that resumes from the cache. On hybrid models the cache restores recurrent state only at block boundaries, so some tokens are recomputed. ForkAttention runs the state and all questions in one forward pass.

Does the speedup change the answers?

In exact arithmetic each question computes what it would compute alone. In floating point, differences stay at the rounding level, and the paper reports accuracy within 1.2 points of vLLM. An optional batch-invariant mode makes the output bitwise identical under any server load.

LLM· 2026-11-03

Flide answers decision queries 9.5× faster than vLLM — ForkAttention exact state forking — What does it mean?

TL;DR

What is it: The Flide paper (arXiv 2610.42148, 30 October 2026) introduces ForkAttention, which lays all the questions about one shared state into a single token stream and forks each question from that state's exact attention keys and recurrent memory.
Why it’s needed: Decision models read a long context and answer many short categorical questions about it, so an engine that recomputes or reloads the context for every question wastes most of its GPU time; Flide reports answers up to 9.5× faster than vLLM and SGLang on its evaluated workloads.
vs previous: Generation engines run every question as its own sequence, and multi-item scoring puts a delimiter token before each question, which shifts the probabilities; ForkAttention keeps each question's computation identical to asking it alone in exact arithmetic.

Jargon

Decision model: A small model that reads a context and returns probabilities over a few fixed choices (route here, approve, pick option B) instead of writing text. It answers in one forward pass, with no token generation.
Shared state (the trunk): The long part of a request that every question about it has in common, such as an article or an agent's history so far. Each question adds only a short tail of its own.
Prefix caching: Saving the keys and values of a prompt prefix so a later request skips recomputing it. See Prefix Caching.
Gated DeltaNet: A recurrent layer that folds every earlier token into one fixed-size matrix, its state, using a gated update rule. Hybrid models use it in most layers in place of attention.
Multi-item scoring: Putting all questions behind one shared context in a single sequence, each introduced by a delimiter token. It shares the context, but the delimiter becomes an extra input to every question.
Tree attention mask: A mask that lets each token see only its own branch and the trunk above it, so several branches can share one forward pass without seeing each other.
Total-variation distance (TV): Half the sum of the absolute differences between two probability distributions. The paper uses it to measure how far an engine's answers drift from a float32 reference.
Batch-invariant mode: A setting where every sum is computed in a fixed order whatever the batch looks like, so the same request returns the same bits under any server load.

The news. On October 30, 2026, researchers at the National University of Singapore and the University of Oxford posted Flide, an inference engine for decision models. It reports answers up to 9.5× faster than vLLM and SGLang on public hybrid and decision models, while staying closest to a float32 reference. Read the paper →

Picture a reader who studies a long case file and then has to answer three questions about it. The slow way is to hand the whole file to three separate people, one per question: each one reads the same 5,000 pages from the start. That is how a generation engine treats a request with three questions. It runs each question as its own sequence, so the shared file is read, or at best fetched back from a cache, once per question. The trick that sounds cheaper has a flaw too. If you staple all three questions behind the file with a sticky note before each, the file is read once, but every question now also reads the sticky notes, and that changes what the reader concludes. Multi-item scoring does exactly this with delimiter tokens, and the shifted probabilities can flip a decision.

ForkAttention does the third thing: read the case file once, then hand a copy of the reader's notes to each of three sealed booths. Every booth answers its own question from the file's notes and nothing else. In the paper's terms the request is a tree. The shared state is the trunk and each question is a branch that must see only the trunk and itself. Flide keeps that tree as the logical structure and lays the tokens out as one flat stream, the state first and then every question, with no padding. Each token carries three small tags: its request, its branch (the trunk or question j) and its position. Position numbering continues from the end of the state, so a question's rotary embedding is the one it would get when asked alone.

Only the layers that mix tokens need to know about the tree. Linear projections, normalization and MLPs treat each token on its own. In full-attention layers the booth walls are a mask: a query token may attend to a key only if it is in the same request, in the shared state or in its own question, and not later in the stream. This is the same tree attention that speculative decoding uses for draft candidates. Gated DeltaNet layers cannot use a mask, because they squeeze every earlier token into one matrix-valued state, and a question that read that state after its siblings would inherit them. So Flide runs each such layer in two calls. The first processes all the shared states and outputs each one's recurrent state and convolution window. The second processes all the questions together, and each question starts from the state of its own request, picked by an index and not copied.

A decision request can take only a few milliseconds of GPU time, so launching thousands of kernels one by one would cost more than the math. CUDA graphs remove the launches but freeze the shape of each one, while the tree of requests changes every round. Flide keeps the tree as data in GPU memory: segment lengths, positions, key ranges and state indices sit in small device buffers that every kernel reads. The launch shape then depends only on the length of the flat stream, so a family of graphs captured at startup, one per stream capacity, serves any mix of requests. Flide picks the smallest graph that fits the round. It also reads the output head at the last token of each question only, which skips most of the final layer's work without changing an output.

Approach	Shared state computed	Each question sees	Weak point
One sequence per question (vLLM, SGLang)	Once, then reloaded per question from the prefix cache	Its own standalone prompt	On hybrid models the cache restores recurrent state only at block boundaries, so tokens past the last boundary are computed again
Multi-item scoring	Once	The state plus a delimiter token	The delimiter changes hidden states and can flip decisions; run without CUDA graphs in the paper's comparison
Flide ForkAttention	Once, in the same pass as the questions	Exactly its standalone prompt, in exact arithmetic	Built for categorical decision workloads, not text generation; measured on one H200

Here is the arithmetic from the paper's own workload table, the QuALITY articles with every question sent in one request. On average a request has a 5,835-token state and 18.1 questions. Run as one sequence per question, the model processes 107,085 tokens per request. In Flide's flat stream the state appears once and each question once, so the stream is 7,454 tokens, which is 14.4× fewer tokens. The questions themselves add only about 1,600 tokens in total (7,454 − 5,835), roughly 90 tokens each, so nearly all the work was the repeated state. The ablation on Qwen3.5-4B agrees: with forking turned off, the same workload takes 1,868.5 ms instead of 135.6 ms, about 13.8× longer. The headline numbers are a ceiling and not a typical case: against the best baseline Flide cuts latency by up to 9.5× and raises throughput by up to 8.5×, the gain grows with questions per state, and a lone request still gets up to 3.2× lower latency on the smallest Qwen3.5 model (0.8B).

The same fork idea covers an agent that decides again after every step. Each step's state is mostly the previous step's state with a short tail added. Flide checkpoints a request at a position p placed slightly before the end of the state, so the parts likely to change, such as the end of a conversational turn, fall after the checkpoint. A checkpoint stores the float32 recurrent state and convolution window of every Gated DeltaNet layer at p, plus the keys and values of the full-attention layers before p. The next step resumes from it, computes only the short tail and the new tokens, saves a new checkpoint in the same pass, and forks its questions from the finished state. This is the decision-model counterpart of what the agent loop carries from one step to the next.

Two limits matter before you read the speedups as general. First, exactness is a statement about exact arithmetic: in floating point the paper says differences stay at the rounding level and checks them with tests on every engine change. Second, bit-for-bit repeatability needs the optional batch-invariant mode. Without it, going from 1 to 64 concurrent clients changes at most 0.2% of Flide's answers, up to 1.7% of SGLang's and up to 3.2% of vLLM's. Accuracy stays within 1.2 points of vLLM in every cell of the paper's table. All timings are from one NVIDIA H200 on Qwen3.5 and Gemma 4 models, and the paper does not claim anything for free-form text generation.

Goes deeper in: LLM Serving → Prefix Caching & RadixAttention → The Prefix Tree (RadixAttention)

Related explainers

Tree speculative decoding on DeepSeek-V4 — the same branch-isolation problem, applied to draft tokens instead of decision questions.
JetSpec parallel tree drafting — another way to verify many branches in one forward pass.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based