K2-Horizon open models — Mixture-of-Value Attention — What does it mean?
The news. On November 16, 2026, the K2 Team at the Institute of Foundation Models posted the K2-Horizon technical report on arXiv. It documents six fully open models (0.9B to 375B-A23B) with checkpoints, data, training logs and code. The architecture section introduces Mixture-of-Value Attention, used together with a softplus attention gate (a learned, always-positive per-element scale that can shrink or boost the attention output) in the 36B-A4B model, which the report says "nearly matches a 32B dense model trained on the same data, with one-seventh the active parameters." Read the report →
Start with the library picture from the Q, K, V step: a reader brings a question (the query), checks every book's spine label (the key), and reads the book's summary card (the value). In a standard transformer, one clerk writes every summary card, and writes it the same way for every book. That clerk is the value projection: a single matrix multiplied by each token's hidden state. MoVA replaces the one clerk with a desk of 64 specialist writers and a dispatcher who, for each book, picks 4 of them and blends their drafts into a single card.
The important part is what does not change. The spine labels are untouched, so readers find books exactly as before; the attention scores, the softmax and the weighted sum of values all run as normal. And the shelf still holds one card per book, the same size as the card the lone clerk used to write, because the drafts are mixed before filing. In the report's words, each value vector "is fully determined once token t is processed, so only the mixed value vector is cached."
In model terms, a router scores the 64 value experts for the current token, keeps the top 4, and takes a weighted sum of their outputs. Each expert is its own value matrix, and the report applies a SiLU to each expert's output before mixing. The reason is a small algebra point: without the activation, a weighted sum of linear projections is just one blended linear projection, so the 64 experts would add little beyond an input-dependent mixing of matrices. With SiLU, each expert behaves like a small nonlinear network, closer to an ordinary MoE expert in the feed-forward layer.
This is why the report calls MoVA "a second axis for scaling sparsity". Mixture-of-Experts models already store far more feed-forward weights than any single token uses. MoVA does the same inside attention, and it does it at the one point where extra weights cost no cache: the value is computed from the token's own hidden state, before any token looks at any other. Because the attention computation itself is unchanged, the report says MoVA stays compatible with FlashAttention, GQA and sparse attention.
A worked example, per layer, using the 36B model's published dimensions. The model has a hidden size of 2,560 and 8 KV heads of dimension 128, so each token's value vector holds 8 × 128 = 1,024 numbers, and one value projection is a 2,560 × 1,024 matrix, about 2.6M weights. MoVA keeps 64 such experts, about 168M value weights in that layer, but each token runs only 4 of them, about 10.5M: 4× the work of the single projection, and roughly 1/16 of the stored experts. The cached value is still 1,024 numbers per token per layer, because the 4 outputs are blended into one vector before it is written. Had the 4 expert outputs been cached separately, the value half of the cache would be 4× larger. (Our arithmetic from the report's dimension table; it ignores the router's own weights.)
| Design | Where the sparsity sits | Value computation per token | Value cache per token, per layer |
|---|---|---|---|
| Standard attention + dense FFN | None | One linear projection (about 2.6M weights at the 36B model's sizes) | 1,024 numbers |
| MoE model | Feed-forward layer only | One linear projection, unchanged | 1,024 numbers |
| MoVA (K2-Horizon 36B-A4B) | Feed-forward layer and value projection | 4 of 64 SiLU value experts, weighted sum | 1,024 numbers, unchanged |
| Caching each picked expert separately (counterfactual, illustrative) | Value projection | 4 expert outputs | ~4,096 numbers, 4× larger |
How much does it buy? The report's 36B-A4B model combines MoVA with MoE feed-forward layers and activates about 4.6B parameters per token. Its pre-training loss "closely tracks" a dense 32B model trained on the same data with identical settings, with the dense model keeping a modest lead, and at 1.1M steps it trails that model by 1.2 to 3.7 points on MMLU, GSM8K, HellaSwag and HumanEval. That comparison measures the whole package — MoVA, MoE feed-forward layers and the softplus gate together — not MoVA on its own, so it shows the design is competitive at about one-seventh the active parameters, not how much of that comes from routing the values. In the library picture: the desk of writers produced cards good enough that readers barely noticed the difference, but the report does not test the desk with the other renovations removed.
Goes deeper in: LLM Internals → Attention → Q, K, V: Three Roles
Related explainers
- K2-Horizon — Uno's LoRA diffusion adapter for block-parallel decoding — the other architecture idea from the same model family, aimed at decoding speed instead of capacity
- Total vs active parameters in a 2-bit MoE — the stored-versus-run distinction that MoVA extends into attention