What are Self and Exchange relations?

They are the two channels that the Relation operator splits pairwise evidence into before turning any of it into information flow. The Self relation is the part a position derives from its own state; the Exchange relation is the part that comes from other positions. Standard attention keeps both in one budget of weights that sums to one, so the two cannot be scaled independently; Relation's claim is that giving them separate channels is worth re-deriving the mixer for. The paper names the two channels and their ordering in the framework, but its summary does not publish the algebra that computes them.

Why does it matter when the normalization happens?

Because a softmax makes the weights compete for a fixed budget: they sum to one, so more weight on one position necessarily means less on another, which is a constraint on what the layer can express rather than a bug in the arithmetic. Deriving the flow after the two channels are formed removes that constraint from the scoring step. The evidence offered for the rearrangement is a loss result — Full Relation reports a lower final validation NLL than multi-head attention at matched decoder-only sizes of roughly 10M, 30M and 100M parameters.

How does FlashRelation relate to FlashAttention?

It occupies the same slot in the framework that FlashAttention occupies for attention: the fast kernel that makes a pairwise operator affordable, as opposed to the naive materialized form that writes the whole quadratic matrix to memory. The summary does not say what technique FlashRelation uses; what it reports is the outcome, at 3.60 to 4.41 times faster than materialized Full Relation and 76.4 to 84.9 percent of PyTorch FlashAttention's throughput. The practical reading is that the kernel closes most, but not all, of the gap to a heavily tuned attention kernel, so the operator's case has to rest on quality rather than speed.

Relation reports lower loss than MHA at 10M–100M params — Self and Exchange relations

TL;DR

What is it: The news anchor is an arXiv paper, "Ask Self, Ask Others: Relation Is All You Need"; the concept it makes concrete is Self and Exchange relations — organizing the pairwise evidence between tokens into two explicit channels, and deriving the information flow only after both are formed — rather than reading it straight off a set of attention weights, the way the standard mixer does.
Why it’s needed: Token mixing is the one operation in a transformer where positions are allowed to see each other, so the shape you give it decides what the model can express — and the paper reports that at matched sizes of roughly 10M, 30M and 100M parameters, its Full Relation operator reaches a lower final validation loss than multi-head attention.
vs previous: Standard attention normalizes immediately — pairwise scores go through a softmax in the same breath that produces them, so a position's own evidence and its evidence about others land in one shared budget of weights that sum to one — whereas Relation keeps the two as separate channels and derives the flow only afterwards.

Think of it as

a detective who keeps two notebooks: what I saw myself, what witnesses said

notebook one, what I observed myself = the Self relation, the evidence a position derives from its own state
notebook two, what each witness told me = the Exchange relation, the evidence that comes from other positions
the old habit of merging both into one ranked list of who to believe, shares adding to 100% = the softmax that turns pairwise scores straight into attention weights
writing the conclusion only after both notebooks are filled = deriving the information flow after the relations are formed, rather than during scoring
a shorthand that fills the same two notebooks without recopying every page = FlashRelation, the efficient kernel for the same operator
the case file carried from interview to interview so old testimony is not re-taken = the Relation Cache, the KV-cache analogue for this operator

Jargon

Token mixing: The step in a transformer layer where positions are allowed to read each other, while everything else in the block — the feed-forward network, the norms — works on each position in isolation. Attention is the usual token mixer; Relation is proposed as a different one.
MHA (multi-head attention): The standard token mixer: project each token into a query, a key and a value, score every query against every key, softmax the scores into weights, and average the values with them — repeated across several independent heads. It is the baseline the paper measures against.
Self relation: The first of the two channels the paper separates out — the part of the pairwise evidence that a position derives from itself. The summary names the channel and its role in the framework; it does not give the algebra that produces it.
Exchange relation: The second channel — the part of the evidence that comes from other positions. Keeping it distinct from the Self channel is what lets the operator "derive information flow" as a later step instead of reading it straight off a softmax.
NLL (negative log-likelihood): The training and validation loss of a language model, in nats per token: how surprised the model was by the text it had to predict. Lower is better, and at a fixed model size and data budget it is the standard way to compare two architectures.
Materialized: Actually writing the full pairwise matrix out to memory before using it. The naive way to run any pairwise operator, and the reason FlashAttention exists — the matrix is quadratic in sequence length, so materializing it makes the operation move far more bytes than it computes on.
Relation Cache: The framework's KV-style cache for this operator: the per-position state that a decode step can reuse instead of recomputing, exactly as the KV cache does for attention. The summary names it as part of the framework and does not give its size.
Hybrid Relation: The configuration in which 75% of the Relation layers are the Linear Relation variant — the same per-layer knob other hybrid architectures expose. That proportion is the only thing the summary reports about it: not what the remaining quarter of the layers is, and not what either variant costs.

The news. On August 21, 2026, researchers posted "Ask Self, Ask Others: Relation Is All You Need", which proposes Relation as a token-mixing primitive that organizes pairwise evidence into explicit Self and Exchange relations before deriving information flow — rather than normalizing pairwise scores into attention weights straight away. The framework defines five pieces: Full Relation, FlashRelation, Linear Relation, Hybrid Relation, and a KV-style Relation Cache. Across matched decoder-only models at roughly 10M, 30M and 100M parameters, Full Relation is reported to reach a lower final validation NLL than MHA at all three scales; FlashRelation is reported 3.60–4.41× faster than materialized Full Relation and to reach 76.4–84.9% of PyTorch FlashAttention throughput; Hybrid Relation uses 75% Linear Relation layers. Read the paper →

Picture a detective working a case. Two things arrive: what they saw at the scene themselves, and what each witness told them. A careless detective merges the two immediately — one ranked list of who to believe, shares adding up to one hundred percent — and from then on the two can no longer be weighed apart: every point of credit given to a witness is a point taken away from what the detective saw, because both are being spent out of the same hundred. The careful detective keeps two notebooks and writes the conclusion afterwards.

That flattening is what attention does, and it is easy to miss because it is one line of math. A query is scored against every key, and those raw scores are handed straight to a softmax that turns them into weights summing to one — the step from scores to weights and the weighted average that follows are the whole mixer. Because normalization happens at the same moment the evidence is produced, a position's reading of itself and its reading of every other position end up sharing a single budget — the weights sum to one, so the two cannot be turned up or down independently.

Relation's proposal is to postpone that moment. The pairwise evidence is first organized into two named channels — a Self relation and an Exchange relation — and the information flow is derived from them afterwards, so the "who did this come from" structure survives into the step that uses it. In the detective's terms, both notebooks get filled before any conclusion is written. What the paper's summary states is the framework and the ordering; it does not publish the algebra that forms the two channels, so how Self and Exchange are computed is not something this article can tell you.

The result reported for that rearrangement is a loss result rather than a speed one: at matched decoder-only sizes of roughly 10M, 30M and 100M parameters, Full Relation reaches a lower final validation NLL than MHA at all three scales. That is a real signal and a narrow one — three points, all of them small by current standards, and the summary makes no claim about what happens at a billion parameters or beyond. A token mixer that wins at 100M has cleared the first hurdle, not the last one.

The second half of the framework is about making the operator affordable, which is where the parallel with attention's own history is hard to miss. Written naively, a pairwise operator is materialized: the whole quadratic matrix is written to memory and read back, so the kernel spends its time moving bytes rather than doing math — exactly the problem FlashAttention was built to solve by never materializing the matrix at all. FlashRelation is the framework's answer to that cost, and it is reported to be 3.60–4.41× faster than the materialized Full Relation and to reach 76.4–84.9% of PyTorch FlashAttention's throughput. How it gets there is not something the summary describes — what it names is the target (efficient execution of the Full Relation operator) and the baseline it is measured against.

Here is what those two numbers mean together, in wall-clock. Hold one quantity fixed: a mixing pass that PyTorch FlashAttention finishes in 100 ms (a round illustrative number — the paper's figures are ratios, not absolute times). Price the same pass in two formats. At the reported 76.4–84.9% of FlashAttention throughput, FlashRelation takes 100 / 0.849 to 100 / 0.764 — that is 118 to 131 ms, a premium of roughly 18–31% over the attention kernel it is competing with. Now the naive version of the same operator: FlashRelation is reported 3.60–4.41× faster than materialized Full Relation, so the materialized form lands at 424 to 577 ms — four to six times the FlashAttention pass. (The two ratios are each the paper's; composing them into one timeline assumes both were measured on the same configuration, which the summary does not state, so treat the 424–577 ms figure as illustrative rather than reported.) The gap between those two prices — a 118 ms operator and a 424 ms one — is the entire reason the framework ships a kernel and not just an equation.

Piece	What it is	What the paper reports about it
Full Relation	the operator itself — evidence split into Self and Exchange channels, flow derived afterwards	lower final validation NLL than MHA at ~10M, 30M and 100M parameters
FlashRelation	the efficient execution path for the Full Relation operator; the summary does not describe the technique	3.60–4.41× faster than materialized Full Relation; 76.4–84.9% of PyTorch FlashAttention throughput
Linear Relation	the linear-named member of the family, standing to Full Relation roughly as linear attention stands to softmax attention	named as part of the framework; no standalone quality or speed figure in the summary
Hybrid Relation	a stack in which three quarters of the layers are the linear variant	uses 75% Linear Relation layers
Relation Cache	the KV-style cache that makes decoding incremental rather than quadratic per step	named as part of the framework; the summary gives no size or hit-rate figure

What makes this worth reading past the benchmark line is the kind of claim it is. Most attention research of the last two years takes the softmax mixer as fixed and changes what it is allowed to look at — sparsity drops entries, linearization trades exactness for a cheaper recurrence, per-head hybrids keep exact attention only where it earns its cost. Relation instead changes the shape of the primitive: it argues the mistake is normalizing at the moment the evidence is produced, and that keeping "from myself" and "from others" as separate channels is worth a full re-derivation of the mixer. That is a bigger claim, which means a bigger burden — and the honest summary of the evidence so far is three small-scale loss wins and a kernel that runs a fifth slower than the one it wants to replace. The interesting question is not whether the operator works, but whether the gap that shows up at 100M parameters survives at the scales anyone deploys.

Goes deeper in: LLM Internals → Self-Attention → Computing Attention Scores

Related explainers

HydraHead — head-axis attention hybridization — the other axis you can hybridise a mixer along: by head instead of by layer, where Hybrid Relation makes the split across layers.
Parallax — local-linear attention — another attempt to change what the mixer computes rather than what it looks at, upgrading the weighted average itself.
I/O-optimal approximate attention — the bytes-moved view that makes the materialized baseline so expensive in the first place: why writing a quadratic matrix out to memory costs several times the speed.

Continue in trackSelf-Attention: how scores become weights

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based