CoMem is a soft context-compression method posted to arXiv on November 12, 2026. An encoder LLM reads the query, a document chunk and a row of memory tokens, and the hidden states of those tokens become a short memory the decoder reads instead of the chunk. Training adds a contrastive loss whose negatives are memories of the same document under different questions.

Why condition compression on the query?

Under a fixed memory budget the compressor must drop most of the document, and what is safe to drop depends on the question. A query-agnostic memory keeps generic content and can miss the one fact an answer needs; in the paper at 32×, ICAE kept a hidden 7-digit needle in 30.8% of cases and CoMem in 61.2%.

What are same-context hard negatives?

They are training negatives built from one document paired with several different questions. Memories of different documents are easy to tell apart, so they teach little; memories of the same document can only be separated by encoding what each question needs. In the paper, they beat different-document negatives on all four QA datasets.

How does this relate to prefix caching?

A query-agnostic memory can be computed once per document and reused, like a cached prefix. A query-conditioned memory must be recomputed for every question. CoMem's generic-instruction mode allows precomputation but cost 17.4 points on HotpotQA at 32× with Qwen3-4B, so the choice is reuse versus recall.

LLM· 2026-11-15

CoMem — Query-conditioned memory slots — What does it mean?

TL;DR

What is it: The CoMem paper (arXiv, November 12, 2026) is a soft context-compression method, and the concept it makes concrete is query-conditioned memory slots: the compressor reads the question together with the document, so the few vectors it keeps hold the evidence that question needs.
Why it’s needed: In RAG and long-document question answering, sending every retrieved token to the model costs latency and KV-cache memory, and a compressed memory is only worth having if the one fact the answer depends on survives the squeeze.
vs previous: Earlier soft compressors such as ICAE and PISCO turn a document into the same memory whatever the question, and a query-aware method that only concatenates the query still produced near-identical slots across questions; at 32× compression ICAE kept the hidden needle in 30.8% of RULER-NIAH cases and CoMem in 61.2%.

Jargon

Soft context compression: Turning a long text into a few learned vectors that the model reads in place of the text. Hard compression instead deletes words and keeps a shorter, readable prompt (LLMLingua-2 is an example).
Memory slots: Extra learnable tokens appended after the document; their final-layer hidden states become the compressed context. At 32× compression a 256-token chunk becomes 8 slots.
Compression ratio: Original tokens divided by memory slots. 4× keeps 64 slots per 256-token chunk; 32× keeps 8.
Contrastive learning: A training signal that pulls a matching pair of vectors together and pushes non-matching vectors apart, so different inputs land in different places in vector space.
Hard negative: A non-matching example that is difficult to tell apart from the match. In CoMem it is the memory of the same document under a different question.
Causal mask: The attention rule in decoder LLMs where each position sees only earlier positions. It is what makes left-to-right generation work, and it also stops early memory slots from seeing later ones.
RULER-NIAH: A needle-in-a-haystack test: a 7-digit number is hidden among five documents and the model must return it. It checks whether compression kept one specific fact.
KV cache: The stored keys and values of every token the model has read; its size grows with context length. See KV Cache → Memory Cost.

The news. On November 12, 2026, researchers from the Institute of Automation (Chinese Academy of Sciences), Alibaba Group and Hello Group posted CoMem to arXiv. It compresses a query-plus-document pair into memory slots, adds a contrastive loss whose negatives all come from one document paired with different questions, and lets the slots attend to each other. On four question-answering benchmarks and RULER-NIAH, with Mistral-7B and Qwen3-4B, the paper reports that it beats hard and soft compression baselines at both 4× and 32× compression. Read the paper →

Picture handing a forty-page report to an assistant who may return only four index cards. If you do not say what you will ask, the assistant writes safe, generic cards: the headline, the main finding, the conclusion. Ask later about a figure buried on page 31 and it is not on any card. Hand over the question first, and the same four cards can hold the page-31 figure, because the assistant now knows what is worth the space. That is the difference between query-agnostic and query-conditioned compression, and it is the fidelity cost of compaction moved inside the model: a lossy summary loses whatever the summarizer did not know would matter.

In CoMem the assistant is an encoder LLM (a Mistral-7B or Qwen3-4B fine-tuned with small adapter weights). Its input is the query, then a 256-token document chunk, then a row of learnable memory tokens; the final-layer hidden states of those tokens are the compressed memory. The decoder then sees the query and the memory instead of the chunk. In the retrieve-then-generate pipeline, the step that stuffs retrieved text into the prompt now stuffs in a handful of vectors.

Retrievequery + 256-token chunks

Compressencoder reads query, chunk, memory tokens

Memory slots8 vectors per chunk at 32×

Decodequery + slots, no raw chunk

AnswerKV cache holds query + slots

Giving the assistant the question is not enough on its own. The authors show that SeleCom, which concatenates the query with the document before compressing, still writes nearly identical slots for different questions about the same document. The paper's explanation is the training signal: an answer-likelihood loss supervises the compressor only indirectly, so as long as the decoder can recover the answer, nothing forces the cards to differ, and the question-specific work drifts to the decoder.

CoMem adds a contrastive loss. The average of a memory's slots is pulled toward the average hidden state of its own document, so the cards stay faithful to the report, and pushed away from the other memories in the batch. The trick is which memories share the batch: each batch is one document paired with eight different questions, so the only way to push those eight memories apart is to encode what each question needs. In the note-taker picture, a trainer hands over the same report eight times with eight different questions and rejects any two card sets that look alike. Memories of different documents are already far apart in vector space, so they make easy negatives; in the paper's ablation, same-document negatives beat different-document negatives on all four datasets (HotpotQA 52.1% vs 49.5%).

The third piece is the attention mask. Under a causal mask, slot 1 cannot see what slot 8 holds, so the slots cannot divide the evidence between them. CoMem keeps causal attention over the query and document but makes the memory slots attend to each other in both directions, like cards laid side by side so no two repeat the same fact. The ablation shows the two changes depend on each other: without the contrastive loss the mask choice matters little, and with it a causal mask causes a large drop.

Method (Mistral-7B, 32×)	Type	HotpotQA	NQ	RULER-NIAH	Source
Full context (no compression)	baseline	77.5%	79.1%	88.6%	Table 1
LLMLingua-2	hard, query-agnostic	22.4%	19.5%	38.2%	Table 1
ICAE	soft, query-agnostic	31.6%	38.6%	30.8%	Table 1
PISCO	soft, query-agnostic	56.8%	67.3%	57.8%	Table 1
SeleCom	soft, query concatenated	34.5%	65.3%	0.0%	Table 1
CoMem	soft, query-conditioned + contrastive	62.0%	73.9%	61.2%	Table 1

The numbers are Cover Exact Match accuracy (the answer contains the gold span). CoMem is the best compressed row at 32× but still sits below the uncompressed baseline, so it narrows the cost of compression rather than removing it. At 4× the paper reports CoMem scoring above full context on some benchmarks under an LLM judge (NQ 81.9% vs 76.7%), and at 64× with Qwen3-4B its first-stage compressor scored 98% on Any Signal, which counts the full 7-digit needle or a recognizable part of it; the gap to exact recovery suggests many remaining misses are corrupted digits rather than lost evidence.

Here is what the trade buys, holding the paper's efficiency setup fixed: a 128-token query, five retrieved 256-token documents (1,280 document tokens), 32 generated tokens, one H100. At 32× the five documents become 1,280 ÷ 32 = 40 memory slots, and decoder time per sample falls from 0.053 s to 0.012 s, about 4.4× faster. That timing deliberately excludes the compression pass. With the true question, compression cannot happen ahead of time, because the memory depends on the question. The paper's way around this is a generic instruction ("preserve all potentially useful information") in place of the question, which lets memories be built once per document and stored. On Qwen3-4B at 32×, that reusable memory costs 17.4 points on HotpotQA (63.5% → 46.1%) and 14.2 points on NQ (70.1% → 56.0%), although the paper reports it still beats ICAE and LLMLingua-2 on all four datasets.

This is the same split as prefix caching for RAG, where the shared system prompt hits the cache and the per-query retrieved chunks miss it. A query-agnostic memory behaves like a cached prefix — compute once per document, reuse for every question — while a query-conditioned memory is recomputed for every question in exchange for keeping the evidence that question needs. For a document asked about thousands of times, the generic memory amortizes; for a one-off question over fresh retrieval results, there is nothing to reuse anyway, so conditioning on the query can buy recall at the price of a compression pass the paper's timing does not include.

Goes deeper in: AI Agents → Context Engineering → The 4 Fixes

Related explainers

Latent Context LMs — Encoder-decoder prompt compression — a decoder pre-trained to read a 16× shorter latent sequence, compressed without the question
ASTC — Agentic soft-token compression — decides what to compress from the agent's own memory state rather than from the query
InfoKV — Entropy-aware KV-cache compression — compresses the KV cache by evicting tokens instead of encoding them into slots

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based