LatComm reuses compressed context across LLMs — Encode-once latent packets — What does it mean?

The news. On October 19, 2026, researchers from Rochester Institute of Technology and NVIDIA published LatComm. It trains only a small interface between two frozen LLMs, and reports up to 10.2 points higher average accuracy than single-vector baselines on 12 multiple-choice datasets, and 3× faster question answering on QuALITY. Read the paper →

Picture the analyst and the consultant. A long case file is useful for many questions, but handing the raw file to the consultant every time means the consultant re-reads all of it every time. In text passing that is exactly what happens: on QuALITY each article has 18.1 questions on average, and the receiver reads about 5,915 tokens for each one, of which 98% is the article. The same re-reading is what prefix caching removes inside one model, and what a KV cache removes if the same model answers again.

LatComm gives the work to the analyst. A frozen encoder LLM reads the context once, never seeing the question, and a small compressor turns its hidden states into a fixed packet of learned slots. The compressor is cross-attention in the other direction: a set of learned slot queries reads the context's token states and collects a summary into each slot. The default packet uses 32 slots of width 256 at each connected layer pair, 384 KiB per context. Because the packet is written without the question, the same file always gives the same sheet, so it can be stored and reused.

The consultant needs a way to use the sheet. The receiver runs as usual on the question, and after selected layers a reader attends over the packet slots, with a gate on every slot that the question itself sets. A plain cross-attention weight cannot switch the whole packet off, because the weights sum to one across slots. The gate scales each slot's values, so a question about a date can turn up the lines about dates and dim the rest. Because only compressors, readers and gates are trained, the encoder and receiver can come from different families, and the paper tests Qwen, Gemma and Llama receivers. For the memory side of this trade, see KV cache memory cost: the packet replaces a cache that grows with the context by one that does not.

Worked example, with the paper's QuALITY numbers. Hold three things fixed: 18.1 questions per article, a 4B encoder with a 0.8B receiver on one H100 at batch size 1, and the measured times. Text passing costs 511 ms per question, so 18.1 questions take 18.1 × 511 ms ≈ 9.26 s. The packet path pays one 0.71 s encode, then 131 ms per question, because the receiver now reads 103 tokens instead of 5,915: 0.71 + 18.1 × 0.131 ≈ 3.09 s, which is 3.0× faster. A question-only pass with no context costs 111 ms, so the readers and gates add only about 20 ms. Break-even is two questions: one question costs 0.511 s as text against 0.71 + 0.131 = 0.84 s with a packet, but two cost 1.02 s against 0.97 s. The packet also keeps its 131 ms as articles get longer, while text-passing time grows from 300 ms to 651 ms across the shortest to longest fifth of articles.

Size and accuracy are tunable, with limits. An 8-slot packet of 96 KiB kept the average accuracy of the 384 KiB default (65.0 to 65.8 across 8 to 64 slots), and every packet size beat the best single-vector baseline (61.98). The paper tests only English multiple-choice questions with receivers of about one billion parameters, and does not test free-form generation, multi-agent use or agent memory, so those uses are a stated direction, not a result. If you build agents, the idea matters most for a shared observation that many later calls read, as discussed in Subagents as Context Isolation.

Goes deeper in: AI Agents → Context Engineering → Subagents as Context Isolation

Related explainers

• Latent Context LMs compress prompts 16x — the single-model version, where the decoder is trained to read latents in place of tokens.