What is formatting-aware document ingestion?

It is a converter step that reads the formatting stored in a Word file, such as colour and highlight, and writes it into the text as tags like `[COLOR:#FF0000]` or `[HIGHLIGHT:YELLOW]`. The agent then sees which lines the author marked, not only the words.

Which formatting cues matter most to an agent?

In the paper's leave-one-out test, highlight and colour carried the loss: removing them cost 0.302 and 0.262 of macro F1. Bold, italic, monospace and strikethrough each changed the score by at most 0.003. The authors attribute this to those cues being too common to signal intent.

How does this relate to chunking and retrieval in RAG?

Chunking and retrieval run after conversion, so they can only use what the converter kept. If colour or highlight is dropped at ingestion, no later step can restore it, which is why the failure is silent and the answers still read fluently.

Agent· 2026-10-24

Document color cues lost before agents read them — Formatting-aware ingestion vs plain-text extraction — What does it mean?

Q: Does adding a typed role to a cue help beyond keeping the cue?

A little. The paper's pre-registered test found +0.085 macro F1 over the same markup without role tokens (Holm-corrected p = 0.049), which the authors call modest. Most of the gain comes from simply preserving the cue.

Jargon

OOXML: Office Open XML, the format inside a .docx file. Text is stored as runs, and each run carries its own properties such as font colour, highlight, bold and strikethrough.
Document ingestion: The converter step that turns a Word file or PDF into the tokens an agent reads. It runs before any reasoning, so whatever it drops, the agent never sees.
Docling: An open-source converter that turns documents into LLM-ready Markdown. It recovers text, tables and structure well, but in this paper it does not carry colour or highlight.
Micro F1: A score that mixes precision (how many listed requirements were right) and recall (how many marked requirements were found), counted over every requirement in the corpus at once.
False-positive rate: How often the agent lists a requirement the author did not mark. Here it shows an agent guessing instead of reading a cue.
Hybrid serialization: Writing a document out as text that mixes Markdown for what models know well, such as **bold**, with custom tags such as [COLOR:#FF0000] for what Markdown cannot express.
Leave-one-out ablation: Removing exactly one kind of cue at a time and measuring how much the score falls. Here, removing highlight and colour cut macro F1 by 0.302 and 0.262.

The news. On October 21, 2026, Amazon researchers posted What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion. They label requirements straight from the formatting stored inside 76 real policy documents, and report that formatting-blind conversion cuts the F1 score for requirements an author marked in red or highlight, a gap that holds across six answering models. Read the paper →

Picture a contract where the lawyer highlighted the clauses that must be met by Friday and wrote one in red. Now photocopy it in black and white. Every word survives the copy, but the ink does not. Hand the copy to an assistant and ask which clauses the lawyer flagged. The assistant cannot know, so it guesses, and a confident guess reads just like a correct answer.

That is what a document converter can do to an agent. A Word file stores each stretch of text as a run with its own colour, highlight and weight. A converter such as Docling or a plain-text export reads the words and drops those properties. In the paper's test, the pipeline reports success, the text looks complete, and the agent answers fluently and wrongly. Nothing fires an alarm, because nothing visibly went missing. The chunking and retrieval steps that come after conversion can only work with what it kept.

Read the .docxrun properties: colour, highlight, bold

Map cues to rolescode only, no LLM calls

Serialize as hybrid textMarkdown plus COLOR and HIGHLIGHT tags

Agent reads the tagscue and meaning both in context

The authors built a task from 76 real policy documents: list every requirement the author singled out. The answer key comes from the file's own run properties, not from anyone's judgement, and a deterministic matcher scores it with no LLM judge. Six models answered, five Claude models and one OpenAI model. The only thing that changed between arms was how the document was converted.

Representation of the same documents	Micro F1 (marked requirements)	Source
Formatting-aware, with typed roles	0.690	Table 1
Same markup, roles removed	0.605	Table 1
docx2python (keeps formatting as HTML)	0.586	Table 1
Docling	0.150	Table 1
Plain text	0.075	Table 1

Dropping the cues does not make the agent say less; it makes the agent guess more. Docling's false-positive rate on unmarked requirements was 0.043, roughly four times the formatting-aware arm's 0.011. A reader who cannot see the cue still has to answer, so it lists plausible lines. This is the same kind of silent failure that error analysis is meant to find: an eval that only checks whether the words are present would pass every one of these runs.

Only two cues carried the loss. A leave-one-out test removed one cue at a time. Removing highlight cost 0.302 of macro F1 and removing colour cost 0.262, while bold, italic, monospace and strikethrough each moved the score by at most 0.003. The authors' reading is that bold is too common to mean anything, since an author who bolds every heading has not singled anything out, while colouring or highlighting a requirement is a deliberate, sparing act. The loss also tracks the cue: removing colour hurt colour-marked documents and left highlight-marked ones at 0.694 micro F1.

Preserving the cue is most of the gain. Attaching a typed role to it adds a smaller amount: a pre-registered test found +0.085 macro F1 over the same markup without role tokens (Holm-corrected p = 0.049), which the authors call modest. Roles are hints, not truth. Red text or red highlight maps to critical_warning, other visible highlights map to warning, and the authors left out medium-confidence mappings such as green text, because a wrong hint biases an agent worse than none.

Now put the rates side by side. The paper reports that 32.8% of the real documents it scanned carry a deliberate red text or non-white highlight, and 10.3% carry one on a sentence stating an obligation (its abstract says 6,460 documents, its methods say 6,462). Take 1,000 policy documents at those rates: about 328 hide a cue, and about 103 hide it on an obligation. Inside one of those 103, suppose there are 10 requirements and the author marked 3 (illustrative). A reader looking at 10 identical lines that picks 3 by guessing gets 3 × 3/10 = 0.9 right, or 30% (illustrative). A reader who sees the three tags lists exactly those three. In the paper's measured numbers the same gap is 0.690 against 0.075 micro F1, about nine times higher.

Two limits matter. The task asks only for requirements the author marked, and the corpus is one public set of Word policy files, so the numbers do not say how much every kind of document loses. The authors also report that progressive disclosure did not reduce token cost, and that the typed layer showed no benefit on the two models that attempt the task only sparsely. And wording alone cannot rescue a formatting-blind reader: a supervised text-only classifier reached only 0.664 AUC, which the paper treats as a ceiling for any method that cannot see the formatting.

Goes deeper in: AI Agents → Retrieval & RAG → RAG Failure Modes

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based