What is a shared omni encoder?

It is one Transformer that encodes images, audio and video together, instead of a separate pre-trained tower for each. In Youtu-Parsing-Omni only thin input stems and output heads are modality-specific, so almost all perception parameters are shared.

Why does a shared time axis matter for video parsing?

Vision tokens take their time coordinate from the frame index and audio tokens from their time offset, so both live on one `t` axis. Together with fusion windows that pair each frame with the sound of the same instant, the encoder aligns picture and sound itself instead of leaving the decoder to reconstruct it.

How does this differ from Gemma 4 12B's encoder-free design?

Gemma 4 12B removes the separate encoder and projects image and audio inputs straight into the language model's token space. Youtu-Parsing-Omni keeps an encoder but makes it one shared encoder for all modalities, with cross-modal alignment computed inside it.

LLM· 2026-10-14

Tencent Youtu-Parsing-Omni — Shared omni encoder with time-aligned audio-visual fusion — What does it mean?

Jargon

Omni-modal: Able to take several input types (images, audio, video, text) in one model, instead of one model per input type.
Encoder tower: A separate network that turns one modality, such as images, into vectors before a language model reads them. A two-tower design has one per modality.
Stem: The first, thin, modality-specific layer that cuts raw input into tokens: image patches for pictures, log-mel frames for sound.
Log-mel frames: Audio represented as short time slices of energy across frequency bands, on a scale close to how people hear pitch. Youtu-Parsing-Omni uses log-mel frames as its audio input.
Bidirectional Transformer: A Transformer in which every token may attend to tokens before and after it, with no causal mask. Encoders use it because they read the whole input at once.
Positional encoding: The signal that tells the model where a token sits. Text uses one number per token; this encoder uses three, (t, h, w): time, row and column. Positional Encoding step
Attention window: The group of tokens that are allowed to attend to each other in one layer. Tokens outside the window are not compared at all.
OmniSchema: The single JSON format the model emits for every task, from document layout to video segments with speech transcripts.

The news. In October 2026, Tencent released Youtu-Parsing-Omni, a 5B-parameter open-weight model that parses document pages, natural images, charts, geometry figures, audio clips and videos into one structured JSON envelope, together with a vLLM plugin and inference examples. The model card reports 96.96 overall on OmniDocBench v1.6 and 75.08 average on OmniParsingBench, the best average score among open-weight models there and second only to Gemini-3-Pro on that average. Read the release →

Picture the usual way to shoot a scene with two crews. The camera crew writes a report about the shots, the sound crew writes a report about the audio, and the two reports first meet on the editor's desk. Existing omni-modal parsers are built the same way: two separately pre-trained towers, one for pixels and one for audio, whose streams first meet inside the language model. The editor here is the decoder, and before it can write anything it has to work out which sound belongs to which shot.

Youtu-Parsing-Omni puts both crews under one clapperboard. Thin stems, the input-end gear that differs per crew, cut image patches and log-mel audio frames into tokens, the same move a text model makes when it turns token IDs into vectors. From there, one shared bidirectional Transformer reads an arbitrary packed mix of images, audio chunks and videos. Modality-specific mergers and projectors sit at the far end and hand the tokens to the decoder. Only the stems and those output heads depend on the modality, so almost all perception parameters are shared. Tencent also starts this encoder from a pre-trained text language model, which the model card says strengthens text reading, the core skill of a document parser.

The clapperboard is the positional encoding. A text model gives each token one number for its place in the sequence. This encoder gives every token three coordinates, (t, h, w). A vision token takes t from its frame index and (h, w) from its row and column in the patch grid. An audio token takes t from its time offset and sets h = w = 0. Because picture and sound share one t axis, "these happened at the same moment" is something the encoder can read straight off the coordinates, not something the decoder has to guess.

Shared coordinates alone do not make two tokens look at each other. The attention windows decide that, in the same way that a mask decides which scores count. Frames and audio chunks are packed in time order. Most encoder layers are non-fusion layers: each frame and each audio chunk gets its own window and attends only within itself. A few fusion layers start a new window at every audio-to-frame boundary, so each frame shares a window with the sound of the same instant, and cross-modal alignment is computed inside the encoder rather than reconstructed by the decoder. That is the sync check in the picture above: the crew compares each shot with the sound recorded at that second, before anything reaches the editor.

Here is the sync check with small numbers (illustrative; the model card does not give frame rates or token counts). Take a 10-second clip sampled at one frame per second, each frame cut into a 2 × 2 patch grid (4 tokens), and one audio chunk per second, also 4 tokens. That is 10 × 4 + 10 × 4 = 80 tokens. The fourth patch of frame 3 sits at (3, 1, 1); an audio token 3 seconds in sits at (3, 0, 0), so both carry t = 3. If every token attended to every other token, one layer would compute 80 × 80 = 6,400 scores. A non-fusion layer has 20 windows of 4 tokens, so 20 × 16 = 320 scores, none of them between picture and sound. A fusion layer has 10 windows of 8 tokens (one frame plus its sound chunk), so 10 × 64 = 640 scores, one tenth as many as full attention, and every picture-to-sound score is between a frame and the audio of its own second.

Design	Where picture and sound first meet	What is shared	Who works out the timing
Two towers (earlier omni-modal parsers)	Inside the language model	Only the decoder	The decoder
Encoder-free projection (Gemma 4 12B)	Inside the language model; no separate encoder at all	Image and audio inputs are projected into the language model's token space	The backbone
Shared omni encoder (Youtu-Parsing-Omni)	Inside the encoder, in the fusion layers	Almost all perception parameters, plus one `(t, h, w)` coordinate system	The encoder, before the decoder sees the tokens

The shared timecode pays off where the output itself pairs picture and sound. For a text-rich video, the model emits segments that combine on-screen text (OCR) with the speech transcript (ASR) of the same time span, plus a report of the whole video. With two towers, the pairing is reconstructed late; here the tokens that reach the decoder already carry it. One limit to keep in mind: the model card reports results for the full system, so it does not show how much of the 96.96 or 75.08 comes from the shared encoder and how much from the rest of the recipe, such as the text-model initialization and the schema-based post-training.

Goes deeper in: LLM Internals → Embeddings → Positional Encoding

Related explainers

Gemma 4 12B — encoder-free multimodal projection — the opposite move: drop the encoder entirely instead of sharing one.
Gemini Omni — one shared token space — unifying modalities on the output side so one model can also generate them.

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based