What is a unified audio-text token space?

It is a design where a model represents both audio and text in one shared embedding space, so a single Transformer decoder handles them the same way. Incoming sound is encoded and projected into the text embedding space; outgoing sound is generated as quantized discrete tokens, one at a time, exactly like text. There is no separate recognizer or synthesizer model — the one decoder both understands and produces audio.

Why does keeping audio in the text token space matter?

Because it lets an audio model reuse everything a text LLM already has. Training and serving run on the standard LLM stack instead of a bespoke speech pipeline, and — critically — the model can add hearing and speaking while keeping its text reasoning, knowledge, and long-context skills, which NVIDIA reports survive with marginal or no regression. Audex is a 30B-A3B Mixture of Experts, so it carries that knowledge at roughly 3B active parameters of compute per token.

How is this different from a cascade of ASR and TTS models?

A cascade wires three separate systems together — a speech recognizer transcribes to text, the language model reads the text, and a text-to-speech model speaks the answer. The language model never hears the audio itself, latency stacks across three models, and tone and timing are lost in the text bottleneck. Audex collapses all three into one decoder that reads and writes audio tokens directly, so nothing is thrown away at a text interface.

NVIDIA Audex — Unified audio-text token space

TL;DR

What is it: NVIDIA's Nemotron-Labs released Audex, an audio model whose whole design is a unified audio-text token space — one Transformer decoder that puts sound in the same space as text, so it both understands and speaks audio instead of leaning on a separate recognizer and synthesizer.
Why it’s needed: It is a bet on reuse: if audio lives in the same token space as text, an audio model can run on the exact same training and serving machinery as a text LLM — and keep the reasoning brain it started with.
vs previous: Versus the usual recipe — a cascade of separate speech models (a recognizer, then the LLM, then a text-to-speech model) or a speech adapter bolted onto a text model — Audex has one decoder, and tuning it for audio causes marginal or no regression on the text skills.

Jargon

Quantized audio tokens: The discrete units Audex generates for sound. Where a continuous wave has no natural "pieces," quantizing it turns the signal into a fixed vocabulary of discrete units — the way a tokenizer chops text into pieces — so the decoder can emit sound one token at a time, exactly like words. (Audio input, by contrast, is encoded and projected into that same space.)
Embedding space: A transformer doesn't read letters or sound; it reads dense vectors. The embedding space is the shared vector format every input must arrive in — putting audio there is what lets one backbone treat it like text.
MoE / A3B: Audex is a Mixture of Experts model written 30B-A3B: 30 billion total parameters, but only about 3 billion active per token. It carries a big model's knowledge at a small model's per-token compute.
ASR / TTS: ASR (automatic speech recognition) turns speech into text; TTS (text-to-speech) turns text into speech. The old pipeline needs a separate model for each; Audex does both inside one decoder.
Text regression: When you fine-tune a model for a new skill, its old skills often degrade — the classic catastrophic-forgetting tax. Audex reports marginal or no such drop: the reasoning and knowledge from the text backbone mostly survive the audio training.
On-policy distillation: A late training stage where a student model is corrected on its own generated outputs by a teacher. Audex uses it multi-domain; see the on-policy distillation explainer for the mechanism.

The news. On July 6, 2026, NVIDIA's Nemotron-Labs published Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text model built on the text-only Nemotron-Cascade-2-30B-A3B backbone. Its design is deliberately plain: a single Transformer decoder where audio inputs are projected into the text embedding space and audio outputs are emitted as quantized tokens, handled exactly like text tokens. It reports state-of-the-art audio understanding, speech recognition, translation, text-to-speech, and speech-to-speech, "while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression." Read the paper →

Picture one long conveyor belt. Every word of a text prompt rides it as a small tile — that is already how a language model works, since a transformer only ever reads tokens. A sound clip doesn't naturally fit: it is a smooth, continuous wave, not a row of tiles. So an intake step reshapes the incoming wave and drops it onto the same belt, in the same space as the letter tiles — no separate side-track for speech. Now there is one belt, and a single reader working down it, treating every tile the same way. And when the reader wants to answer out loud, it lays down sound tiles the same way it lays down letters — discrete pieces of audio drawn from a fixed set — which play back as speech.

Underneath the metaphor, "the same belt" is the model's embedding space — the dense vectors a transformer actually consumes. A text token becomes a vector by a lookup; an audio segment becomes one by an encode-and-project step that lands it in that exact same space. Generation then runs one token at a time with no branch for modality: at each step the decoder emits the next token, and that token can be a word piece or a quantized audio token — a slice of sound drawn from a fixed vocabulary, the acoustic analog of a sub-word piece. Because the whole thing is just tokens in and tokens out, Audex runs on the standard LLM training and serving stack — no bespoke speech pipeline bolted on the side.

Approach	How audio moves through	Separate speech model?	Text skill
Cascade (ASR → LLM → TTS)	a recognizer transcribes speech to text; the LLM reads text; a TTS model speaks the reply	yes — three models chained	untouched, but latency stacks and the LLM never hears tone or timing
Bolt-on speech adapter	an audio encoder + adapter feed a frozen or fine-tuned text model	partly — an extra encoder network	at risk — tuning for audio can drag text skills down
Unified token space (Audex)	audio projected into the text embedding space in, emitted as discrete tokens out; one decoder reads and writes them	no — one decoder does both	"marginal or no regression" (NVIDIA, reported)

Consider the training diet, because the token counts are unusually concrete. Audex is fed 157.4 billion audio tokens and 320.5 billion text tokens — roughly 478 billion tokens in all. Do the split: audio is 157.4 / 478 ≈ one in three tokens, and text keeps the other two in three. Keeping text the majority of the diet is plausibly part of how a whole new modality gets added without the usual forgetting tax — though the paper credits the full recipe (the token mix plus multi-stage supervised training, text-only reinforcement learning, and on-policy distillation), not the ratio alone. The exact contribution of each ingredient is not broken out.

The efficiency comes from the backbone being a Mixture of Experts. The name 30B-A3B says it: 30 billion parameters of stored knowledge, but only about 3 billion active for any given token. So the model reasons with the breadth of a 30B model while each generated token — word or sound — runs only its ~3 billion active parameters, not all 30 billion, keeping the per-token compute low. NVIDIA reports state-of-the-art results across audio understanding, recognition, translation, and generation; the exact codec, vocabulary size, latency, and layer-level specifics beyond the single-decoder description are not fully detailed in the paper.

The payoff is a cleaner idea of what an "audio model" has to be. You don't need a separate ear, brain, and mouth wired together — if sound can live as tokens in the text model's space, one decoder can listen, think, and speak. Audex is a bet that the simplest unification, one belt for every kind of tile, beats a specialized pipeline — and that you can add a new sense to a language model without eroding the language.

Goes deeper in: LLM Internals → Embeddings → Tokens in Space

Related explainers

Gemini Omni — modality unification in a shared token space — the same "one token space for every modality" idea, taken to full any-to-any generation across text, image, and audio
Gemma 4 12B — encoder-free multimodal projection — a different lever on the same problem: dropping the encoder to get inputs into the token space, versus Audex's focus on audio output and keeping the text skills intact
GLM-5V — native multimodal vs vision-bolted — the training-side axis: building a model multimodal from the start rather than adapting a finished text model

Continue in trackLLM Internals — Embeddings: how every input becomes a vector

Frequently Asked Questions

Check what you knowMap your AI & GPU knowledge across every track — free, role-based