NemotronLabs VoiceChat listens while it speaks — Full-duplex vs half-duplex turn-taking — What does it mean?
The news. In September 2026, NemotronLabs published VoiceChat, an open full-duplex speech-to-speech model with native tool calling. It pairs a 600M-parameter streaming speech encoder with NVIDIA's Nemotron-Nano-9B-v2-Base as the language backbone, then adds a dedicated function channel for structured tool calls, an auxiliary RNN-T branch for live user transcription, and a 977M-parameter streaming TTS decoder. On Full-Duplex-Bench 1.0 the paper reports 100% takeover after a user interruption and a 4.33/5 post-interruption response-quality score judged by GPT-4o. Read the paper →
Picture the interpreter in the glass booth at a conference. The headphones never come off. They are speaking the last sentence into the microphone while the next one is already arriving in their ears, and they are scribbling names and numbers onto a notepad that nobody will ever hear. Three things are happening at once, on one clock. That simultaneity is the whole of full duplex — and it is precisely the assumption a turn-taking agent tick does not make.
VoiceChat gives the booth a literal clock. A 600M-parameter streaming encoder turns the 16 kHz microphone signal into one state every 80 ms, using 70 frames of left context and no right context at all — the same causal constraint you met in causal masking, except that the future it may not look at is audio nobody has spoken yet. Every other channel is laid out on that same 80 ms grid. The agent-text channel still generates one token at a time, but now each token occupies a frame, and on frames where the model should say nothing it emits a padding token. Silence here is a trained output, not a gap between outputs. The start and stop tokens sit at frame positions, so the beginning-of-sequence token teaches the model when to open its mouth, the end-of-sequence token teaches it when to stop, and all that padding teaches it to stay quiet while the user is still going.
The notepad matters more than it looks. An auxiliary RNN-T branch reads the same encoder states and produces a running transcript of the user — but that transcript is exposed as an output and never fed back into the language model. A classic voice stack does the opposite: transcribe first, then hand text to the LLM. Keeping the transcript off the input path preserves what the paper calls a direct speech-conditioned response path, so the model is answering the sound, not a lossy text rendering of it.
| Channel | What it emits | Why it is its own channel | Fed back into the LLM? |
|---|---|---|---|
| Agent text | Response subword tokens, plus start, stop and padding tokens on the 80 ms grid | It carries the words and the turn-taking decision in one stream | Yes |
| Function channel | <SOTC>, a <TOOLCALL> JSON payload, <EOTC>, then <EOTR> to close the tool response | A tool call needs a strict grammar that must not corrupt spoken text | Yes, at double the fusion weight |
| RNN-T transcription | An incremental transcript of the user | An auxiliary readout, so the answer stays conditioned on audio rather than on text | No |
| Streaming TTS | Acoustic frames, each worth 80 ms of waveform | It stays active across the whole conversation, silence included, and follows the upstream timeline rather than inferring turns itself | No |
A tool call is where a half-duplex stack gives itself away. The model has to stop generating, wait for a function to return, and start again — and in a spoken conversation that pause is simply dead air. VoiceChat runs the call on a dedicated autoregressive function channel sitting beside the agent-text channel, with the same structured-call discipline you set up in the tool schema. At every frame a fusion layer blends three things into the model's input: the encoded user audio, the previous agent-text token, and the previous function token, at weights of 1, 1 and 2 — the function history counts for twice either of the others. A separate head then predicts the next function token. Most of the time that head emits padding, and the paper is explicit that training it on all of that padding is what stops the model firing tools nobody asked for.
The dead air itself is handled in the runtime rather than in the weights. While a tool executes, the agent-text channel is padded — so the trained model would simply fall silent — and the runtime overwrites that padded interval with a filler message defined alongside the tool in the system prompt. This is the interpreter saying one moment, let me check that term while they flip through the glossary. The paper's guidance is blunt and practical: make the filler long enough to mask the tool's latency, skip it entirely for fast tools with short responses, and accept a trade-off in between.
| Behaviour | Desired direction | NemotronLabs VoiceChat | Source |
|---|---|---|---|
| Pause handling, synthetic | lower takeover — do not grab the floor mid-pause | 15.3%, the lowest of the open-weight systems reported | FDB 1.0 |
| Pause handling, natural pauses (CANDOR) | lower takeover | 25.5%, again the lowest reported | FDB 1.0 |
| Smooth turn-taking | higher takeover, lower latency | 81.5% at 448 ms — PersonaPlex beats it on both (90.8%, 170 ms) | FDB 1.0 |
| User interruption | higher takeover, better response | 100% takeover, 4.33/5 response quality judged by GPT-4o, 480 ms | FDB 1.0 |
| Backchannel handling | resume, do not answer the "uh-huh" | 93% resume; next best open-weight 80% | FDB 1.5 |
| Spoken-instruction intelligence | higher | 55.1 normalized average | VoiceBench |
| Tool selection | higher | 82.5% F1 — but argument accuracy and end-to-end execution are called out as weak | FDB 3.0 |
The real-time budget. Hold three things fixed: one NVIDIA H100 PCIe with 80 GB, the precision the paper actually measured (BF16 for the perception encoder and the LLM backbone, FP32 for the TTS backbone and the cached recurrent states, with no quantized variant evaluated), and four concurrent conversations sharing that one GPU. The runtime takes audio in 160 ms chunks — two encoder frames at a time. At p95 it spends 118 ms of compute on each chunk. So every chunk arrives worth 160 ms of wall-clock and costs 118 ms to process, which leaves 160 − 118 = 42 ms of slack per chunk, a real-time factor of 160 ÷ 118 = ~1.36×. That 42 ms is the entire margin the system has. It is what absorbs a scheduling hiccup, and it is why the figure that matters is p95 rather than the mean: a system that is comfortably real-time on average can still fall behind on one chunk in twenty and never recover, because the backlog a late chunk creates is carried straight into the next one.
The limits are worth reading as closely as the wins, because they mark the edge of the design rather than a bug list. The model is trained with audio context windows of roughly two minutes, so anything said earlier in a conversation may not survive. Balancing conversational skill against general ability costs it something: the paper states that its knowledge, instruction-following, reasoning and safety may be weaker than the Nemotron-Nano-9B-v2 backbone it was built from. Tool use degrades as the tool list grows, with a practical recommendation of no more than five tools per session, and calls can still be skipped, misselected, or handed invented arguments — the same pressure on tool design that applies to text agents, sharpened by the fact that there is no quiet retry the user will not hear. And the sharpest limit is an irony: user barge-in is unavailable while a tool is executing — for the length of that call the full-duplex model is half-duplex again, which is exactly the failure mode the whole architecture exists to remove.
Goes deeper in: AI Agents → The Agent Loop & State → Inside a Tick
Related explainers
- NVIDIA Audex — Unified audio-text token space — the other half of the question: how audio and text end up in one vocabulary at all.
- AsyncFC — Symbolic futures in the decode stream — the text-agent answer to a tool call that must not stall the decoder.
- AdaSR — Streaming reasoning — reasoning that starts before the input has finished arriving.