NVIDIA has released the open weights for VoiceChat-11B, an end-to-end speech-to-speech model that holds a live, full-duplex conversation and calls tools mid-sentence. Announced August 3, 2026, it is the newest open drop in NVIDIA's Nemotron family, and the company describes it as the first open full-duplex model to support tool calling. Instead of chaining separate speech-to-text, language, and text-to-speech stages, VoiceChat-11B runs all three inside one 11-billion-parameter network and answers in roughly 450 milliseconds. For anyone building voice agents, that pairing of open weights plus mid-conversation tool calls is the piece the open ecosystem has been missing.
Background
Most voice assistants shipping today are pipelines. Audio goes into a speech-to-text model, the transcript goes into a language model, and the reply goes into a text-to-speech model. Every handoff adds delay, and the whole chain is half-duplex: it listens, then it talks, then it listens again. Interrupting it feels wrong because the system was never hearing you in the first place. Full-duplex models collapse that structure into a single network that listens and speaks on separate channels at the same time, which is what makes barge-in and natural turn-taking possible.
The open community has had one strong full-duplex option in Kyutai's Moshi, released in 2024, but Moshi cannot call tools, so it can hold a conversation yet not actually do anything on your behalf. Closed real-time APIs like OpenAI's GPT-Live and Google's Gemini Live pair full-duplex speech with function calling, but you rent them, you cannot self-host them, and you send every second of audio to someone else's servers. VoiceChat-11B is the first open drop to close that gap: full-duplex, tool calling, and weights you can download and run.
Deep Analysis
VoiceChat-11B is worth a close read because of how NVIDIA assembled it and where it lands on the benchmarks. This is not a from-scratch voice model; it is a language model the company already trusts, wrapped in streaming speech components and taught to emit tool calls without stopping the conversation.
One network instead of three
The core is NVIDIA's Nemotron Nano v2 9B language backbone, a hybrid Mamba and Transformer model. NVIDIA puts a Fast Conformer speech encoder in front of it to turn incoming audio into tokens and a streaming TTS decoder behind it to speak. The result is one 11-billion-parameter model that inherits real reasoning from the Nano backbone rather than the shallow scripted replies of a bolt-on voice layer. It takes 16 kHz audio in and produces 22.05 kHz audio out, processing in short chunks faster than real time so the reply starts before you have finished forming the next thought. NVIDIA reports training on roughly 550,000 hours of audio.
Tool calling without breaking the conversation
The differentiator is that VoiceChat-11B can trigger a function call in the middle of speaking and keep the conversation flowing. It emits tool calls on a separate output channel using a simple tagged format, so the model can say "let me check that for you" out loud while the tool script fires underneath. That is the capability no other open voice model ships, and it is what turns a talking demo into an agent that can look up weather, query a database, or book a slot. If you have followed NVIDIA's broader Nemotron line, this is the voice-native member of that family arriving with the same open-weights posture.
Benchmarks: where it lands
On the Full-Duplex-Bench suite the model posts a smooth turn-taking latency of 448 milliseconds and handles user interruptions cleanly, and on VoiceBench it ranks second among all open full-duplex models. Tool calling is measured on the Berkeley Function Calling Leaderboard: on BFCL-v3 it averages 56.1 percent accuracy (58.5 percent on simple calls, 62.5 percent on multiple), and on the Full-Duplex-Bench tool-calling track it selects the right tool 82.5 percent of the time while getting the arguments fully right 44.2 percent of the time. Read that split honestly: the model is good at knowing which tool to reach for and still rough at filling in every argument perfectly, so production use wants validation and retries around each call.
Here is how the open release compares to the peers builders actually weigh it against.
| Model | Open weights | Full-duplex | Tool calling | Turn latency | License |
|---|---|---|---|---|---|
| NVIDIA VoiceChat-11B | Yes | Yes | Yes | ~450 ms | OpenMDW 1.1 (research) |
| Kyutai Moshi | Yes | Yes | No | Sub-second | Permissive |
| OpenAI GPT-Live | No | Yes | Yes | Sub-second | Proprietary API |
| Google Gemini Live | No | Yes | Yes | Sub-second | Proprietary API |
Open weights, research license
The catch is the license. VoiceChat-11B ships under NVIDIA's OpenMDW License version 1.1 as a research release, and it is English-only for now. Those two constraints decide whether it fits your project before any benchmark does: you can download the weights, study them, and prototype freely, but read the license terms before you point it at paying users, and do not plan on non-English support yet. It runs on A100, H100, H200, B100, B200, and RTX 6000 cards through a Python 3.12 and PyTorch 2.10 environment, so a single modern data-center or workstation GPU is enough to stand it up.
Impact on Creators
For builders, the practical unlock is owning the whole voice stack instead of renting it. A self-hosted full-duplex agent means your audio never leaves your infrastructure, your per-minute cost is GPU time rather than metered API calls, and you can tune the behavior directly. The path from download to working endpoint is short. Clone the NeMo Speech branch that carries VoiceChat, serve the weights through vLLM on one supported GPU, and connect over the WebSocket streaming interface NVIDIA ships. Start by wiring up a single tool, a weather lookup or a calendar query, because the tool-calling loop is the part worth validating first and the capability that sets this model apart. From there you add your own functions, put argument validation around each call to cover that 44 percent argument-accuracy gap, and test whether the self-hosted agent can replace a closed real-time API in your prototype.
The honest read is that this is a research release, not a drop-in production product. English-only rules out a lot of consumer use, the license needs a real look before you ship, and the argument accuracy means you cannot trust every tool call blindly. But for a private voice assistant, an internal agent, or any prototype where data control and cost matter more than polish, it is the cheapest path to an interruptible, tool-using voice agent you fully own. Compared to the full-duplex work from other labs and the real-time voice modes in closed assistants, the open weights are the whole point.
Key Takeaways
- VoiceChat-11B is the first open full-duplex voice model that can call tools mid-conversation, released with open weights on August 3, 2026.
- It unifies speech-to-text, reasoning, and text-to-speech in one 11B network built on the Nemotron Nano v2 9B backbone, answering in about 450 milliseconds.
- Benchmarks are strong on turn-taking and tool selection (82.5 percent) but rougher on tool-argument accuracy (44.2 percent), so wrap every call in validation.
- The OpenMDW 1.1 research license and English-only support are the real gating factors, not the model quality.
- Self-hosting through NeMo Speech and vLLM on one supported GPU keeps audio private and turns per-minute API cost into GPU time.
What to Watch
The immediate question is whether NVIDIA extends VoiceChat beyond English and relaxes the research license toward production use, because those two limits, not the architecture, are what keep it out of shipping products right now. The second is whether the open ecosystem closes the argument-accuracy gap, either through fine-tunes on top of these weights or through community tooling that validates and retries calls automatically. If both happen, the closed real-time APIs lose their clearest advantage, which was never the full-duplex speech but the reliable function calling wrapped around it. VoiceChat-11B is the first open model to make that a live contest, and the next few releases in this space will show how fast the gap closes.
Frequently Asked Questions
What makes VoiceChat-11B different from other open voice models?
It is the first open full-duplex voice model that can call tools in the middle of a conversation. Earlier open full-duplex models like Moshi can talk naturally but cannot trigger functions, which limits them to conversation rather than action.
Can I use VoiceChat-11B in a commercial product?
It ships under NVIDIA's OpenMDW License version 1.1 as a research release, so review the license terms carefully before deploying to paying users. It is also English-only for now, which rules out many consumer use cases.
What hardware do I need to run it?
The model runs on NVIDIA A100, H100, H200, B100, B200, and RTX 6000 GPUs through a Python 3.12 and PyTorch 2.10 environment, served with vLLM. A single modern data-center or workstation GPU is enough to stand up an endpoint.
How fast is it?
NVIDIA reports a smooth turn-taking latency of about 448 milliseconds and clean handling of user interruptions, processing audio in short chunks faster than real time so replies begin almost immediately.
How reliable is the tool calling?
On BFCL-v3 it averages 56.1 percent accuracy and selects the correct tool 82.5 percent of the time, but fully correct arguments land 44.2 percent of the time. Add argument validation and retries around each call for production use.
