Real-time voice AI is moving beyond the traditional pipeline of automatic speech recognition, large language models, and text-to-speech systems. NVIDIA is pushing that transition further with NemotronLabs VoiceChat 11B, an open 11-billion-parameter speech-to-speech model designed for natural, real-time and full-duplex conversations.
Unlike a conventional voice assistant that first converts speech into text, sends that text to an LLM, and then converts the response back into speech, NemotronLabs VoiceChat 11B combines speech understanding and speech generation within one end-to-end architecture.
The result is a system designed to listen and speak simultaneously, allowing users to interrupt an AI agent while it is talking. NVIDIA reports a 448 ms smooth turn-taking latency on Full-Duplex-Bench 1.0, while its user-interruption take-over rate reaches 1.00 at 480 ms.
The model also introduces another important capability for open full-duplex systems: live tool calling during an ongoing conversation. Instead of forcing the user to wait in silence while an API request executes, the model can generate an operator-defined “on-hold” message while the external tool is running.
Although the model is publicly available and can be tested today, NVIDIA describes the current checkpoint as intended for research purposes only. The repository also documents several limitations that currently make it more suitable for experiments and pilot deployments than production-critical voice applications.
What Is NVIDIA NemotronLabs VoiceChat 11B?
NemotronLabs VoiceChat 11B is an open 11B end-to-end speech-to-speech model developed by NVIDIA for full-duplex conversational AI.
Full-duplex means the AI can listen while speaking rather than forcing a strict sequence in which one participant must stop completely before the other begins.
This distinction is important for natural voice interaction.
Human conversations frequently include interruptions, acknowledgements, pauses and changes in direction. A user might say “wait,” “no, that’s not what I meant,” or provide additional information while an assistant is responding.
Traditional voice pipelines can struggle with these situations because they are built around separate components:
Speech → ASR → LLM → TTS → Speech
Each stage introduces processing time and requires data to move between separate models or services.
NemotronLabs VoiceChat 11B instead attempts to combine these operations into a unified model.
The system continuously processes incoming speech while generating its own response. This design reduces the need for separate ASR, LLM and TTS orchestration and is intended to lower conversational latency.
NVIDIA reports a measured 448 ms smooth turn-taking latency using Full-Duplex-Bench 1.0.
The model also achieved a take-over rate of 1.00 at 480 ms for user interruptions, demonstrating its ability to yield when a user barges in during the agent’s response.
Why Full-Duplex Voice AI Matters
Many existing voice assistants still behave more like a digital walkie-talkie than a human conversation.
The user speaks.
The system waits.
The speech is transcribed.
The language model generates a response.
The response is converted into speech.
Only then can the user continue.
This sequential architecture can create noticeable delays and awkward interactions.
Full-duplex voice AI aims to remove that rigid interaction pattern.
An assistant can begin responding while continuing to monitor the user’s speech. If the user interrupts, the system can detect the interruption and adjust its response.
For applications such as customer service, automotive assistants, gaming characters and ordering systems, this behavior can make voice interaction feel substantially more natural.
NemotronLabs VoiceChat 11B is specifically designed around this conversational model.
448ms Turn-Taking Latency
One of the most important specifications of the NVIDIA model is its reported turn-taking performance.
On Full-Duplex-Bench 1.0, the model recorded:
- Smooth turn-taking TOR: 0.82 at 448 ms
- User-interruption TOR: 1.00 at 480 ms
- Pause-handling TOR: 0.153 on synthetic data
- Pause-handling TOR: 0.255 on Candor
For pause handling, lower values are better.
These numbers demonstrate the focus NVIDIA has placed on conversational responsiveness rather than simply measuring speech recognition or text-generation quality.
The 448 ms figure is particularly relevant because delays below roughly half a second can make an interactive voice experience feel considerably more responsive than traditional multi-stage systems.
A Hybrid Mamba and Transformer Architecture
NemotronLabs VoiceChat 11B uses a hybrid Mamba/Transformer architecture assembled from several NVIDIA technologies.
The system includes four major components.
Fast Conformer Speech Encoder
The incoming audio is processed by a Fast Conformer speech encoder derived from NVIDIA’s Nemotron-Speech-Streaming-En-0.6b model.
The encoder continuously processes a 16 kHz audio stream and converts incoming speech into representations that the language-model backbone can understand.
This streaming approach is essential for full-duplex operation because the model cannot simply wait until the user has finished speaking before beginning processing.
NVIDIA Nemotron Nano v2 Backbone
The central language component is the NVIDIA Nemotron Nano v2 LLM backbone.
It consumes audio tokens and predicts text tokens as part of the conversational process.
The language model therefore remains responsible for understanding the content of the conversation and determining what the agent should communicate.
NVIDIA TTS Decoder and Codec
The generated response is converted back into audio using an NVIDIA TTS decoder and codec.
The system produces agent speech at 22.05 kHz.
Because speech generation is integrated into the broader model architecture, the system does not require a conventional standalone TTS API to turn every completed text response into audio.
Dedicated Tool-Calling Channel
The fourth major component is a separate output channel dedicated to tool calls.
This allows the model to produce machine-readable instructions independently from its conversational audio and text output.
The result is a system capable of simultaneously producing:
- Agent audio
- Agent text
- Running user transcription
- Tool-calling instructions
This architecture is particularly important for voice agents that need to interact with external APIs.
Training Data and Model Foundations
NVIDIA says training involved approximately 550,000 hours of audio across real and synthetic datasets.
The development builds on previous research and technologies including SALM-Duplex and Audio Flamingo 3.
SALM-Duplex focuses on conversational speech systems capable of simultaneous listening and speaking, while Audio Flamingo 3 contributes to the broader development of advanced audio-language capabilities.
The combination of these technologies provides the foundation for a model designed to handle streaming speech rather than simply processing isolated voice commands.
Live Tool Calling Without Silence
One of the most interesting capabilities of NemotronLabs VoiceChat 11B is its support for tool calling during an active conversation.
Traditional voice agents generally need to pause while an external service executes.
For example, imagine a customer asks:
“Can you check whether my order has shipped?”
The voice assistant needs to call an order-management API.
In a conventional system, the agent might stop talking while waiting for the API response.
NemotronLabs VoiceChat 11B introduces a mechanism intended to avoid this awkward silence.
Tool calls are emitted through a separate channel using a <TOOLCALL> block.
After the external system completes the operation, the result is returned using a <TOOL_RESPONSE> block.
The system can also use an on-hold message.
This message is defined by the operator for each tool and can be spoken immediately when the model generates the text that triggers the API request.
Instead of silence, the user might hear a short acknowledgement while the external operation is completed.
This approach could be useful for voice assistants performing tasks such as:
- Checking order status
- Retrieving prices
- Looking up weather information
- Querying internal databases
- Calling enterprise APIs
- Retrieving account information
Tool-Calling Restrictions
The tool-calling implementation is powerful, but NVIDIA also specifies several limitations.
The company recommends using a maximum of five tools per session.
The model cannot reliably execute multiple tools simultaneously, which limits the complexity of workflows it can safely manage.
There is also an important interaction restriction: the user cannot interrupt the agent during tool execution.
System prompts and tool responses must also be ASCII-only and TTS-friendly.
These limitations mean developers should carefully design tool interfaces before attempting to use the model in complex voice workflows.
Tool-Calling Performance
NemotronLabs VoiceChat 11B was also evaluated for spoken tool calling.
On the AU Harness BFCL-v3 benchmark, NVIDIA reports:
- Simple tool calling: 58.5%
- Multiple tool calling: 62.5%
- Parallel tool calling: 42.5%
- Parallel-multiple: 27.5%
- Irrelevance detection: 89.6%
- Overall average: 56.1%
On Full-Duplex-Bench v3, the reported results were:
- Tool selection: 82.5%
- Argument accuracy: 44.2%
- Pass@1: 33%
These results show that while tool selection is relatively strong, complex parallel tool workflows remain a challenge.
For developers, this means the model may be better suited to controlled voice-agent workflows than highly autonomous multi-tool orchestration at its current stage.
VoiceBench and Full-Duplex Rankings
NVIDIA reports that NemotronLabs VoiceChat 11B ranks second among open full-duplex models on VoiceBench.
It also ranks second among open models on Full-Duplex-Bench 1.0.
These results are notable because the full-duplex voice model category is still developing.
Rather than optimizing only for text quality, the model is being evaluated on characteristics such as interruption handling, turn-taking and conversational timing.
For real-time voice applications, these metrics can be just as important as conventional language-model benchmarks.
Is NemotronLabs VoiceChat 11B Ready for Production?
The short answer is partially.
The model can be deployed today for research, demonstrations and pilot projects, but it should not yet be treated as a production-ready drop-in replacement for established voice-agent platforms.
NVIDIA provides both the model weights and a container, and the license is permissive.
However, NVIDIA explicitly describes the checkpoint as being ready for research purposes only.
The repository also identifies several real-world failure modes.
Two-Minute Audio Context Limit
The system currently has a two-minute audio context ceiling.
Longer conversations therefore require careful session management and potentially additional application-level handling.
Non-Recoverable Gibberish
After several turns, the model can degrade into non-recoverable gibberish.
This is a major consideration for customer-facing deployments because a voice agent must remain stable throughout extended conversations.
Runaway Self-Talk
Another documented problem is runaway self-talk after a turn ends.
This can cause the model to continue producing speech when it should have stopped.
For a production voice assistant, reliable end-of-turn detection is critical, so this behavior would need to be mitigated.
Dropped Words
The model can also experience dropped words in user transcription.
Although the system performs speech understanding directly, developers should not assume that its running transcription will always be equivalent to a high-quality dedicated ASR system.
Hardware Requirements
NemotronLabs VoiceChat 11B is designed to run on systems with at least 80 GB of GPU VRAM.
Supported hardware includes:
- NVIDIA A100
- NVIDIA H100
- NVIDIA RTX 6000 Pro
- NVIDIA B200
The deployment environment is based on x86_64 Linux.
This hardware requirement makes the model accessible to organizations with substantial GPU resources, including AI startups, enterprise R&D groups, university research teams and GPU cloud providers.
However, it also creates a significant barrier for developers who want to experiment without dedicated hardware.
At present, there is no hosted API or inference provider serving the model, meaning users without access to an appropriate GPU cannot simply call the model through an online API.
Potential Industry Applications
Despite its current research status, the model has several potential applications.
Contact Centers
Customer-support platforms could use full-duplex voice agents to make conversations more natural.
Agents could listen while speaking and call internal systems to retrieve customer information.
Automotive Assistants
In-car voice assistants could benefit from lower-latency interaction, particularly when drivers interrupt or change their requests.
Retail and Drive-Thru Ordering
Voice ordering systems could use tool calls to retrieve product availability, prices and order information while maintaining a conversational experience.
Telecom and IVR Systems
The technology could also contribute to modernizing traditional interactive voice response systems.
Instead of navigating rigid menus, users could interact naturally with an AI agent.
Games and NPCs
Full-duplex speech could make non-player characters more dynamic by allowing players to interrupt characters naturally.
Accessibility
Responsive voice interfaces may also benefit users who rely heavily on speech-based computer interaction.
How It Differs From Traditional Voice AI
The biggest architectural difference can be summarized simply.
A conventional system looks like:
User Speech → ASR → LLM → TTS → Agent Speech
NemotronLabs VoiceChat 11B instead moves toward:
User Speech → Unified Speech-to-Speech Model → Agent Speech
The model also maintains additional output channels for text, transcription and tool execution.
This reduces the number of independent components that need to be orchestrated.
However, a unified model also means that a failure within the model can affect multiple functions simultaneously. This is one reason why NVIDIA’s research-only designation and documented failure modes are important for developers evaluating the technology.
Open Model and Developer Opportunity
The release is significant because developers can access the model weights rather than being limited to a proprietary hosted voice API.
The model uses the OpenMDW-1.1 permissive license, giving organizations considerable flexibility for experimentation.
For AI researchers, this provides an opportunity to investigate full-duplex speech-to-speech systems without relying entirely on closed commercial platforms.
For startups, the model can serve as a foundation for pilot applications where control over model deployment and infrastructure is important.
The requirement for an 80 GB GPU does, however, mean that experimentation is considerably more demanding than using lightweight speech models or cloud APIs.
The Future of Full-Duplex AI Agents
The development of NemotronLabs VoiceChat 11B reflects a broader shift in conversational AI.
Voice assistants are increasingly expected to behave less like a sequence of independent software components and more like continuous conversational systems.
Users do not naturally speak in perfectly separated turns.
They interrupt.
They pause.
They change their minds.
They ask follow-up questions before an answer is complete.
They expect assistants to understand context while simultaneously responding.
Full-duplex models are designed around these realities.
The addition of tool calling makes the technology even more interesting because future voice agents will need to do more than talk. They will need to interact with enterprise software, databases, search systems and APIs while maintaining natural conversations.
NemotronLabs VoiceChat 11B demonstrates how those capabilities can be combined into a single open model.
Key Takeaways
NVIDIA’s NemotronLabs VoiceChat 11B represents a significant development in open real-time speech-to-speech AI.
The most important points are:
11-billion-parameter speech-to-speech model: The system combines speech understanding and speech generation rather than relying on separate ASR, LLM and TTS models.
448 ms turn-taking latency: NVIDIA reports smooth turn-taking latency of 448 ms on Full-Duplex-Bench 1.0.
Strong interruption handling: The model achieves a user-interruption take-over rate of 1.00 at 480 ms.
Live tool calling: It is presented as the first open full-duplex model with tool calling, using a dedicated output channel and <TOOLCALL> instructions.
On-hold voice responses: Operators can define messages that the agent speaks while an external API is executing, reducing dead air.
550,000 hours of audio training: Training used approximately 550k hours of real and synthetic audio data.
80 GB GPU requirement: Deployment requires hardware such as an A100, H100, RTX 6000 Pro or B200 with at least 80 GB of VRAM.
Research-stage limitations: NVIDIA documents issues including a two-minute context limit, gibberish after several turns, runaway self-talk and dropped transcription words.
56.1% average tool-calling score: The model achieved a 56.1% average on the reported AU Harness BFCL-v3 spoken tool-calling evaluation.
Open and accessible: The model weights and container are publicly available under the OpenMDW-1.1 permissive license.
Conclusion
NVIDIA NemotronLabs VoiceChat 11B demonstrates how the next generation of AI voice assistants could move beyond traditional ASR-to-LLM-to-TTS pipelines.
Its unified speech-to-speech architecture, full-duplex interaction, 448 ms reported turn-taking latency, interruption handling and live tool calling make it an interesting platform for developers researching real-time conversational agents.
The tool-calling capability is particularly promising. By combining <TOOLCALL> and <TOOL_RESPONSE> channels with configurable on-hold messages, the system provides a practical mechanism for connecting voice conversations to external services without forcing users to experience complete silence during API execution.
At the same time, developers should not overlook the model’s current limitations. NVIDIA describes the release as research-oriented, and documented problems such as the two-minute context ceiling, runaway self-talk, dropped words and non-recoverable degradation after multiple turns make careful testing essential before considering production deployment.
For researchers and organizations with access to an 80 GB GPU, however, NemotronLabs VoiceChat 11B provides an unusually accessible opportunity to experiment with open full-duplex speech-to-speech AI.
If these reliability issues improve in future versions, architectures of this type could become an important foundation for customer-service agents, automotive assistants, interactive gaming characters, accessibility tools and enterprise voice systems that need to listen, speak and act at the same time.
Discover more from AiTechtonic - AI & Informative News
Subscribe to get the latest posts sent to your email.