Voice AI is entering a phase where milliseconds increasingly determine whether an agent feels intelligent or frustrating. A response that begins almost instantly can feel natural, while the same answer delayed by another second can make users think the system has stopped listening.
That is why Time to First Token (TTFT) has become one of the most important performance metrics for teams building voice assistants, realtime AI agents, customer-service bots and conversational applications.
But there is a problem with treating TTFT as the ultimate benchmark.
TTFT measures how quickly an inference API begins generating output. In a traditional text chatbot, that is often a useful approximation of responsiveness. In a voice agent, however, the first token is not necessarily something the user can hear.
A text-to-speech system generally needs enough text to construct a meaningful audio segment before playback begins. As a result, the time between the first generated token and the first spoken sentence can be just as important as TTFT itself.
This distinction has led developers to look beyond a single metric and examine the entire voice pipeline — from speech-to-text (STT) and endpoint detection to LLM inference, text-to-speech (TTS), network transport and speech-to-speech models.
This benchmark examines those layers and looks at which inference APIs and realtime models currently offer the strongest latency characteristics for conversational AI.
Why TTFT Matters for Voice AI — But Cannot Be Used Alone
Time to First Token describes the amount of time between an inference request reaching an AI system and the arrival of the first generated token.
IBM describes TTFT as the point at which an inference system transitions from waiting to producing visible output. For conventional chat applications, that measurement is extremely useful because users can see the answer immediately as tokens stream onto the screen.
Voice applications work differently.
An AI model might produce its first token after 300 milliseconds, but a downstream TTS system cannot necessarily turn that single token into useful speech. The application may have to wait for a complete phrase, clause or sentence.
That creates another measurement: Time to First Sentence (TTFS).
For conversational voice agents, TTFS can be a better representation of perceived responsiveness because it measures the point at which enough generated text exists for the system to begin producing meaningful audio.
This creates two separate performance variables:
- TTFT: How quickly the LLM begins generating.
- Output speed: How quickly the model produces enough text for speech.
- TTFS: How quickly a complete spoken response can begin.
A model with excellent TTFT but poor output throughput may therefore perform worse in an actual voice conversation than a model with slightly higher TTFT and significantly faster generation.
The Voice AI Latency Budget
A voice agent is not powered by an LLM alone. Every conversational turn passes through several components, and each one contributes latency.
LiveKit’s voice-agent guidance places typical latency roughly in these ranges:
- Speech-to-text: 100–200 milliseconds
- Streaming LLM inference: 300–500 milliseconds
- Text-to-speech: 100–200 milliseconds
- Network transport: 50–150 milliseconds
Together, that produces a practical end-to-end target of approximately 700 milliseconds to 1.2 seconds.
Kwindla Hultman Kramer, co-creator of Pipecat, has similarly argued for an approximately 800-millisecond median voice-to-voice latency target for high-quality conversational experiences. A proof-of-concept system can tolerate considerably more, with around 1.5 seconds being a more forgiving threshold.
The basic lesson is simple: a voice agent cannot afford to spend the entire latency budget inside the LLM.
Human conversation provides another useful benchmark. Daily has previously highlighted research suggesting that typical human conversational responses occur at around 500 milliseconds, while pauses beyond roughly 800 milliseconds can begin to sound unnatural.
For AI systems, that means infrastructure, endpoint detection, model inference and speech generation all have to work together.
Why Benchmark Methodology Changes the Results
Before comparing providers, it is important to understand why published latency numbers can differ dramatically.
1. Prompt size has a major impact
The amount of input supplied to a model affects both TTFT and generation performance.
Artificial Analysis changed its default benchmark workload in March 2026 to use 10,000 input tokens instead of 1,000 tokens. That is particularly relevant to AI agents because real applications often send large system prompts containing instructions, persona definitions, safety policies, retrieved information, tool descriptions and conversation history.
A 10,000-token workload therefore provides a different picture from a short chatbot prompt.
2. Geography affects latency
Network location matters.
Artificial Analysis conducts its API performance testing from a Google Cloud virtual machine located in the us-central1-a zone. Consequently, the reported TTFT includes network effects.
A provider may therefore perform differently depending on where the user, application server and inference infrastructure are located.
3. Reasoning changes TTFT
Reasoning models introduce another complication.
For reasoning-capable systems, the first token returned by the API may represent an internal reasoning step rather than the first visible answer token. Benchmarks need to distinguish between these measurements.
Increasing reasoning effort can significantly increase response latency.
4. Client-side measurement is important
Provider-reported TTFT may only measure internal inference time.
For a real application, developers care about the time from sending the request to receiving usable output at the application.
That difference includes network travel, routing, queueing and other infrastructure overhead.
5. Latency is not static
AI inference infrastructure changes continuously.
Providers can modify GPU allocation, routing, batching, caching, model weights and serving software without changing the public model name. Consequently, a benchmark performed today may not reproduce exactly the same result several weeks later.
For voice applications, developers should therefore benchmark the complete production architecture rather than relying exclusively on historical leaderboard rankings.
LLM Benchmark: Which APIs Start Generating Fastest?
According to the Artificial Analysis provider leaderboard data supplied for this analysis, retrieved on August 30, 2026, several inference providers achieved sub-500-millisecond first-chunk latency on a 10,000-token workload.
Fastest measured first-chunk latency
| Provider | Model | TTFT | Output Speed |
|---|---|---|---|
| Baseten | gpt-oss-120b (high) | 0.23s | 266 tok/s |
| Baseten | gpt-oss-120b (low) | 0.24s | 271 tok/s |
| DeepInfra | Nemotron 3 Ultra | 0.28s | 371 tok/s |
| Cohere | North Mini Code | 0.32s | 104 tok/s |
| Cohere | Command A+ | 0.40s | 239 tok/s |
| Baseten | Inkling Small | 0.42s | 337 tok/s |
| Modular | Gemma 4 31B (NVFP4) | 0.44s | 243 tok/s |
| Nebius | GLM-5.3-Flash | 0.46s | 206 tok/s |
| Fireworks | Nemotron 3.5 Lightning | 0.46s | 501 tok/s |
| Together AI | Kimi K2.7 Code | 0.47s | 245 tok/s |
| Cerebras | gpt-oss-120b (high) | 0.49s | 1,697 tok/s |
Baseten’s reported 0.23-second TTFT for gpt-oss-120b is the fastest first-chunk result in the supplied Artificial Analysis dataset.
But the table also illustrates why TTFT should not be evaluated in isolation.
Cerebras, for example, records a slightly slower 0.49-second TTFT while producing an extraordinary 1,697 tokens per second. That combination can be highly attractive for voice agents because the model can rapidly produce enough text for the TTS engine to begin speaking.
The Throughput Trap in Voice Inference
The difference between TTFT and generation speed becomes particularly obvious when comparing high-throughput inference providers.
| Provider | Model | TTFT | Output Speed |
|---|---|---|---|
| Cerebras | gpt-oss-120b (high) | 0.49s | 1,697 tok/s |
| Celeris | Celeris-1 | 0.62s | 1,612 tok/s |
| Cerebras | Gemma 4 31B | 0.53s | 1,351 tok/s |
| Groq | gpt-oss-20b (high) | 0.82s | 957 tok/s |
| SambaNova | gpt-oss-120b (high) | 0.92s | 706 tok/s |
| Groq | gpt-oss-120b (low) | 0.69s | 473 tok/s |
| Inception | Mercury 2 | 3.07s | 770 tok/s |
Inception’s Mercury 2 is a particularly interesting example.
Its output rate is 770 tokens per second, which is impressive on paper. However, the reported first chunk does not arrive until 3.07 seconds.
For a voice application, that delay can overwhelm the entire latency budget before speech generation has even started.
Cerebras and Groq demonstrate the opposite scenario: relatively competitive TTFT combined with extremely high output throughput.
For a voice agent, that combination can be much more valuable than maximum tokens-per-second performance alone.
Frontier AI Models Show a Different Latency Profile
Proprietary and frontier models often prioritize reasoning capability, general intelligence and instruction following over raw latency.
The supplied Artificial Analysis data shows the following results:
| Provider | Model | TTFT | Output Speed |
|---|---|---|---|
| Amazon Bedrock | GPT-5.6 Luna, non-reasoning | 0.59s | 181 tok/s |
| Amazon Bedrock | GPT-5.6 Terra, non-reasoning | 0.72s | 103 tok/s |
| OpenAI | GPT-5.6 Luna, non-reasoning | 0.74s | 113 tok/s |
| Gemini 3.7 Flash, low | 0.84s | 315 tok/s | |
| Anthropic | Claude 4.5 Haiku | 0.84s | 82 tok/s |
| Amazon Bedrock | Nova Micro | 0.86s | 264 tok/s |
| Gemini 3.5 Flash, minimal | 0.90s | 202 tok/s | |
| OpenAI | GPT-5.6 Sol, non-reasoning | 1.06s | 71 tok/s |
One of the most interesting observations is that the same model can have different latency depending on where it is hosted.
GPT-5.6 Luna, for example, records 0.59 seconds through Amazon Bedrock compared with 0.74 seconds through the OpenAI API in the supplied measurements.
That does not necessarily mean one platform is universally faster. Routing, geography, infrastructure, queueing and test conditions can all influence the result.
For production teams, model selection and hosting strategy should therefore be considered together.
LiveKit’s Low-Latency Gemma 4 Deployment
LiveKit provides another useful example of how infrastructure optimization can change inference performance.
In its published measurements, Gemma 4 31B on LiveKit Inference recorded approximately 192 milliseconds TTFT, compared with 911 milliseconds for Gemini 2.5 Flash, 966 milliseconds for GPT-5.5, 1,006 milliseconds for GPT-4.1 and 1,876 milliseconds for the same Gemma 4 31B model through OpenRouter.
LiveKit attributes its performance to infrastructure choices including SGLang, speculative decoding and deliberate GPU under-packing designed to reduce queueing delays.
The company reports that warm requests can begin returning tokens in approximately 100 milliseconds.
However, low latency comes with an infrastructure cost. LiveKit lists pricing of approximately $1.20 per million output tokens for this inference offering.
The company’s measurements also emphasize why TTFS matters. Its reported conversation-level TTFS for Gemma 4 31B was approximately 354 milliseconds, substantially lower than the other models included in that comparison.
This is the kind of result voice-agent developers should pay close attention to: not merely when generation starts, but when enough output exists to make the agent speak.
Speech-to-Text Is More Than Transcription
LLM latency receives much of the attention in AI benchmarking, but speech-to-text can become a major bottleneck in a voice application.
The important question is not simply how quickly an STT system produces words.
The more important question is:
How quickly does the system recognize that the user has finished speaking?
If the STT engine transcribes a sentence quickly but waits another 500 milliseconds before deciding the user has finished, the LLM cannot respond until that delay has passed.
Artificial Analysis measures streaming STT using both time to first partial transcript and time to final transcript.
Some vendor-published latency figures include:
| Provider / Model | Reported Latency |
|---|---|
| Deepgram Flux | ~260ms p50 end-of-turn detection |
| Deepgram Nova-3 | Sub-300ms streaming latency |
| AssemblyAI Universal-Streaming | ~300ms immutable word emission |
| Cartesia Ink-2 | ~100ms transcript latency |
| Speechmatics Voice SDK | ~451ms end-of-speech to final |
Deepgram Flux is particularly interesting because it integrates end-of-turn detection into the recognition architecture instead of relying exclusively on a separate voice activity detection system.
Its configuration exposes controls such as eot_threshold and eager_eot_threshold, while the EagerEndOfTurn event can allow the application to begin LLM generation before the final endpoint decision.
That can potentially remove part of the LLM’s latency from the critical path.
The Risk of Starting the LLM Too Early
Preemptive generation is powerful, but it introduces a tradeoff.
If an application starts generating a response from a partial transcript and the user’s final words change the meaning of the sentence, the system may need to discard the initial response and start again.
That can waste tokens and potentially increase rather than reduce total latency.
LiveKit’s approach to preemptive generation illustrates the broader challenge: latency optimization is ultimately about predicting the user’s intent correctly, not simply starting computation earlier.
TTS Latency: When the First Token Still Cannot Be Heard
Once the LLM produces text, the next stage is text-to-speech.
This is another area where published latency numbers need careful interpretation.
ElevenLabs reports approximately 75 milliseconds for Flash v2.5 model inference. But that figure represents model inference time rather than the complete journey from request to audio reaching the user’s ears.
Network round-trip time can add another 20–200 milliseconds depending on geography and network conditions.
Audio playback itself can introduce additional buffering. ElevenLabs notes that around 500 milliseconds of buffering is common in some playback configurations.
This distinction is critical.
A TTS model may technically have 75-millisecond inference latency while the user experiences considerably more delay before hearing the first spoken sound.
Cartesia has similarly reported sub-90-millisecond TTS latency for its newer models and approximately 100 milliseconds for transcript processing.
The key takeaway is that model latency is not the same as user-perceived latency.
TTS Quality and Speed Are Also Connected
Fast speech synthesis is only useful if the resulting voice is natural enough for the application.
The supplied Artificial Analysis provider voice data gives the following blind-listener Elo results:
| TTS Model | Elo | Price / 1M Characters |
|---|---|---|
| Cartesia Sonic 3.6 | 1,288 | $49.00 |
| SpeechifyAI Simba 3.2 | 1,243 | $10.00 |
| Alibaba Qwen-Audio-3.0-TTS-Plus | 1,243 | $27.60 |
| Inworld Realtime TTS-2 Flash | 1,228 | $10.40 |
| BreezeBlue Breeze TTS 2 | 1,220 | $34.00 |
| ElevenLabs v3 Conversational | 1,215 | $50.00 |
| Google Gemini 3.1 Flash TTS | 1,210 | $18.30 |
| ElevenLabs Flash v2.5 | 1,083 | $50.00 |
These numbers demonstrate another tradeoff.
The lowest-latency TTS model is not automatically the highest-quality voice model, and the model with the strongest listener preference may cost more.
For production applications, teams need to balance latency, voice quality, pricing and reliability.
Speech-to-Speech Models Change the Architecture
Speech-to-speech systems attempt to simplify the voice pipeline by processing spoken input and generating spoken output through a unified realtime model.
In theory, eliminating separate STT, LLM and TTS calls should reduce latency.
In practice, the picture is more complicated.
The supplied Artificial Analysis speech-to-speech benchmark shows:
| Model | TTFA |
|---|---|
| Deepslate Opal | 0.44s |
| Gemini 2.5 Flash Native Audio Dialog | 0.63s |
| Grok Voice Think Fast 2.0 High | 0.70s |
| Grok Voice Fast 1.0 | 0.78s |
| Qwen3.5 Omni Flash Realtime | 0.79s |
| OpenAI GPT-Realtime-1.5 | 0.81s |
| OpenAI GPT Realtime Mini | 0.81s |
| OpenAI GPT-Realtime-2.1 Mini Minimal | 0.85s |
| Google Gemini 3.1 Flash Live Minimal | 0.96s |
| OpenAI GPT-Realtime-2.1 Minimal | 0.97s |
| Amazon Nova 2.0 Sonic | 1.14s |
| OpenAI GPT-Realtime-2 High | 1.14s |
| OpenAI GPT-Realtime-2.1 High | 1.21s |
| Google Gemini 3.1 Flash Live High | 2.99s |
| OpenAI GPT-Realtime-2.1 Mini High | 4.28s |
Grok Voice Think Fast 2.0 High is notable because its reported 0.70-second TTFA is accompanied by a 97% speech-reasoning score and 94.7% task success in the supplied benchmark.
The results also reveal a substantial reasoning-effort penalty.
Gemini 3.1 Flash Live increases from 0.96 seconds in its Minimal configuration to 2.99 seconds at High reasoning.
OpenAI’s GPT-Realtime-2.1 increases from 0.97 seconds to 1.21 seconds between Minimal and High reasoning.
This illustrates a fundamental problem for realtime AI: deeper reasoning can improve task performance, but every additional computation step consumes part of the user’s patience.
Why Cascaded Voice Pipelines Still Matter
Speech-to-speech models are attractive because they can reduce the number of separate components involved in a conversation.
However, cascaded systems — STT → LLM → TTS — continue to offer important advantages.
Developers can replace each component independently, inspect transcripts, control prompts, add specialized tools and tune individual latency bottlenecks.
Daily’s benchmarks have also highlighted capability differences between realtime speech models and strong text-based LLMs, particularly for complex instruction following and tool-use scenarios.
That explains why several production voice-agent platforms continue to use a combination of streaming STT, a fast LLM and specialized TTS.
Artificial Analysis’ benchmark of vendor-configured voice-agent systems also shows that multiple commercial platforms use Gemini 2.5 Flash as the LLM component.
The market therefore appears to be converging on a practical strategy: use a fast general-purpose model inside a carefully optimized realtime pipeline rather than assuming a single speech-to-speech model will always win.
A Practical Latency Budget for Voice Agents
Based on the component figures supplied in this benchmark, an aggressive cascaded voice architecture could target approximately:
| Component | Target Budget |
|---|---|
| WebRTC transport and media | 50–150ms |
| STT and end-of-turn detection | ~260ms |
| LLM first chunk | 230–500ms |
| Sentence generation | ~100ms |
| TTS, network and first audio | 150–300ms |
| Estimated total | ~790ms–1.3s |
This puts a well-optimized cascaded system close to the 800-millisecond target, although actual results will vary considerably.
A speech-to-speech architecture might instead look like:
| Component | Target Budget |
|---|---|
| Transport and media | 50–150ms |
| Realtime model TTFA | 700ms–1.0s |
| Estimated total | ~750ms–1.15s |
These are planning estimates rather than measurements of one complete production system.
The important point is that both architectures can potentially deliver conversational latency when properly engineered.
How Developers Can Reduce Voice AI Latency
The benchmark suggests several practical optimization strategies.
Colocate your infrastructure
Network distance can erase the advantage of a fast model.
Running the application, voice transport and inference infrastructure in geographically close regions can often produce a larger improvement than switching between similarly performing models.
Control reasoning effort
Realtime applications rarely need maximum reasoning depth for every sentence.
Where the model supports configurable reasoning levels, lower reasoning settings can dramatically reduce latency.
Optimize for TTFS
Do not stop at TTFT.
Measure how long it takes to generate enough text for the TTS system to start speaking.
A model producing 1,000 tokens per second with a 500-millisecond TTFT may be more useful for voice than a model with a 200-millisecond TTFT but very low output speed.
Reduce tool-call delays
Tool calls can add another round of model inference and network communication.
For voice agents, developers should minimize unnecessary sequential tool calls, combine external API operations where possible and limit the number of tool steps.
Use audio feedback during longer operations
If a tool call or backend process must take longer, silence can make the system appear broken.
Short thinking sounds, acknowledgement phrases or other audio feedback can make waiting periods feel shorter.
Track p95 and p99 latency
Median latency tells only part of the story.
An agent that responds in 700 milliseconds most of the time but occasionally takes four seconds may still feel unreliable to users.
Tail latency is particularly important for telephone agents and customer-support systems.
Instrument every stage
Production systems should record:
- STT latency
- End-of-turn detection
- LLM TTFT
- LLM generation speed
- Tool-call duration
- TTS time to first byte
- Audio playback delay
- End-to-end voice-to-voice latency
Without those measurements, developers can easily optimize the wrong component.
The Bottom Line: The Fastest API Is Not Always the Fastest Voice Agent
The biggest lesson from this benchmark is that TTFT is an important starting point, but it is not the final metric for voice AI.
A model can have an exceptional TTFT and still produce a sluggish conversational experience if its output rate is slow, its TTS layer buffers heavily or its STT system takes too long to determine that the user has finished speaking.
Conversely, a model with slightly higher TTFT can feel extremely fast when it streams output rapidly and feeds an optimized TTS system.
The most useful metric is therefore closer to time to first usable audio, or ultimately the complete voice-to-voice response time.
For teams building realtime agents in 2026, the winning architecture is likely to be the one that balances all of these factors rather than chasing a single leaderboard number.
The current benchmark points toward several strong strategies: use low-latency inference infrastructure, colocate services, limit unnecessary reasoning, optimize endpoint detection, stream aggressively, choose fast TTS and measure the entire pipeline from microphone input to audible response.
Key Takeaways
- TTFT remains an important inference metric, but it does not fully represent voice-agent responsiveness.
- Time to First Sentence (TTFS) can be more meaningful because TTS requires enough generated text to produce useful audio.
- Baseten’s gpt-oss-120b recorded the fastest supplied Artificial Analysis first-chunk result at 0.23 seconds.
- Cerebras demonstrated why throughput matters, recording approximately 1,697 tokens per second alongside 0.49-second TTFT.
- High output speed cannot compensate for an extremely slow first response; Mercury 2’s 3.07-second TTFT illustrates the problem.
- STT endpoint detection can contribute hundreds of milliseconds to a voice turn.
- Vendor TTS latency figures may measure model inference only and exclude network and playback buffering.
- Speech-to-speech systems can simplify the architecture, but they do not automatically outperform carefully optimized cascaded pipelines.
- Higher reasoning effort can substantially increase realtime latency.
- 800 milliseconds of median voice-to-voice latency is an aggressive but useful target for highly conversational systems.
- Developers should benchmark p50, p95 and p99 end-to-end latency, not just model TTFT.
- The best voice AI stack is not necessarily the one with the lowest individual metric; it is the architecture that delivers the fastest usable spoken response while maintaining accuracy, quality and reliability.
As voice AI moves from experimental demos into customer service, enterprise automation, realtime assistants and autonomous agents, latency will become increasingly competitive. The next generation of voice systems will not simply be judged by how intelligent their answers are. They will also be judged by how naturally those answers arrive.
Discover more from AiTechtonic - AI & Informative News
Subscribe to get the latest posts sent to your email.