Full-Duplex-Bench-v3: Benchmarking Tool Use for Full-Duplex Voice Agents Under Real-World Disfluency

FDB-v3 is a new benchmark that stress-tests full-duplex spoken language models on multi-step tool/API-calling tasks using real, naturally disfluent human speech, jointly measuring accuracy, latency, and turn-taking behavior across six leading voice-agent systems.

Problem Statement

Existing full-duplex voice agent benchmarks typically rely on clean or synthetic (TTS-generated) speech and rarely test agentic, multi-step tool use, yet real users speak with fillers, false starts, and self-corrections that can silently break API-calling pipelines. This gap matters because production voice assistants increasingly need to chain function calls while conversing naturally, so evaluating only on idealized speech overstates real-world readiness. FDB-v3 fills this gap with authentic disfluent audio paired with chained-API scenarios across multiple task domains.

Key Novelty

First benchmark to combine entirely real (not synthetic) human speech, explicitly annotated for five disfluency categories, with multi-step/chained tool-use tasks for full-duplex voice agents
Introduces a four-domain task suite requiring sequential API calls, testing agentic reasoning under naturalistic speech rather than isolated single-turn commands
Provides a unified multi-axis evaluation protocol (accuracy, latency, turn-taking) that allows direct, apples-to-apples comparison between native speech-to-speech models and traditional cascaded ASR-LLM-TTS pipelines
Systematically benchmarks six current-generation systems (GPT-Realtime, Gemini Live 2.5/3.1, Grok, Ultravox v0.7, and a Cascaded baseline), exposing consistent trade-offs and shared failure modes across the field

Evaluation Highlights

Pass@1 tool-calling accuracy: GPT-Realtime best at 0.600, indicating substantial headroom on multi-step API-calling tasks under disfluent speech
Latency and turn-taking trade-off: Gemini Live 3.1 fastest (4.25s) but weakest turn-take rate (78.0%), while the Cascaded pipeline achieves a perfect turn-take rate at the cost of highest latency (10.12s); GPT-Realtime best on interruption avoidance (13.5%)
Consistent failure modes across all six systems: mishandling speaker self-corrections and multi-step reasoning collapse on 'hard' scenarios

Signal Assessment

5/10 This is a well-motivated, timely benchmark (not a new model or training method) that fills a genuine evaluation gap by combining real disfluent speech with agentic tool-use, but as the third iteration in a benchmark series it is an incremental, high-value evaluation contribution rather than a paradigm shift.

Methodology

Curate and record entirely real human audio (no TTS-synthesized speech) spanning naturalistic conversational disfluencies
Annotate the audio corpus with five distinct disfluency categories (e.g., fillers, restarts, self-corrections, repetitions, hesitations)
Design scenarios across four task domains that require chained, multi-step API/tool calls rather than single-turn function invocation
Run six model configurations (native full-duplex models plus a Whisper -> GPT-4o -> TTS cascaded pipeline) through identical scenarios
Score each system along three axes -- task-completion accuracy (Pass@1), end-to-end latency, and turn-taking behavior (turn-take rate, interruption handling) -- and analyze failure patterns, especially under self-correction and hard multi-step cases

System Components

Disfluency-annotated real-speech corpus

Real (non-synthetic) human recordings labeled across five disfluency categories to capture naturalistic speaking patterns

Multi-step tool-use task suite

Scenarios spanning four task domains that require chained/sequential API calls, testing agentic reasoning beyond single-turn commands

Model evaluation harness

Standardized pipeline for running and scoring both end-to-end full-duplex speech models and a modular Cascaded (ASR -> LLM -> TTS) baseline under identical conditions

Multi-dimensional metric suite

Combines Pass@1 tool-call accuracy, response latency, turn-take rate, and interruption-avoidance rate into a joint evaluation profile per system

Results

Metric	Cascaded Baseline (Whisper->GPT-4o->TTS)	Best Full-Duplex Model	Delta
Pass@1 tool-call accuracy	Not top performer (qualitative: lower than GPT-Realtime)	0.600 (GPT-Realtime)	GPT-Realtime leads accuracy
End-to-end latency	10.12 s (highest of all systems)	4.25 s (Gemini Live 3.1, fastest)	~5.87 s faster (~58% reduction)
Turn-take rate	Perfect (~100%)	78.0% (Gemini Live 3.1, lowest)	~22 pp worse than Cascaded
Interruption avoidance	Not specified (qualitative: weaker)	13.5% (GPT-Realtime, best)	GPT-Realtime leads

Key Takeaways

No single system dominates all axes: native full-duplex models win on latency and interruption avoidance, while the cascaded pipeline wins on turn-taking precision, forcing practitioners to choose based on deployment priorities
Cascaded ASR-LLM-TTS pipelines remain far too slow (~10s) for responsive agentic voice interactions despite near-perfect turn-taking, limiting their viability for real-time tool-calling assistants
Even the best model (GPT-Realtime) only reaches 0.600 Pass@1 on multi-step, disfluency-laden tool-use tasks, showing current voice agents are not yet reliable for production-grade agentic API orchestration
Self-correction handling and multi-step reasoning under hard scenarios are shared weaknesses across all six systems, marking a clear, high-priority research direction for spoken LLM agents
Benchmarking with real disfluent speech (rather than clean synthetic audio) surfaces failure modes invisible in prior full-duplex benchmarks, arguing for its adoption as standard practice when evaluating voice agents intended for real users

Abstract

We introduce Full-Duplex-Bench-v3 (FDB-v3), a benchmark for evaluating spoken language models under naturalistic speech conditions and multi-step tool use. Unlike prior work, our dataset consists entirely of real human audio annotated for five disfluency categories, paired with scenarios requiring chained API calls across four task domains. We evaluate six model configurations -- GPT-Realtime, Gemini Live 2.5, Gemini Live 3.1, Grok, Ultravox v0.7, and a traditional Cascaded pipeline (Whisper$\rightarrow$GPT-4o$\rightarrow$TTS) -- across accuracy, latency, and turn-taking dimensions. GPT-Realtime leads on Pass@1 (0.600) and interruption avoidance (13.5\%); Gemini Live 3.1 achieves the fastest latency (4.25~s) but the lowest turn-take rate (78.0\%); and the Cascaded baseline, despite a perfect turn-take rate, incurs the highest latency (10.12~s). Across all systems, self-correction handling and multi-step reasoning under hard scenarios remain the most consistent failure modes.