Open Full-duplex Voice Agent with Speech-to-Speech Language Model
Problem Statement
Human-like conversational voice agents require full-duplex interaction (simultaneous listening and speaking), which is difficult to achieve with traditional cascaded ASR→LLM→TTS pipelines due to latency and rigid turn-taking. Existing end-to-end S2S models that support duplex behavior typically demand extensive, costly speech-text pretraining, restricting access to well-resourced labs. This work targets that accessibility gap by offering a generalizable, low-data, fully open-source alternative.
Key Novelty
- A data-efficient adaptation method that converts any standard text LLM into a full-duplex S2S model without extensive speech-text pretraining
- A modeling approach enabling simultaneous listen-and-speak behavior directly within the LLM, rather than through separate turn-based modules
- An integrated open-source stack combining modeling, inference optimization, and serving for real-time, low-latency voice interaction
- A generalizable framework designed to work across different base LLMs rather than being tied to one specific architecture
Evaluation Highlights
- System demonstration of a working low-latency, full-duplex voice agent built end-to-end from open-source components
- Qualitative comparison highlighting reduced speech-text pretraining data requirements relative to conventional S2S approaches
Signal Assessment
Methodology
- Start from a pretrained, standard text-only LLM as the reasoning/generation backbone
- Apply a lightweight, data-efficient adaptation procedure to add speech input/output modeling without full-scale speech-text pretraining
- Introduce a full-duplex architecture allowing concurrent processing of incoming audio and generation of outgoing speech tokens
- Optimize inference (e.g., streaming decoding, caching) to minimize end-to-end response latency
- Package the model with open-source serving infrastructure to deploy a real-time conversational voice agent
System Components
Any standard pretrained language model that provides the core reasoning and generation capability
Converts raw audio into discrete tokens and back, bridging speech and the LLM's token-based interface
Enables the model to process incoming speech and generate outgoing speech simultaneously, supporting natural, interruption-tolerant dialogue
Fine-tuning/adaptation technique that imparts speech capabilities to the LLM without requiring extensive speech-text pretraining corpora
Techniques (e.g., streaming, efficient decoding) that reduce latency to enable responsive, real-time conversation
Deployment infrastructure that assembles the model and optimizations into a usable, low-latency voice agent
Results
| Aspect | Conventional S2S Approach | This Paper | Improvement |
|---|---|---|---|
| Speech-text pretraining data needed | Extensive/large-scale | Minimal (data-efficient) | Substantially reduced data requirement |
| Duplex capability | Often half-duplex/turn-based | Full-duplex (simultaneous listen & speak) | More natural, human-like conversation |
| Latency | High (cascaded pipeline overhead) | Low-latency, real-time | Improved responsiveness |
| Accessibility/openness | Often closed or resource-intensive | Fully open-source stack (model, inference, serving) | Lower barrier to entry for practitioners |
Key Takeaways
- Practitioners can turn existing open-source text LLMs into speech-capable conversational agents without collecting or training on massive speech-text corpora
- Full-duplex (simultaneous listen/speak) behavior can be engineered into standard LLM architectures rather than requiring bespoke speech-native models built from scratch
- The full open-source release (modeling + inference optimization + serving) offers a ready blueprint for deploying production-grade, low-latency voice agents
- This approach could accelerate adoption of natural, human-like voice AI by smaller teams lacking access to large proprietary speech datasets or infrastructure
Abstract
We present the system demonstration and opensource code release of a novel, data-efficient framework that converts any standard text Large Language Model (LLM) into a full-duplex end-to-end (E2E) speech-to-speech (S2S) model, for building conversational voice agents. Our new modeling method enables any LLMs to simultaneously listen and speak without requiring extensive speech-text pretraining. Moreover, we demonstrate how to put together a low-latency and full-duplex voice agent with open-source modeling, inference optimization, and serving solutions. This work significantly lowers the barrier to entry for developing low-latency, human-like voice agents by providing a generalizable, end-to-end solution built on open-source technologies.