BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Problem Statement
Current turn-based SpeechLMs (e.g., LLaMA-Omni, GLM-4-Voice) depend on external VAD to detect end-of-turn, which prevents them from modeling natural conversational phenomena like overlapping speech, hesitation, and barge-in interruptions. This turn-based bottleneck fundamentally caps the interactivity and naturalness of spoken chatbots, motivating a need for models that jointly and continuously handle listening and speaking within a single decision-making process.
Key Novelty
- Native full-duplex control implemented entirely within a single autoregressive LLM using only a few added special tokens, avoiding any auxiliary turn-taking/VAD module
- Architecture-agnostic design that adds minimal vocabulary changes, enabling transfer across different backbone LLMs while reusing existing training and serving infrastructure without adaptation
- Highly sample-efficient retrofitting of full-duplex capability onto an existing turn-based checkpoint (GLM-4-Voice) using only 400K samples plus a lightweight DPO alignment stage
Evaluation Highlights
- 92% turn-taking success and 100% interruption (barge-in) success on InstructS2S-Eval
- Speech-response quality score improved from 2.17 (Moshi) to 3.39
- Matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, indicating no quality regression from full-duplex modeling
Signal Assessment
Methodology
- Initialize from the public GLM-4-Voice turn-based SpeechLM checkpoint to leverage existing speech-text pretraining
- Extend the vocabulary with a small set of special control tokens representing listen/speak/stop/interrupt decisions, requiring no architectural changes
- Fine-tune the single autoregressive LLM on 400K full-duplex dialogue samples so it learns to jointly model interleaved listening and speaking streams
- Apply a lightweight DPO stage to refine turn-taking timing and interaction quality preferences
- Evaluate full-duplex behavior on InstructS2S-Eval and general response quality on QA/instruction benchmarks
System Components
Backbone decoder that jointly models both listening and speaking channels, making turn-taking decisions as part of standard next-token prediction
Small set of added vocabulary tokens that signal when to listen, speak, stop, or handle interruption, replacing the need for an external VAD/turn-taking module
Warm-start from a public pretrained turn-based SpeechLM checkpoint to reduce training cost and data requirements
Dual-channel dialogue data used to teach the model natural overlap, hesitation, and barge-in behavior
Preference-optimization step applied after supervised fine-tuning to further align turn-taking and response behavior with desired interaction quality
Evaluation protocol measuring turn-taking success, interruption success, and speech-response quality in full-duplex settings
Results
| Metric/Benchmark | Baseline | This Paper | Delta |
|---|---|---|---|
| Turn-taking success (InstructS2S-Eval) | Not natively supported by turn-based/VAD systems | 92% | New full-duplex capability |
| Interruption/barge-in success (InstructS2S-Eval) | Not natively supported by turn-based/VAD systems | 100% | New full-duplex capability |
| Speech-response quality score | 2.17 (Moshi) | 3.39 | +1.22 absolute (~56% relative) |
| General QA/instruction quality (Llama Questions, Web Questions, Alpaca-Eval) | Turn-based GLM-4-Voice counterpart | Matches or exceeds baseline | No quality sacrifice |
Key Takeaways
- Full-duplex conversational ability can be added to an existing turn-based SpeechLM via a small vocabulary extension rather than a bespoke multi-stream architecture, simplifying deployment
- Retrofitting full-duplex behavior is data- and compute-efficient (400K samples + lightweight DPO), making it practical to upgrade existing checkpoints rather than training from scratch
- Because the approach reuses standard training/serving stacks, it should generalize to other autoregressive LLM backbones with minimal engineering overhead
- Enabling natural interruption and overlap handling does not require trading off general response quality, as shown by parity with turn-based baselines on standard QA benchmarks
Abstract
Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.