Learning When to Think While Listening in Large Audio-Language Models

The paper proposes a learnable 'wait-think-answer' controller for streaming Large Audio-Language Models that decides, moment-to-moment during partial audio input, whether to keep listening, emit a compact intermediate reasoning update, or commit to an answer, trained via SFT followed by multi-reward DAPO reinforcement learning.

Problem Statement

In real-time spoken dialogue, LALMs face a fundamental trade-off: deferring reasoning until the speech endpoint improves answer quality but adds user-visible latency, while answering early risks committing before enough evidence has arrived. Existing LALMs typically reason only after full utterance capture, lacking a mechanism to adaptively interleave thinking and responding during the audio stream itself, which limits their usability in low-latency conversational settings.

Key Novelty

A new 'wait-think-answer' action-control formulation that lets the model decide under partial audio evidence when to wait, when to externalize a compact reasoning update, and when to answer, mimicking incremental human conversation
Construction of aligned wait-think-answer training traces from spoken reasoning data to bootstrap the controller via SFT
A six-component reward (answer correctness, action validity, update timing, latency synchronization, reasoning quality, chain consistency) optimized end-to-end with DAPO over the full trajectory rather than just the final answer
Cross-domain transfer evaluation from a synthetic TTS-based benchmark to a human-recorded Real Audio Bench to test robustness beyond synthetic speech

Evaluation Highlights

On the six-task synthetic SRQA benchmark, the six-reward DAPO controller raises row-weighted accuracy from 67.6% to 70.3% while cutting post-endpoint final-think length by 14% under the same deployment harness
On the 186-item human-recorded Real Audio Bench (TTS-to-real-speech transfer), SFT achieves the highest accuracy among controller variants, while the six-reward DAPO controller is the only learned variant whose final-think length drops below the untuned base model

Signal Assessment

6/10 The work applies known techniques (SFT + DAPO) to a well-motivated new control formulation for streaming audio reasoning, yielding modest but consistent accuracy and latency gains plus a useful real-audio robustness check, making it a solid rather than paradigm-shifting contribution.

Methodology

Start from Qwen2.5-Omni-7B as the base LALM and define a wait/think/answer action space over the incoming audio stream
Construct aligned wait-think-answer traces by aligning spoken reasoning data with the timing of these actions
Warm-start the controller with supervised fine-tuning (SFT) on these traces
Refine the policy with Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO), using a six-component reward that scores correctness, action validity, update timing, latency synchronization, reasoning quality, and chain consistency across the entire trajectory
Evaluate on a synthetic six-task SRQA benchmark and a human-recorded Real Audio Bench to test in-domain performance and out-of-domain transfer

System Components

Wait-Think-Answer Controller

A policy layered on the LALM that, at each point in the audio stream, chooses to wait, emit a compact reasoning update, or produce a final answer

Qwen2.5-Omni-7B base model

The underlying pretrained large audio-language model providing multimodal (speech + text) understanding and generation

Aligned wait-think-answer trace construction

A data pipeline that converts spoken reasoning data into supervised sequences labeling wait/think/answer decisions in time

SFT stage

Initial supervised fine-tuning of the controller on constructed traces to learn baseline action timing and reasoning behavior

DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization)

An RL algorithm used to optimize the controller's full decision trajectory rather than only the final answer

Six-component reward function

Combines answer correctness, action validity, update timing, latency synchronization, reasoning quality, and chain consistency into a single trajectory-level training signal

SRQA Benchmark

A six-task synthetic spoken reasoning question-answering benchmark used for controlled in-domain evaluation

Real Audio Bench

A 186-item human-recorded evaluation set used to test transfer beyond TTS-synthesized speech

Results

Metric/Benchmark	Baseline	This Paper (6-reward DAPO)	Delta
SRQA row-weighted accuracy (6-task synthetic)	67.6%	70.3%	+2.7 pp
SRQA post-endpoint final-think length	Base length	-14%	-14% reduction
Real Audio Bench accuracy (186 human-recorded items)	Untuned base model	SFT variant is strongest; DAPO remains functional	Qualitative gain (SFT > DAPO for accuracy)
Real Audio Bench final-think length vs base	Base model length	Only 6-reward DAPO falls below base	Reduced latency signal (DAPO only)

Key Takeaways

Explicit, learnable control over when to 'think' versus 'answer' during streaming audio can decouple reasoning depth from user-perceived latency, rather than forcing a single wait-then-reason pipeline
Reward-shaping across the whole decision trajectory (correctness + timing + latency + reasoning quality + consistency) via DAPO can jointly improve accuracy and reduce unnecessary post-endpoint reasoning, rather than trading one for the other
Synthetic TTS-trained controllers do not automatically transfer to human speech: real-audio evaluation revealed SFT generalizing better on accuracy than RL-tuned variants, underscoring the need for real-audio validation in streaming LALM development
The wait-think-answer formulation offers a practical template for building latency-aware conversational/agentic audio systems where intermediate reasoning must be surfaced adaptively rather than always deferred or always immediate

Abstract

Recent advances in Large Audio-Language Models (LALMs) have made real-time, streaming spoken interaction increasingly practical. In this setting, reasoning quality and responsiveness are tightly coupled: delaying reasoning until the speech endpoint can improve answer quality but moves deliberation into user-visible response delay, while answering too early risks committing before decisive evidence arrives. We introduce a learnable wait-think-answer control formulation for LALMs. Motivated by the incremental nature of human conversation, the controller decides under partial audio evidence when to wait, when to externalize a compact reasoning update, and when to answer. Using Qwen2.5-Omni-7B as the base model, we construct aligned wait-think-answer traces from spoken reasoning data, train the controller with supervised fine-tuning (SFT), and then apply Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO). The reward combines answer correctness, action validity, update timing, latency synchronization, reasoning quality, and chain consistency, optimizing the complete wait-think-answer trajectory and not the final answer alone. On a six-task synthetic spoken reasoning question answering (SRQA) benchmark, the six-reward DAPO controller improves the row-weighted accuracy from 67.6% to 70.3% while reducing post-endpoint final-think length by 14% under the same Qwen deployment harness. On a 186-item human-recorded Real Audio Bench, a transfer check beyond text-to-speech (TTS)-rendered speech, the controller family remains functional: SFT achieves the strongest accuracy, while the six-reward DAPO controller is the only learned variant whose final-think length falls below the base. These results suggest that a streaming model should learn when to make intermediate reasoning explicit during the audio stream.