arXiv CSopen access

Learning When to Think While Listening in Large Audio-Language Models

Learning When to Think While Listening in Large Audio-Language Models

arXiv:2605.27190v1 [cs.CL] 26 May 2026

Zhiyuan Song

Weici Zhao Yang Xiao Suhao Yu University of Pennsylvania

Cheng Zhu

Jiatao Gu∗

Abstract Recent advances in Large Audio-Language Models (LALMs) have made real-time, streaming spoken interaction increasingly practical. In this setting, reasoning quality and responsiveness are tightly coupled: delaying reasoning until the speech endpoint can improve answer quality but moves deliberation into user-visible response delay, while answering too early risks committing before decisive evidence arrives. We introduce a learnable wait-think-answer control formulation for LALMs. Motivated by the incremental nature of human conversation, the controller decides under partial audio evidence when to wait, when to externalize a compact reasoning update, and when to answer. Using Qwen2.5-Omni-7B as the base model, we construct aligned wait-think-answer traces from spoken reasoning data, train the controller with supervised fine-tuning (SFT), and then apply Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO). The reward combines answer correctness, action validity, update timing, latency synchronization, reasoning quality, and chain consistency, optimizing the complete wait-think-answer trajectory and not the final answer alone. On a six-task synthetic spoken reasoning question answering (SRQA) benchmark, the six-reward DAPO controller improves the row-weighted accuracy from 67.6% to 70.3% while reducing post-endpoint final-think length by 14% under the same Qwen deployment harness. On a 186item human-recorded Real Audio Bench, a transfer check beyond text-to-speech (TTS)-rendered speech, the controller family remains functional: SFT achieves the strongest accuracy, while the six-reward DAPO controller is the only learned variant whose final-think length falls below the base. These results suggest that a streaming model should learn when to make intermediate reasoning explicit during the audio stream. The public repository is available on GitHub.

1

Introduction

Large Audio-Language Models (LALMs) are moving audio research from offline perception tasks toward real-time spoken interaction. Recent systems can process streaming audio and generate text or speech with low delay [10–12, 15, 16], making it increasingly realistic to ask an audio model to reason during live speech, before the utterance has finished. In this setting, answer quality and responsiveness are coupled: a spoken assistant that answers correctly only after a long silent delay can still fail as an interactive system, while a fast response can be wrong if late-arriving evidence changes the answer. Reasoning-intensive speech exposes a basic control problem. Most spoken reasoning evaluations follow a complete-audio protocol: the model receives the full spoken question, generates a reasoning trace, and then produces a response. This protocol is useful for measuring final answer quality, but it removes the online decision faced by a streaming LALM. At each time step in the audio stream, the model has partial acoustic and linguistic evidence plus whatever reasoning state it has already ∗ Corresponding author.

Preprint.

emitted. Waiting until the speech endpoint can preserve accuracy, but it moves deliberation into the user-visible response delay; answering early is responsive, but risks committing before decisive evidence arrives. The central question is when the model should update its reasoning, alongside what the reasoning should contain. This temporal question is natural in human conversation, where listeners often prepare responses before the current turn has ended and where short response gaps require overlap between comprehension and production planning [3–5, 7–9]. It is also becoming explicit in speech-model research. Chain-of-thought and reasoning-oriented post-training improve complex problem solving in text and audio models [22–24, 35, 39, 40], while recent streaming systems ask whether models can think while listening or interleave reasoning with speech generation [31–34]. We build on this direction but train a different primitive: a stateful wait-think-answer controller whose visible thoughts accumulate across streaming audio windows, instead of a single post-audio rationale or a question-completeness trigger. We introduce a learnable wait-think-answer formulation for streaming speech reasoning. In this formulation, the LALM is an online controller whose serialized action space contains three primitives: <wait/>, <think>...</think>, and <answer>...</answer>. A wait action consumes more audio without changing the visible reasoning state. A think action emits a grounded intermediate update, and an answer action commits to the final response. Restricting the action space to these three primitives keeps timing trainable and inspectable while leaving the model free to generate open-ended text inside thinking and answering spans. Distributing reasoning across pre-endpoint think actions amortizes the total reasoning cost: each visible update emitted during listening absorbs deliberation that would otherwise accumulate into the final post-endpoint think, reducing residual post-endpoint deliberation as a controller-level proxy for user-visible response delay. We instantiate this controller on Qwen2.5-Omni-7B [15]. The data pipeline constructs aligned waitthink-answer traces from spoken reasoning examples, renders the user input as speech, aligns text actions to audio time, and converts each example into chained streaming windows. We first train the controller with supervised fine-tuning (SFT), then use the resulting controller to initialize Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO). The reward combines answer correctness, action validity, update timing, latency synchronization, thought quality, and chain consistency, so optimization covers the complete streaming trajectory. The experiments focus on the resulting accuracy–residual-latency trade-off under the deployment protocol. On synthetic spoken reasoning, the six-reward DAPO controller improves row-weighted accuracy from 67.6% for the base deployment controller to 70.3%, while reducing post-endpoint final-think length from 10.44 to 8.99 tokens. We also evaluate the controllers on Real Audio Bench, a 186-item human-recorded transfer check. There, SFT gives the strongest accuracy, while DAPO variants expose shorter-reasoning operating points. The six-reward variant is the only learned controller whose final-think length falls below the base deployment controller. These results support the main claim that online LALM reasoning can be trained as wait-think-answer control: the model learns when to wait and when to externalize intermediate reasoning during listening, while answer commitment is endpoint-gated in this paper. The paper contributes a training formulation for wait-think-answer control in LALMs, focusing on the learning problem and controller semantics, not optimized cache-native serving: a learned policy over partial audio evidence, an aligned spoken-controller data pipeline with supervised fine-tuning and policy optimization, and a reward that scores correctness, timing, latency, and thought consistency. We evaluate the resulting controllers on synthetic spoken reasoning and human-recorded audio, showing how post-training moves the controller along the accuracy–residual-latency frontier.

2

Related work

We review three lines of work: large audio-language and omni models, audio chain-of-thought reasoning, and online spoken interaction. 2.1

Large audio-language and omni models

LALMs extend text large language models (LLMs) with audio encoders and cross-modal alignment modules, allowing speech, sound, and music to be used as model context. Surveys of this area 2

I sent 20 emails today

and I just sent 5 more

How many emails I sent?

email count so far is 20.

updated email count: 20 + 5 = 25.

the total includes both batches, so it is 25.

Answer: 25 emails

Figure 1: Wait-think-answer controller in full-prefix mode. At decision step k, the controller observes the audio prefix x1:tk and the committed text memory z<k , then emits <wait/>, a visible state update, or a final answer. The diagram focuses on controller-level information flow, not the internal multimodal token layout. I sent 20 emails today, and I just sent 5 more, how many emails I sent? 25 emails Tought Quality

Latency Control

Judge

Rule

Format Control

Rule

describe a rapid shift from audio<think> systems toward general<think> instruction-following models <wait/> </think> <think> Need totask-specific answer the The user send 20, <answer> </answer> <wait/> question. </think> 5 more </think> for audio understanding and dialogue [1, 2]. then Qwen-Audio and Qwen2-Audio are representative <think> </think> <wait/>and music tasks large-scale audio-language trained <think> across heterogeneous speech, sound, <think> Count sosystems far = 20 </think> 20+5 = 25 </think> <wait/><answer> </answer> [13, 14]. More recent omni models aim to remove the cascaded automatic speech recognition (ASR)– LLM–text-to-speech (TTS) boundary. Moshi uses parallel speech streams and an inner monologue <think> </think> <wait/> <wait/> <think> </think> <answer> </answer> channel [10], Mini-Omni and Freeze-Omni target<wait/> low-latency speech-to-speech dialogue [11, 12], ... and Qwen2.5-Omni introduces a streaming Thinker–Talker architecture that perceives audio, vision, and text while generating text and speech [15]. WeUpdate use this as the substrate, but Timingomni-model setting Answer Correctness Chain Consistency study a different problem: how the model should update an explicit reasoning state while streaming <think>count = 20 </think> I bought<think></think> <answer> 20 emails </answer> audio is still arriving. <answer> 25 emails </answer> 25 emails<wait/> ... Rule

Judge

2.2

<think>count = 20 </think> <think> 20+5 = 25 </think> <answer> 25 emails </answer>

I bought<wait/> 25

Audio chain-of-thought reasoning andemails<think></think>... post-training

Rule

<answer> 25 emails </answer>

Chain-of-thought (CoT) prompting established that explicit intermediate reasoning can improve text LLM performance on arithmetic, commonsense, and symbolic tasks [35]. Audio-CoT transfers this idea to LALMs and finds that CoT can help easy and medium audio reasoning tasks, while hard tasks can suffer when the reasoning chain is low quality or unnecessarily long [22]. Subsequent audio reasoning systems improve the reasoning process through structured data, curriculum learning, or reward-based post-training [23, 25–30]. Audio-Thinker studies adaptive audio reasoning by deciding whether to produce a final pre-answer rationale and by training the reasoning process to stay consistent [24]. Our problem is different because the controller must act before the complete utterance is available. It repeatedly decides under partial audio evidence whether to wait, emit a visible state update, or answer, instead of choosing only whether to reason after hearing the audio. SFT teaches this action language. DAPO, building on recent reasoning reinforcement learning methods [39, 40], then optimizes the whole streaming trajectory for answer correctness, update timing, thought quality, and response latency. 2.3

Thinking while listening and turn-taking

Human conversation is fast enough that listeners often cannot wait for a turn to finish before preparing a response. Conversation analysis and cross-linguistic studies characterize turn-taking as a locally managed system that avoids both overlap and long silence [3, 4]. Levinson and Torreira argue that such short response gaps imply overlap between comprehension and production planning [5]. Electroencephalography (EEG) and speech-planning studies provide evidence that response preparation can begin before the current speaker finishes [6–8]. Speaker–listener neural coupling further links successful communication to shared, time-evolving representations [9]. Recent speechLLM work turns this observation into an engineering problem. Shih et al. ask whether speech LLMs can think while listening: at inference time, a question-completeness check (using entropy decrease over the streaming ASR distribution) decides when the user has finished speaking, and the model then emits a single post-utterance chain of thought, with the accuracy–latency trade-off shaped 3

by length-biased preference optimization [31]. STITCH, SHANKS, and StreamingThinker study related forms of simultaneous reasoning over speech or text streams [32–34]. The wait-think-answer formulation follows this motivation but trains the controller end-to-end to act under partial audio: instead of a single terminal chain-of-thought triggered by an inference-time completeness signal, the model learns to emit multiple grounded intermediate state updates over the audio stream, each visible to and reusable by later controller decisions.

3

Method

3.1

Architecture

We treat streaming speech reasoning as an online control problem over an explicit text memory (Figure 1). Let x1:T denote the spoken input, let 0 < t1 < · · · < tK ≤ T be controller decision times, and let z<k be the visible reasoning states already emitted by the model. We use a full-prefix controller observation: at decision step k, the audio input is the complete prefix x1:tk heard so far, including the newest chunk and all earlier audio, while the text input is the visible reasoning state z<k . The controller observation is ok = (x1:tk , z<k ),

ak = arg max πθ (a | ok ), a∈Ak

(1)

where ok denotes the controller observation, ak denotes the action taken at step k, πθ is the controller policy with parameters θ, and Ak is the legal action set at that step. Equation (1) is the deterministic deployment rule. The underlying policy πθ (· | ok ) is a distribution over legal actions, and DAPO training samples from it to generate rollout groups. We omit the fixed controller instruction from the notation because it is constant across examples. Before the speech endpoint, the legal actions are WAIT and THINK. After the endpoint, the controller emits one final THINK and then an ANSWER. We serialize these actions as <wait/>, <think>...</think>, and <answer>...</answer>. Thus, answer timing is not a learned pre-endpoint stopping decision in our experiments; the ANSWER action is retained to train and validate the complete interaction contract. A WAIT advances the stream without changing memory. A pre-endpoint THINK appends a short semantic state update zk , so every later controller call can condition on both the complete audio prefix and the previously committed thoughts. At the endpoint, the final THINK receives the complete audio x1:T and all earlier visible thoughts, and the answer turn receives the same complete audio plus the final-think state before emitting ŷ. The same information pattern would appear in a cache-native streaming deployment. In that deployment, audio chunks would keep entering one persistent Qwen2.5-Omni context [15], the audio key-value (KV) cache would not be discarded, intermediate THINK text would be appended back into the visible context, the final-think turn would see the complete cached audio plus all earlier thoughts, and the answer turn would see that same context plus the final-think state. The available Qwen2.5-Omni serving path does not expose an official controller-style cache interface for this loop, and an efficient custom implementation would require lower-level runtime and kernel work. We study the same information flow through full-prefix replay: every controller call replays the observed prefix while preserving the evidence available in the intended deployment semantics. Appendix B and Section 6 discuss this approximation further. 3.2

Training data construction

Following the spoken reasoning data format of Shih et al. [31], we first generate semantic controller traces with GPT-4o [19] and then ground them in speech. The initial candidate corpus contains 80,000 records: 40,000 verifiable items and 40,000 open-ended items. Generation is stratified by verifiability, difficulty, and topic. Appendix C gives the prompt and field schema. Each record stores a spoken surface form, a TTS style instruction, lexical anchors for answer-relevant state changes, a semantic wait-think-answer trace, and the final answer. After validation, deduplication, and human spot checks, the aligned corpus used for the reported experiments contains 75,723 audio-text records: 38,213 verifiable and 37,510 open-ended. The train/validation split contains 73,675 and 2,048 records, respectively. SFT uses the full aligned corpus. DAPO keeps the verifiable training branch, yielding 37,180 scorable training records. To expose the reward to benchmark-style reasoning formats during training, we add a small set of training-split 4

Table 1: Reward terms for wait-think-answer control. Each term targets a specific failure mode in the streaming trajectory, from invalid action format to unsupported final reasoning. Term Role

Training signal

Ra Rf Rs Ru Rt Rc

Rewards the final answer using deterministic task scoring whenever possible; a local judge is used only as a semantic-equivalence fallback for open short-answer cases. Enforces the wait-think-answer contract: listen-time actions are wait/think, and the answer appears only after the completed utterance. Favors compact final-think states, so reasoning already resolved during listening is not repeated after the endpoint. Rewards thoughts near answer-relevant state changes and penalizes both missed updates and irrelevant interruptions. Prefers short, concrete semantic states that support the answer over generic or verbose commentary. Checks whether the visible thought chain supports the final answer without unsupported jumps or contradictions.

Answer correctness Protocol validity Response latency Update timing Thought quality Chain consistency

examples from ARC-Challenge, ARC-Easy, GSM8K, PIQA, and SocialIQA. These rows remain below 2% of the DAPO mix and never include held-out evaluation examples. We synthesize each spoken input as a single full utterance with Qwen3-TTS [18]. This preserves global prosody and avoids artificial breaks at action boundaries. Connectionist temporal classification (CTC)-style forced alignment [41] maps transcript words to timestamps, and controller boundaries are snapped upward to the 0.5s decision grid. This grid controls reconsideration frequency, not the amount of audio evidence visible at each call. Appendix B reports the tick sweep. 3.3

Supervised fine-tuning

We first train the controller with supervised fine-tuning on Qwen2.5-Omni-7B [15] using low-rank adaptation (LoRA) [36] in MS-Swift [37]. The supervised objective teaches the action serialization, the short semantic state style, and the distinction between ordinary waits, answer-relevant pre-endpoint thoughts, final-think compression, and the final answer. The supervised controller is trained from the base Qwen2.5-Omni-7B model for one epoch over the aligned controller export. Training and validation curves show decreasing loss and increasing token accuracy, with both stabilizing during training. DAPO runs are initialized from supervised controllers trained with this recipe. In the reported tables, the SFT row reports this supervised controller, and the DAPO rows report four-, five-, and six-reward policy-optimization variants. Appendix B reports the training curves, hyperparameters, and compute details. 3.4

DAPO policy optimization

After SFT, we optimize the controller with a controller-specific DAPO policy-optimization loop [40]. SFT can be handled directly in MS-Swift, but DAPO requires live streaming rollouts, controller-action parsing, local reward computation, dynamic resampling, and adapter updates inside the controller runtime. This requirement motivates a custom trainer instead of the generic MS-Swift Group Relative Policy Optimization (GRPO) path. DAPO belongs to the GRPO family [38, 39]. Each prompt is sampled into a group of G rollouts scored with the trajectory reward in Eq. 3. We compute group-relative advantages and apply token-level clipped updates to controller, thought, and answer tokens. Token-level credit is useful because the policy mixes actions with different failure modes: bad waits, useful short thoughts, malformed final-think turns, and correct answers should not all receive identical pressure. Appendix B gives the objective, clipping settings, Kullback–Leibler (KL) regularization, and protocol-gate details. 3.5

Reward design

The reward targets the main failure modes of online speech reasoning. Optimizing only the final answer encourages a wait-all policy that shifts deliberation to the user-visible response delay. Optimizing only latency can produce premature answers, empty thoughts, or malformed action traces. We score the full wait-think-answer trajectory so that a good rollout needs to answer correctly, follow the interaction format, update its state when the evidence changes, and keep the final post-endpoint deliberation compact. Table 1 summarizes the six reward terms, and Figure 2 illustrates how they score a wait-think-answer trajectory. Let τ be a completed streaming trajectory. The reward combines four rule-based terms with two judge-assisted reasoning terms. Ra measures final answer correctness through task scoring, with judge scoring used only as a semantic-equivalence fallback for open short-answer cases. Rf enforces the controller format, Rs penalizes residual final-think latency, and Ru rewards well-timed state updates. Inspired by process-level audio reasoning rewards [24], Rt scores whether each visible 5

thought is short and answer-supporting, while Rc scores whether the resulting thought chain supports the final answer. The shaped reward for a valid trajectory is Rvalid (τ ) = λa Ra + λf Rf + λs Rs + λu Ru + | {z } | {z } | {z } | {z } answer

sync/latency

format

update timing

and the final reward applies a protocol gate:  λ f Rf , R(τ ) = Rvalid (τ ),

λt Rt | {z }

thought quality

+ 1[Ra > 0]λc Ra Rc , | {z }

(2)

consistency bonus

Rf ≤ 0, Rf > 0.

(3)

We use λa = 1.0, λf = 1.0, λs = 1.0, λu = 3.0, λt = 1.0, and λc = 0.45. The gate gives protocol validity priority over all other objectives: if the model answers before the utterance is complete, omits the required final-think/answer structure, or emits an ill-formed trace, answer correctness cannot rescue the rollout. The consistency term is also gated by Ra > 0, so a fluent but wrong chain does not receive extra credit simply for being self-consistent. The ablations in Section 5 add these terms progressively. The four rule-based terms (Ra , Rf , Rs , Ru ) establish the basic accuracy–residual-latency–timing trade-off. Adding Rt and Rc asks a stricter question: whether improving the content of the intermediate state also improves the ability of the controller to spend reasoning earlier in the stream and less after the endpoint.

4

Experiments and evaluation

Synthetic spoken reasoning question-answering benchmark. We adopt the spoken reasoning question answering (SRQA) evaluation construction introduced by Shih et al. [31]. Following their setup, text reasoning problems are rewritten into spoken questions, rendered with TTS, and then evaluated as audio inputs. Our benchmark contains 8,959 items across ARC-Easy (ARC-E), ARCChallenge (ARC-C) [42], Physical Interaction: Question Answering (PIQA) [43], SocialIQA (SIQA) [44], GSM8K [45], and a 300-item short-answer factual set (LLaMA-QS) following the LLaMAQuestions/Spectron source family [46]. We follow their spoken-rewrite prompt lineage and use a local I sent 20 and I just How many emails today emails I sent? Qwen3.6-35B-A3B judge [17]sent for5 more short-answer scoring, together with task-specific normalization for numeric and multiple-choice answers. Appendix A gives task counts and conversion details. email count so far is 20.

updated email count: 20 + 5 = 25.

the total includes both batches, so it is 25.

Answer: 25 emails

Evaluation protocols. We report two protocols. Offline mode is the complete-audio setting: the model receives the full spoken question once, emits one final <think>...</think> state, and then emits the answer. This protocol measures answer quality after complete audio observation and serves as the standard upper-bound reference for models that do not expose a controller interface. Deployment mode is our streaming protocol: audio arrives on a fixed 0.5s grid, each controller call sees the full audio prefix plus previous visible thoughts, and the policy chooses <wait/> I sent 20 emails today, and I just sent 5 more, how many emails I sent? Thought Quality

Latency Control

Judge

25 emails Rule

Format Control

<think> Need to answer the question. </think>

<think> The user send 20, then 5 more </think>

<think> <wait/> </think> <answer> </answer> <wait/>

<think> Count so far = 20 </think>

<think> 20+5 = 25 </think>

<think> </think> <wait/> <wait/><answer> </answer>

<think>

</think>

<wait/>

<wait/>

<wait/>

<think>

</think>

Rule

<answer>

</answer>

Answer Correctness

Rule

... Chain Consistency

Judge

Update Timing

<think>count = 20 </think> <answer> 25 emails </answer>

I bought<think></think> 25 emails<wait/> ...

<think>count = 20 </think> <think> 20+5 = 25 </think> <answer> 25 emails </answer>

I bought<wait/> 25 emails<think></think>...

Rule

<answer> 20 emails </answer> <answer> 25 emails </answer>

Figure 2: Trajectory reward for wait-think-answer control. Rule-based terms enforce action format, update timing, answer correctness, and final-latency budget, while judge-assisted terms score local thought quality and chain consistency. 6

or <think>...</think> before the endpoint. At the endpoint, the controller emits one final <think>...</think> and then an <answer>...</answer>. If a controller call is still running when a later tick arrives, stale triggers can be skipped, matching the practical constraint that an online system cannot spend unbounded work at every trigger. Deployment mode is only available for Qwen-family rows, because the controller requires direct control over the Qwen2.5-Omni inference loop [15]: prefix replay, explicit wait/think/answer action parsing, and persistence of intermediate thought states must all be implemented inside a single benchmark harness. External baselines do not expose a comparable controller interface in our setup, so we evaluate them only in offline mode. To separate model quality from protocol effects, the base Qwen2.5-Omni-7B is reported in both offline and deployment modes. Human-recorded real audio. We further introduce Real Audio Bench, a human-recorded streaming benchmark collected for this work. Five speakers recorded 200 GPT-4o-generated [19] candidate items designed for natural spoken delivery. Human screening removed ambiguous or unanswerable prompts and corrected answer keys, leaving 186 recordings. Scoring uses normalized rules together with the same local Qwen3.6-35B-A3B judge [17] for cases requiring semantic matching. We treat this as a compact transfer benchmark for real human delivery, not as a substitute for a large-scale user study. Baselines and metrics. We compare Qwen-family controller stages: the base Qwen2.5-Omni-7B model [15], the SFT controller, and DAPO controllers trained with four, five, or six reward terms. The base model is reported in both offline and deployment modes to isolate the effect of the streaming controller protocol itself. The SFT controller provides the supervised initialization for the DAPO policy-optimization runs. The learned controller variants are evaluated in deployment mode, which is the setting they are trained to optimize. External baselines, including Audio Flamingo 3 [21], Audio Flamingo 3 + AF-Think [25], and GLM-4-Voice-9B [20], are evaluated only in offline mode because our benchmark harness cannot impose the same controller interface on third-party models without re-implementing their inference stacks. We also include the literature-reported Moshi rows from Shih et al. [31] for context. Our main latency metric is the length of the post-endpoint final-think state, used as the controllerlevel measure of residual deliberation after the user stops speaking. This matches the interactive concern behind the latency metric of Shih et al. [31], but Qwen2.5-Omni does not expose Moshi-style time-aligned monologue tokens. We treat cross-family token-latency comparisons as descriptive. The core latency claims compare Qwen streaming controllers under the same harness. Because full-prefix replay also incurs repeated prefix prefill cost, Appendix B.1 reports replay-harness real-time factor (RTF) for the full-prefix deployment audit.

5

Results

5.1

Synthetic spoken SRQA

Table 2 reports per-task accuracy on the six-task synthetic spoken reasoning benchmark, plus the row-weighted average and a token-level residual-output measure. The offline rows establish the strength of modern audio models under complete-audio evaluation. The controller rows isolate the question optimized by wait-think-answer post-training: under the same deployment protocol, can policy optimization improve the controller without changing the underlying audio model? Within the Qwen streaming-controller family, the SFT controller provides the initialization for DAPO policy optimization, while the DAPO rows compare reward-stack variants trained with four, five, and six reward terms. The SFT row should be read as protocol learning under token-level supervision, not direct task-reward optimization. The DAPO stage supplies the task-level reward that moves the controller back toward task-aligned behavior. The six-reward DAPO controller is the strongest accuracy row in this block and is second only to the complete-audio Qwen row in the full table. It improves the synthetic average from 67.6% for the base controller and 66.1% for the SFT controller to 70.3%, with gains on ARC-C, ARC-E, SIQA, PIQA, and GSM8K. It also reduces mean post-endpoint final-think length from 10.44 to 8.99 tokens relative to the base controller. Figure 3 visualizes the per-task accuracy comparison between the two controllers, and Appendix D reports the ablation summary across reward stacks. 7

Table 2: Results on synthetic spoken SRQA. Task columns report accuracy (%). Avg. is row-weighted accuracy. Final ↓ reports the mean length of the final post-audio <think>...</think> state when that span is produced by the evaluation prompt. For grey Moshi rows, the latency metric is copied from Shih et al. [31] and is not directly comparable to Qwen controller tokens. Bold and underlined values mark the best and second-best within the Qwen streaming controllers block, since completeaudio baselines and reported Moshi rows operate under different evaluation protocols. QC denotes question completeness, and DPO denotes direct preference optimization. ARC-E ARC-C SIQA PIQA GSM8K LLaMA-QS Avg. ↑ Final ↓

Method Complete-audio baselines Qwen2.5-Omni-7B Audio Flamingo 3 Audio Flamingo 3 + AF-Think GLM-4-Voice-9B

89.6 75.5 47.3 64.9

81.1 59.4 35.0 48.5

70.9 42.2 31.6 41.2

71.3 32.7 21.7 22.1

26.7 10.1 3.0 6.7

71.0 66.3 48.0 63.7

70.8 47.4 30.5 40.2

10.27 4.97 9.18 20.88

Reported streaming baselines from Shih et al. Moshi baseline 30.2 21.5 Moshi + CoT 77.7 59.8 Moshi + CoT w/o streaming ASR 55.8 44.0 Moshi QC-SFT 62.8 43.2 Moshi QC + length-DPO 65.4 46.0

22.8 56.1 50.1 45.1 45.3

23.8 56.9 46.3 40.7 46.0

8.7 16.1 12.2 13.8 14.7

42.8 57.8 59.9 56.2 56.9

23.4 56.6 44.8 44.4 46.7

– – – 52.42 19.31

Qwen streaming controllers Qwen2.5-Omni-7B SFT controller DAPO controller (4 rewards) DAPO controller (5 rewards) DAPO controller (6 rewards)

68.6 68.6 68.4 69.6 71.0

63.5 60.5 65.1 66.4 69.2

22.8 21.6 24.6 24.9 25.9

71.0 71.7 70.3 71.0 71.0

67.6 66.1 68.5 69.2 70.3

10.44 9.82 10.87 10.94 8.99

87.8 86.3 88.9 89.1 89.6

80.8 78.1 81.7 81.7 81.7

Base controller

Six-reward DAPO 87.8

ARC-E ARC-C 71.0 71.0

LLaMA-QS 68.6

SIQA 63.5

PIQA

71.0

69.2

60

70 22.8

GSM8K 20

89.6

80.8 81.7

22

80

90 25.9

24

26

28

Synthetic SRQA accuracy (%)

Figure 3: Per-task synthetic SRQA accuracy for the base controller and the six-reward DAPO controller. GSM8K is shown in a separate panel because its accuracy lies near 25%. 5.2

Real human-recorded audio

Table 3 evaluates the same controller family on the human-recorded real-audio benchmark. The SFT controller improves base accuracy by 4.8 points while staying close to the base final-think length. The DAPO variants occupy different points along the accuracy–residual-latency trade-off: the five-reward row has the strongest real-audio accuracy among DAPO variants, while the six-reward row has the shortest final-think length and is the only learned controller below the base. Real Audio Bench clarifies the scope of the claim: controller training transfers beyond TTS-rendered speech, and the SFT and DAPO rows expose different operating points along the accuracy–residuallatency trade-off. On this 186-item benchmark, the 95% bootstrap confidence intervals overlap substantially across learned controllers (Appendix D.1). We therefore treat Real Audio Bench as

8

Table 3: Results on Real Audio Bench. Accuracy is computed over 186 human-recorded recordings collected for this work. Final ↓ reports mean post-endpoint final-think length, and ∆ columns compare each controller with the base controller. Method

Training signal

Base controller SFT controller DAPO controller (4 rewards) DAPO controller (5 rewards) DAPO controller (6 rewards)

– SFT Ra + R f + R s + R u +Rt +Rt + Rc

Acc. ↑

∆Acc.

Final ↓

∆Final

64.0 68.8 65.6 67.7 65.1

– +4.8 +1.6 +3.7 +1.1

6.52 6.64 7.74 7.39 6.33

– +0.12 +1.22 +0.87 -0.19

a compact transfer check, not a fine-grained controller ranking benchmark. The primary accuracy– residual-latency claim comes from the larger synthetic SRQA evaluation. 5.3

Controller behavior

Policy optimization changes the trace-level operating point of the streaming system. The SFT controller initialization sits at 66.1% synthetic accuracy with 9.82 final-think tokens. The sixreward DAPO controller reaches 70.3% synthetic accuracy with 8.99 final-think tokens, the strongest accuracy and the shortest synthetic final-think length among the controller variants in Table 2. On Real Audio Bench, the controller family remains functional beyond TTS-rendered speech, but the best accuracy and shortest final-think points come from different learned variants (Table 3). These shifts show that wait-think-answer control learns where to place reasoning in the stream. 5.4

Reproducibility

Implementation details are documented in Sections 3–4 (formulation, training recipe, evaluation protocol) and Appendices B–D (hyperparameters, reward definitions, data-generation contract, and ablations).

6

Limitations

Real Audio Bench is a small five-speaker, 186-recording transfer benchmark; scaling to a broader user study across accents, environments, and interaction styles is the planned follow-up. Latency measurements are within-harness only, separating residual reasoning length (post-endpoint finalthink) from implementation runtime (RTF audit, Appendix B.1). Our implementation uses full-prefix replay, so the RTF audit measures the paper implementation rather than an ideal cache-native server that would forward the audio KV cache and avoid repeated prefill. Cache-native serving, larger real-audio validation, and a head-to-head implementation of question-completeness controllers in the same Qwen harness are useful future work; we do not claim a fully optimized production speech system. Broader impacts. Learning when a language model should think during streaming audio has implications beyond benchmark performance. Low-latency spoken reasoning supports accessibility tools, real-time captioning, on-device translation, and conversational tutoring, where response delay disproportionately harms users who rely on these aids. The same capability admits misuse: a controller reasoning over partial speech is a step toward systems that monitor or pre-empt conversations without user awareness, lowers the cost of real-time social engineering, and can produce asymmetric advantage against parties without comparable systems. Our design constrains the misuse surface: the action space is restricted to wait/think/answer with a hard format gate, and the reward explicitly penalizes spurious thinking, aligning the controller with non-interruptive conversational norms.

7

Conclusion

We formulate streaming speech reasoning as wait-think-answer control for LALMs: the model learns when to externalize intermediate reasoning during the audio stream, shifting the accuracy–residuallatency operating point without changing the base architecture. 9

References [1] Yi Su, Jisheng Bai, Qisheng Xu, Kele Xu, and Yong Dou. Audio-language models for audiocentric tasks: A systematic survey. arXiv preprint arXiv:2501.15177, 2025. [2] Chih-Kai Yang, Neo S. Ho, and Hung-yi Lee. Towards holistic evaluation of large audiolanguage models: A comprehensive survey. arXiv preprint arXiv:2505.15957, 2025. [3] Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. A simplest systematics for the organization of turn-taking for conversation. Language, 50(4):696–735, 1974. [4] Tanya Stivers, N. J. Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertie Hoymann, Federico Rossano, Jan Peter de Ruiter, Kyung-Eun Yoon, and Stephen C. Levinson. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Sciences, 106(26):10587–10592, 2009. [5] Stephen C. Levinson and Francisco Torreira. Timing in turn-taking and its implications for processing models of language. Frontiers in Psychology, 6:731, 2015. [6] Lilla Magyari, Jan Peter de Ruiter, and Stephen C. Levinson. Temporal preparation for speaking in question-answer sequences. Frontiers in Psychology, 8:211, 2017. [7] Sara Bögels, Lilla Magyari, and Stephen C. Levinson. Neural signatures of response planning occur midway through an incoming question in conversation. Scientific Reports, 5:12881, 2015. [8] Gregg A. Castellucci, Christopher K. Kovach, Matthew A. Howard III, Jeremy D. W. Greenlee, and Michael A. Long. A speech planning network for interactive language use. Nature, 602:117–122, 2022. [9] Greg J. Stephens, Lauren J. Silbert, and Uri Hasson. Speaker-listener neural coupling underlies successful communication. Proceedings of the National Academy of Sciences, 107(32):14425– 14430, 2010. [10] Alexandre Defossez, Laurent Mazare, Manu Orsini, Amelie Royer, Patrick Perez, Herve Jegou, Edouard Grave, and Neil Zeghidour. Moshi: A speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037, 2024. [11] Zhifei Xie and Changqiao Wu. Mini-Omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725, 2024. [12] Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-Omni: A smart and low latency speech-to-speech dialogue model with frozen LLM. arXiv preprint arXiv:2411.00774, 2024. [13] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-Audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. [14] Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-Audio technical report. arXiv preprint arXiv:2407.10759, 2024. [15] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-Omni technical report. arXiv preprint arXiv:2503.20215, 2025. [16] Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, and others. Qwen3-Omni technical report. arXiv preprint arXiv:2509.17765, 2025. [17] Qwen Team. Qwen3.6-35B-A3B: Agentic coding power, now open to all. Hugging Face model card, 2026. URL https://huggingface.co/Qwen/Qwen3.6-35B-A3B.

10

[18] Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, Xinyu Zhang, Pei Zhang, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. Qwen3-TTS technical report. arXiv preprint arXiv:2601.15621, 2026. [19] OpenAI. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. [20] Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. GLM-4-Voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02612, 2024. [21] Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, ChaoHan Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. Audio Flamingo 3: Advancing audio intelligence with fully open large audio language models. arXiv preprint arXiv:2507.08128, 2025. [22] Ziyang Ma, Zhuo Chen, Yuping Wang, Eng Siong Chng, and Xie Chen. Audio-CoT: Exploring chain-of-thought reasoning in large audio language model. arXiv preprint arXiv:2501.07246, 2025. [23] Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio-Reasoner: Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02318, 2025. [24] Shu Wu, Chenxing Li, Wenfu Wang, Hao Zhang, Hualei Wang, Meng Yu, and Dong Yu. AudioThinker: Guiding audio language model when and how to think via reinforcement learning. arXiv preprint arXiv:2508.08039, 2025. [25] Zhifeng Kong, Arushi Goel, Joao Felipe Santos, Sreyan Ghosh, Rafael Valle, Wei Ping, and Bryan Catanzaro. Audio Flamingo Sound-CoT technical report: Improving chain-of-thought reasoning in sound understanding. arXiv preprint arXiv:2508.11818, 2025. [26] Cheng Wen, Tingwei Guo, Shuaijiang Zhao, Wei Zou, and Xiangang Li. SARI: Structured audio reasoning via curriculum-guided reinforcement learning. arXiv preprint arXiv:2504.15900, 2025. [27] Gijs Wijngaard, Elia Formisano, Michele Esposito, and Michel Dumontier. AudSemThinker: Enhancing audio-language models through reasoning over semantics of sound. arXiv preprint arXiv:2505.14142, 2025. [28] Gang Li, Jizhong Liu, Heinrich Dinkel, Yadong Niu, Junbo Zhang, and Jian Luan. Reinforcement learning outperforms supervised fine-tuning: A case study on audio question answering. arXiv preprint arXiv:2503.11197, 2025. [29] Andrew Rouditchenko, Saurabhchand Bhati, Edson Araujo, Samuel Thomas, Hilde Kuehne, Rogerio Feris, and James Glass. Omni-R1: Do you really need audio to fine-tune your audio LLM? arXiv preprint arXiv:2505.09439, 2025. [30] Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Yuxin Li, Daijiao Liu, Yayue Deng, Donghang Wu, Jun Chen, Liang Zhao, Chengyuan Yao, Hexin Liu, Eng Siong Chng, Xuerui Yang, Xiangyu Zhang, Daxin Jiang, and Gang Yu. Step-Audio-R1 technical report. arXiv preprint arXiv:2511.15848, 2025. [31] Yi-Jen Shih, Desh Raj, Chunyang Wu, Wei Zhou, SK Bong, Yashesh Gaur, Jay Mahadeokar, Ozlem Kalinli, and Mike Seltzer. Can speech LLMs think while listening? In International Conference on Learning Representations, 2026. [32] Cheng-Han Chiang, Xiaofei Wang, Linjie Li, Chung-Ching Lin, Kevin Lin, Shujie Liu, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, and Lijuan Wang. STITCH: Simultaneous thinking and talking with chunked reasoning for spoken language models. In International Conference on Learning Representations, 2026. [33] Cheng-Han Chiang, Xiaofei Wang, Linjie Li, Chung-Ching Lin, Kevin Lin, Shujie Liu, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, and Lijuan Wang. SHANKS: Simultaneous hearing and thinking for spoken language models. arXiv preprint arXiv:2510.06917, 2025. 11

[34] Junlong Tong, Yingqi Fan, Anhao Zhao, Yunpu Ma, and Xiaoyu Shen. StreamingThinker: Large language models can think while reading. arXiv preprint arXiv:2510.17238, 2025. [35] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. [36] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. [37] Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Hong Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. SWIFT: A scalable lightWeight infrastructure for fine-tuning. arXiv preprint arXiv:2408.05517, 2024. [38] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [39] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, and others. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [40] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, and others. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. [41] Alex Graves, Santiago Fernandez, Faustino Gomez, and Jurgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the International Conference on Machine Learning, 2006. [42] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457, 2018. [43] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020. [44] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of EMNLP-IJCNLP, 2019. [45] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [46] Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using spectrogram-powered LLM. arXiv preprint arXiv:2305.15255, 2023.

12

A

Benchmark task and data details

The synthetic spoken SRQA benchmark contains six task families. ARC-Easy and ARC-Challenge are multiple-choice science questions. PIQA tests physical commonsense, SocialIQA tests social commonsense, GSM8K tests grade-school math, and LLaMA-QS tests short-answer factual question answering. The row counts in the reported benchmark are: 2376 ARC-Easy, 1172 ARC-Challenge, 1838 PIQA, 1954 SocialIQA, 1319 GSM8K, and 300 LLaMA-QS examples, for 8,959 total rows. All tasks are converted into spoken prompts. Multiple-choice tasks are rewritten so that the options are naturally spoken. GSM8K questions are kept close to the original wording where possible, with spoken-friendly normalization of notation. LLaMA-QS items are evaluated as short-answer factual questions. Each rendered audio file is evaluated under the same benchmark harness used by the controller rows. The real-audio benchmark contains 186 final rows after screening and correcting an initial 200recording candidate set. The construction process removed ambiguous or unanswerable items and corrected answer keys before the final benchmark pass. Qwen controller lanes, Qwen offline lanes, and the retained external offline baselines all completed on this same corrected evaluation set.

B

Training, inference, and compute details

We summarize the main training settings for SFT and DAPO. SFT uses MS-Swift [37] LoRA training [36]; DAPO uses a custom streaming controller trainer. Table 4: Training hyperparameters for supervised fine-tuning and DAPO controller optimization. Setting

Supervised fine-tuning

DAPO controller optimization

Base model Training framework Adaptation Target modules Precision Hardware Maximum length Learning rate Batching Schedule Optimizer Frozen modules Model selection Policy regularization Reward weights

Qwen2.5-Omni-7B initialized from base model weights MS-Swift 4.1.2 LoRA, rank 8, alpha 32, dropout 0.05 All linear layers bfloat16 4 NVIDIA B200 GPUs 8192 tokens 1 × 10−5 Per-device batch 4, gradient accumulation 2; effective batch 32 rows/update One-epoch SFT with cosine decay and warmup Fused AdamW, weight decay 0.1, betas 0.9/0.95 Audio encoder and aligner frozen; language model trainable Periodic validation using loss and token accuracy – –

Qwen2.5-Omni-7B initialized from the SFT LoRA adapter Custom streaming-controller trainer LoRA actor, rank 8, alpha 32 All linear layers bfloat16 4–5 NVIDIA B200 GPUs, depending on reward stack 8192-token rollout context; 48-token think and answer caps 4 × 10−7 actor learning rate 8 rollouts 1000 steps, 50 warmup steps AdamW actor update Same base frozen modules as controller LoRA training DAPO variants selected by validation behavior KL coefficient 0.01; asymmetric clipping 0.20 / 0.28 λa = 1.0, λf = 1.0, λs = 1.0, λu = 3.0, λt = 1.0, λc = 0.45

B.1

Replay runtime audit

Table 5 reports an implementation-level runtime audit for the reported full-prefix deployment harness. RTF is total controller wall-clock time divided by source-audio duration. These measurements include the repeated prefix replay used in the reported controller implementation, so they complement the final-think token metric and do not replace it. The SFT initialization controller is the supervised checkpoint used to initialize the DAPO run in this audit. Table 5: Replay-harness RTF for the full-prefix deployment audit. Lower is faster. Controller lane Base controller SFT controller DAPO controller

LLaMA-QS Real Audio ARC-C ARC-E SIQA PIQA GSM8K Mean 1.317 1.584 1.278

1.101 1.179 1.136

1.091 1.317 1.080

1.112 1.088 1.097 1.362 1.316 1.376 1.198 1.035 1.088

1.155 1.137 1.336 1.353 1.123 1.134

The DAPO controller in this audit is the reported six-reward deployment lane. Its replay-harness RTF is below the SFT controller and nearly matches the base controller on average. This is why the main text treats final-think length as a residual reasoning metric and reports RTF separately as an implementation diagnostic.

13

B.2

Stateful KV-cache controller prototype

To check whether the same wait-think-answer information flow can be implemented cache-natively, we built an experimental stateful controller. Audio chunks are appended into a single persistent Qwen2.5-Omni thinker KV cache. Each controller decision is generated from a temporary fork of that persistent state. <wait/> actions are not committed back into the persistent text cache; <think>...</think> updates are normalized and appended back; the final-think and answer turns are generated from the full cached audio plus all committed thoughts. The prototype runs at the level of model.forward(..., past_key_values=..., use_cache=True) with a manual short-token decode loop, instead of the high-level generate() calls used by the production benchmark path. Mechanism checks on a four-second real-audio sample confirm that cache lengths match attention lengths under both 2.0s and 0.5s chunking, and that wait actions correctly do not enter the persistent cache. These are wiring-level smoke runs, not benchmark scores; turning the prototype into a score-producing batch runner remains future work. We treat the prototype as evidence that the same wait-think-answer information flow can be implemented in a cache-native deployment without changing the controller’s training-time semantics. B.3

SFT and DAPO training details

The audio-only cold-start SFT run shows decreasing training loss and increasing token accuracy during training (Figure 4). DAPO update points are chosen based on validation behavior, since the streaming reward is non-monotonic across policy updates. For the controller tick sweep, 0.5s, 1.0s, and 1.5s grids achieved validation token accuracies of 0.6108, 0.6105, and 0.6102, respectively; we use 0.5s because accuracy was tied while timing resolution was finer. Qwen2.5-Omni calls are still padded or extended to the 2.0s minimum audio window required by the model, so the sweep changes the reconsideration cadence while leaving the minimum audio exposure fixed. SFT loss

SFT token accuracy train eval

0.75 Accuracy

Loss

1.5

1.0

0.70 0.65

train eval

0.5 0

5000

10000

15000

20000

0

Global step

5000

10000

15000

20000

Global step

Figure 4: SFT training curves from the audio-only cold-start run. Training and validation metrics improve during supervised fine-tuning, providing the initialization for policy optimization. For DAPO, each prompt is sampled into a group of G rollouts. We compute group-relative advantages PG 1 Ri − G j=1 Rj Ai = , std(R1 , . . . , RG ) + ϵ and apply token-level clipped updates over the completion-token mask M . With yi,t denoting the t-th generated token in rollout i, oi,t its token-level context, and ri,t (θ) = exp(log πθ (yi,t | oi,t ) − log πold (yi,t | oi,t )), the objective is X min(ri,t Ai , clip(ri,t , 1 − ϵl , 1 + ϵh )Ai ), Ai ≥ 0, 1 LDAPO = − (4) max(ri,t Ai , clip(ri,t , 1 − ϵl , 1 + ϵh )Ai ), Ai < 0. |M | (i,t)∈M

Before a DAPO update, sampled groups must contain enough format-valid rollouts, at least one valid final-think rollout, and at least one valid pre-endpoint thought. Groups that collapse to all-wait or malformed final-think behavior are resampled up to a fixed retry budget; persistent failures are skipped and logged. Table 6 gives the operational definitions used by the reward implementation.

14

Table 6: Operational definitions of reward terms. Positive signals define the intended controller behavior, while failure modes describe behavior penalized during DAPO. Term

Positive signal

Ra

Correct final answer after deterministic task scoring; local judge Fluent but incorrect final answers. scoring is used only as a semantic-equivalence fallback for open short-answer cases. Valid wait-think-answer format: wait/think while listening, one Early answers, malformed tags, missing final-think turns, final think, and answer only after the utterance ends. or invalid action order. Short post-endpoint final-think state once the audio stream is Repeating reasoning after the endpoint that should have complete. been resolved during listening. Intermediate thoughts placed near answer-relevant evidence up- Missing an important state change or interrupting the dates in the audio. stream with irrelevant thoughts. Local judge preference for short, concrete, answer-supporting Generic, verbose, or meta-level thoughts that do not semantic states. update the answer state. Local judge preference for thought chains that support the final Internally inconsistent chains or unsupported jumps to answer when the answer is correct. the final answer.

Rf Rs Ru Rt Rc

Failure mode discouraged

Reward implementation details. The six terms above are the conceptual reward categories; the implementation adds only local guardrails inside these categories. For Ra , deterministic scoring is used whenever possible: answers are normalized, multiple-choice labels are mapped to option text when possible, and numeric answers are compared after lightweight numeric normalization. Local judge scoring is used only as a semantic-equivalence fallback for open short-answer cases where exact matching is insufficient. The same term uses a difficulty-aware effort calibration, so a correct answer is not rewarded equally when the controller uses an implausibly empty or unnecessarily verbose reasoning state. For Rs , the measured quantity is the post-endpoint final-think length. The runtime token count is used when available; otherwise the reward path falls back to a character-based token estimate. The reported configuration gives the final-think state a six-token free budget and then applies a linear penalty with coefficient 0.30, capped at 3.0. Correct and well-formed trajectories can receive a small bonus when the final-think state is a compact answer cue of three to six tokens; this bonus is disabled when the answer shape is invalid. The answer-shape guardrail penalizes question-form answers, label-only answers when a semantic answer is expected, and yes/no type mismatches. For Ru , pre-endpoint thoughts are matched to answer-relevant evidence-update ticks derived from the controller metadata. Matching uses a tolerance of two 0.5s ticks and includes sparsity pressure, so the controller is penalized both for missing important updates and for inserting thoughts at nearly every tick. The judge-assisted terms use a local Qwen3.6-35B-A3B endpoint [17]. Rt scores emitted thoughts as local semantic state updates, and Rc scores whether the visible thought chain supports the final answer. Judge outputs are mapped to coarse scores in {0, 0.5, 1} and are used only as reward signals; the full judge prompts are part of the released training code rather than the main paper text. Reward weight selection. The reward weights in Table 4 were selected through validation-set sweeps during method development. Among the soft-coefficient terms, the update-timing weight λu and the latency weight λs had the largest qualitative effect on the accuracy–residual-latency operating point because they directly shape when and how often the controller emits visible reasoning state. The correctness, protocol, thought-quality, and chain-consistency weights (λa , λf , λt , λc ) were comparatively less sensitive in the explored range, with λf acting mainly as a protocol gate once it exceeded the threshold needed for valid action format and λt , λc providing soft preferences over judge-scored signals. The reported configuration corresponds to the validation-best operating point under the synthetic SRQA setup.

C

Prompts, alignment, and data construction notes

The canonical synthetic record contains the fields question_text, tts_text, tts_instruct, transcript_text, anchor_words, logical_actions, and final_answer. Here question_text is the canonical task question, tts_text is the spoken surface form passed to the TTS renderer, tts_instruct is the delivery/style control prompt, transcript_text is the normalized transcript aligned to the waveform, anchor_words are lexical anchors used to localize answer-relevant state changes, logical_actions is the semantic wait-think-answer trace before serialization, and final_answer is the gold answer string. The teacher prompt requires 15

spoken-friendly wording, avoids unnatural written math notation where possible, and asks for thoughts that preserve short semantic state instead of generic summaries, tone descriptions, or pause descriptions. Acoustic style is controlled through tts_instruct; it should influence the rendered speech, not become the content of the reasoning state unless it changes the answer. Full spoken inputs are rendered once. We do not synthesize each controller segment independently, because segment-wise synthesis would inject artificial prosodic breaks exactly where the model is supposed to learn natural timing. Forced alignment maps transcript words to audio timestamps, and action anchors are snapped upward to the 0.5s controller grid. The supervised export and the DAPO export are both generated from this aligned representation. C.1

Synthetic data generation prompt

The project-specific synthetic data prompt is implemented in the data-generation code and mirrored in the reviewer-facing prompt document. The core GPT-4o [19] contract is: System: You create high-quality English training records for a streaming speech reasoning model. Return one JSON object only. The spoken script is always user speech only. Do not solve the task or speak as the assistant inside tts_text or transcript_text. Do not use inline TTS markup such as <break>. Use punctuation and a separate tts_instruct field to induce natural speech style when needed. transcript_text must be clean English alignment text. logical_actions must use only <wait/>, <think>...</think>, and <answer>...</answer>. Some anchors may map to <wait/> because a physical pause or filler does not necessarily require thinking. Every <think> should be short, incremental, grounded in the current evidence, and useful for the answer state. Prefer concrete facts, constraints, eliminations, quantities, corrections, answer-type constraints, or current candidate state. Avoid generic meta commentary and avoid long explanations. Every sample must be unique. Treat the scenario seed only as a starting situation and vary names, numbers, locations, items, and wording. Do not use LaTeX, bullet lists, stage directions, or duplex terminology. User template: Create one English training example. sample_id: {sample_id} task_family: {task_family} topic: {topic} verifiability: {verifiability} task_type: {task_type} difficulty: {difficulty} variant_index: {variant_index} scenario_seed: {scenario_seed} Target about {pause_count} meaningful anchor boundaries. {family_guidance} {difficulty_guidance} Return JSON fields: task_type, topic, verifiable, difficulty, question_text, tts_text, tts_instruct, transcript_text, anchor_words, logical_actions, final_answer, difficulty_metadata. Key requirements: - tts_text and transcript_text contain only user speech. - anchor_words are copied from transcript_text. - logical_actions has one key per anchor plus anchor_AUDIO_END. - non-final anchors use <wait/> or <think>...</think>, never <answer>. - anchor_AUDIO_END contains one final <think>...</think> followed by one <answer>...</answer>. - the answer text exactly matches final_answer.

16

C.2

Spoken SRQA prompt provenance

For the six synthetic SRQA benchmark families, we follow the spoken-rewrite prompt contract from Shih et al. [31]. The implementation uses their task-specific prompts for ARC, PIQA, SocialIQA, and GSM8K style conversion: preserve all information needed for the answer, verbalize choices inside the spoken question, and return a JSON object containing the converted spoken question and, where applicable, the converted answer. We cite their prompt without reproducing it as a new project-specific prompt. C.3

Real Audio Bench generation prompt

Real Audio Bench uses a separate GPT-4o candidate-generation prompt because its goal is real human delivery, not TTS rendering. The source generation spec fixes 200 planned items, five speakers, and 40 recordings per speaker, then filters the recorded set to 186 final rows. The universal prompt is: System: You are writing candidate items for a real-human benchmark for streaming spoken reasoning. This benchmark is NOT ordinary QA. It is for a streaming controller that hears audio over time and must wait until a late decisive cue arrives. Hard constraints: - Write natural spoken English only. - The utterance must sound like something a person would actually say aloud to an assistant. - Every item must be verifiable from the utterance itself. - Each item must have exactly one preferred gold answer. - The spoken utterance must contain an early plausible answer that later becomes wrong, incomplete, or premature. - A late cue must overwrite or finalize the answer. - Avoid textbook math, school trivia, science, history, and canned benchmark style. - Keep spoken duration within 8 to 18 seconds. - Keep the answer short and cleanly scoreable. - Across a batch, diversify the reasoning mechanism. User template: Generate {count} candidate items. Coverage requirements: - use the requested categories, scenario families, difficulties, and reasoning mechanisms as evenly as possible - all items must be verifiable - prioritize distinctness from the existing synthetic corpus - every item must pass a uniqueness test: after reading the full utterance, there should be exactly one reasonable gold answer - when using time windows, thresholds, eligibility, or scheduling conflicts, state the deciding rule explicitly Allowed categories: mid_way_reversal, adversarial_pause, incremental_accumulation, noisy_environment, fast_speech_hesitation, real_world_situational Allowed scenario families: day_of_travel_logistics, home_landlord_appliance, workplace_school_admin_it, food_shopping_returns, health_pharmacy_finance_lite Allowed reasoning mechanisms: overwrite_final_slot, cumulative_total, tiered_discount_total, fee_or_threshold_decision, exclusion_choice, bounded_window_selection, quantity_update, eligibility_decision, refund_or_credit_total, schedule_window_resolution Return one JSON object with an items array. Each item has: item_id, category, difficulty, scenario_family, reasoning_mechanism,

17

verifiability, question_spoken, gold_answer, answer_check_type, why_unique_answer, ambiguity_check, why_early_answer_would_be_wrong, critical_audio_cue, speaker_notes, estimated_duration_sec, acoustic_target.

After generation, an LLM verifier and human audit screened the candidate pool for a unique answer, spoken naturalness, reasoning strength, and distinctness. The final manifest keeps 186 recorded items, drops 14 ambiguous or unsuitable items, and applies 23 audited answer-key corrections.

D

Additional results and ablations

Table 7 summarizes the Qwen-family controller variants used in the main results table, separating the SFT controller from the four-, five-, and six-reward DAPO controllers on synthetic SRQA accuracy and mean final-think length. Table 7: Ablation summary for Qwen controller variants on synthetic SRQA. The table reports the base controller, the SFT controller, and DAPO controllers trained with four, five, or six reward terms.

D.1

Syn. acc. ↑ Syn. final ↓

Variant

Reward stack

Base controller SFT controller DAPO controller (4 rewards) DAPO controller (5 rewards) DAPO controller (6 rewards)

– SFT Ra + Rf + Rs + Ru Ra + Rf + Rs + Ru + Rt Ra + Rf + Rs + Ru + Rt + Rc

67.6 66.1 68.5 69.2 70.3

10.44 9.82 10.87 10.94 8.99

Real Audio Bench bootstrap confidence intervals

Real Audio Bench contains 186 human-recorded items (Section 4). To quantify the uncertainty introduced by this small evaluation size, Table 8 reports per-lane accuracy with 95% bootstrap confidence intervals over 10,000 resamples of the 186 items. The lanes match the controller variants reported in Table 3. Table 8: Real Audio Bench accuracy with 95% bootstrap confidence intervals (10,000 resamples). The lanes match the controller variants reported in Table 3. Lane Base controller SFT controller DAPO controller (4 rewards) DAPO controller (5 rewards) DAPO controller (6 rewards)

Correct/Total

Acc.

95% bootstrap CI

119/186 128/186 122/186 126/186 121/186

0.640 0.688 0.656 0.677 0.651

[0.570, 0.710] [0.618, 0.753] [0.586, 0.726] [0.608, 0.742] [0.581, 0.720]

The 95% confidence intervals overlap substantially across all controller lanes, including between the base controller and the post-trained variants. We therefore treat Real Audio Bench primarily as a transfer check, not a fine-grained controller ranking benchmark. Per-controller synthetic numbers are summarized in Table 7.

E

Asset licenses

Table 9 summarizes the public licenses of the upstream assets used in this work. All assets are accessed through their official release channels and used within their published terms. License or terms information was checked against each asset’s official model card, dataset card, repository, LICENSE file, or source documentation where available, at the time of submission.

18

Table 9: Public licenses or terms of use for upstream assets used in this work. Asset

Type

License

Qwen2.5-Omni-7B [15] Qwen3-TTS [18] Qwen3.6-35B-A3B [17] GLM-4-Voice-9B [20] Audio Flamingo 3 [21] GPT-4o [19] MS-Swift [37] ARC [42] PIQA [43] SocialIQA [44] GSM8K [45] LLaMA-Questions/Spectron [46]

Model Model Model Model Model API Software Dataset Dataset Dataset Dataset Dataset

Apache 2.0 Apache 2.0 Apache 2.0 GLM-4 Model License Agreement; code under Apache 2.0 NVIDIA OneWay Noncommercial License OpenAI API terms of service Apache 2.0 CC-BY-SA 4.0 Academic Free License 3.0 CC-BY 4.0 MIT Source paper / archived GitHub terms; license not explicitly specified

19