Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors Michael Finkelson1,2 Daniel Segal1 Eitan Richardson1 Shahar Armon1 Nani Goldring1 Poriya Panet1 Nir Zabari1 Benjamin Brazowski1 Or Patashnik2 Yoav HaCohen1
arXiv:2606.19325v1 [cs.SD] 17 Jun 2026
1
Reference Speaker 1
2
Lightricks
Tel Aviv University
Reference Speaker 2
Soft piano plays gently in the background. The speaker from reference 1 says quickly: “Hi, I'm Philipp!” The speaker from reference 2 says quickly: “I'm Grant!” Then both speakers shout together at the exact same moment, their two voices overlapping in perfect unison as one single voice: “Welcome to the channel!”
Natural Language Prompt
ScenA
Figure 1: Our S CENA framework transforms free-form natural language prompts and a set of reference voices into rich, multi-speaker conversational scenes. The prompt alone determines which reference speaks where, with no per-turn tags, transcripts, or identity encoders. This natural language interface enables complex human interactions, including overlapping speech, spontaneous paralinguistic events, and scene-level ambient sound.
Abstract Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings. These systems operate within speech-only pipelines that produce clean vocal sequences without the ambient texture of real conversations. We take a different approach. Our method, S CENA, conditions a text-to-audio flow-matching foundation model, pretrained on large-scale in-the-wild data, directly on multiple reference voices and a free-form natural language prompt that describes an entire multi-speaker audio scene. Leveraging such a foundational model allows us to inherit its capacity for natural, non-studio audio: background noise, room acoustics, overlapping dialogue, and spontaneous paralinguistic events, while adding multi-speaker control without any per-turn structure. Concretely, reference latents are concatenated into the model’s token sequence and distinguished by lightweight identity-aware positional encodings. However, we identify a critical obstacle to this approach: the Reference Shortcut. During training under standard noise schedules, the model can identify the matching reference by acoustic similarity to the noisy target, bypassing the text prompt entirely. We address this with a high-noise-biased timestep distribution that forces the model to rely on the text prompt for speaker assignment. We evaluate S CENA on the CoVoMix2-Dialogue benchmark, showing that it outperforms existing multi-speaker systems on speaker-binding metrics while generating rich conversational audio with overlapping speech, emotional vocalizations, and ambient sound. Our results demonstrate the advantage of using a general-purpose audio model conditioned on a free-form scene description, rather than passing structured dialog scripts through a speech-only pipeline. Project page: https://finmickey.github.io/scena/
1
Introduction
A real multi-speaker conversation is more than ordered turns of speech: voices overlap, laughter cuts in, and the room itself shapes how it all sounds. However, existing speech generation systems capture little of this. Zero-shot voice cloning models address only the single-speaker case [39, 9, 6, 3], leaving multi-speaker conversations to be assembled segment-by-segment outside the model. Recent dialogue-TTS systems generate multi-turn conversations directly [55, 33, 56, 54], but bind speakers to utterances through structured supervision (per-turn speaker tags, multi-stream transcriptions, or learnable speaker-turn embeddings). All are speech-only pipelines, producing clean vocal tracks stripped of the ambient texture real conversations carry. We take a different starting point with S CENA, our flow-matching framework for multi-speaker audio scene generation. Flow-matching [27, 29] text-to-audio foundation models [13], pretrained on large-scale in-the-wild audio, already capture the texture of natural sound scenes. What they lack is a way to tie specific voices to specific roles. S CENA adds this capability with a deliberately minimal interface: reference latents are concatenated with the input latents and distinguished by lightweight identity-aware positional encodings. A single free-form natural language prompt describes the entire scene: who speaks, what is said, and what else is in the room. No per-turn tags, no multi-stream transcripts, and no identity encoders or reference-side adapters [50, 41]; the prompt alone determines which reference speaks where (Figure 1). Our experiments show that naïvely training this design fails to learn which reference speaks where, due to a previously unrecognized failure mode of reference-conditioned flow matching that we call the reference shortcut. We find that under the standard logit-normal timestep distribution [12], the noised target retains enough acoustic information for the model to pick the matching reference by similarity. This shortcut sidesteps the text prompt entirely, and yields low training loss but catastrophic inference. At test time, generation begins from pure noise, where the shortcut is unavailable and text is the only signal that can resolve which voice goes where. Text, however, is precisely the signal the model has learned to ignore. To locate the noise levels where the shortcut works, we train a small probe on frozen audio features and find that it can match references to targets by similarity across the entire low-to-moderate noise range — exactly where standard training concentrates. A noiseschedule ablation then shows that binding-aware metrics improve monotonically as we shift training mass toward higher noise. We close this shortcut with a high-noise-biased timestep distribution, a Beta+Uniform mixture that concentrates training on noise levels where the target is uninformative and text is the only binding signal. We evaluate our method on the public C OVO M IX 2-D IALOGUE benchmark, where S CENA matches or surpasses current multi-speaker dialog baselines on every speaker-binding metric. The advantage widens on a harder in-the-wild reference subset, where studio-clean references give way to noisy real-world recordings. Beyond binding, S CENA generates overlapping dialogue, spontaneous paralinguistic events (laughter, sighs, breaths), and scene-level ambient sound jointly with the conversation. We encourage readers to visit our project page for the full spectrum of capabilities beyond two-speaker dialogue.
2
Related Work
Multi-speaker speech generation. Zero-shot voice cloning has converged on two architectural families. Autoregressive zero-shot TTS systems (VALL-E [39, 5], Seed-TTS [1], Spark-TTS [42], CosyVoice 2/3 [9, 10], VoiceStar [32], XTTS [3], MiniMax-Speech [52]) autoregressively generate discrete speech tokens from a single speaker reference clip. Flow-matching and diffusion generators (Voicebox [23], E2-TTS [11], F5-TTS [6], NaturalSpeech 3 [19], MegaTTS 3 [18], ZipVoice [57], StyleTTS 2 [25]) produce mel-spectrograms or latents in a single non-autoregressive pass. Both families condition on a single reference; multi-speaker conversations are obtained post hoc by synthesizing each speaker’s segments independently and concatenating, an arrangement that is incompatible with overlapping speech, shared acoustic environments, and scene-level descriptions. A more recent wave of dialog-TTS systems generates multi-turn conversations directly [20, 53, 54, 56, 33, 49, 48, 47, 55, 51, 30], but binds speakers to utterances through structured supervision: per-turn speaker tags (e.g., [S1]/[S2]), multi-stream transcriptions, learnable speaker-turn embeddings, or LLM-generated annotations.
2
Reference-conditioned generation. Beyond zero-shot TTS, reference conditioning has been studied across audio generation more broadly. Audiobox [38] is closest to our setting: a flow-matching audio model that conditions on a text caption together with a single voice prompt to jointly synthesise speech, sound, and music. MusicGen [7] pairs text with a melody reference for music generation; AudioLDM 2 [28] conditions sound generation on text together with an audio prompt. All of these condition on a single reference. We are, to our knowledge, the first to address the multi-reference audio setting where natural language alone determines speaker assignment. Parallel work in the image and video domains is informative for our design choices. IP-Adapter [50] introduces decoupled cross-attention dedicated to image references; InstantID [40] and PhotoMaker [26] use face-specific encoders for identity preservation. Multi-subject methods rely on additional structure beyond text: bounding-box layouts in MS-Diffusion [41], segmentation maps in MuDI [17], localized cross-attention at training and image-augmented prompt tokens at inference in FastComposer [45], and region-aware masked guidance for video in MAGREF [8]. Closer in spirit to our setup, three image-domain methods condition transformers on references in context. OmniGen [46] replaces each <|image_k|> marker in the prompt with that reference image’s tokens, producing a single sequence that interleaves text and image references. UNO [44] concatenates references in the attention sequence with offset rotary positions, and In-Context LoRA [16] stitches references and target into a composite image; both bind references by matching descriptive subject names in the prompt (the toy, the man with blond hair) to the reference list. This requires references to be separately describable in language, which fails when they share content, as is typical in a multi-speaker dialog. S CENA binds references through indexed mentions (reference 1, reference 2) that need not describe their content, and ties them to dynamic spans of a jointly generated multi-speaker output. This setting exposes a flow-matching-specific failure mode, the reference shortcut (§3.2), in which the model can bypass the prompt by matching references against the noised target. We close it by modifying the timestep distribution alone, rather than by adding an explicit binding mechanism. Timestep distributions and noise schedules. The choice of noise schedule has a large effect on diffusion-model training. EDM [21] proposes log-normal sigma sampling for image generation; Stable Diffusion 3 [12] introduces the logit-normal timestep distribution for flow matching, with a mild upward shift at higher resolutions; Min-SNR weighting [14] reweights the loss by signal-to-noise ratio; and Simple Diffusion [15] shifts schedules toward higher noise for high-resolution images. All of these choices are motivated by generation quality or training stability.
3
Method
We describe our approach in four parts: the reference conditioning mechanism (§3.1), the reference shortcut and its analysis (§3.2), the timestep distribution that eliminates it (§3.3), and the multireference dataset on which we train (§3.4). Figure 2 provides an overview. Noise Prediction
"The speaker from reference 2 says, 'This area is growing fast, and opportunities are running out.' The speaker from reference 1 replies, 'Yeah, the location and design really stand out.' Upbeat electronic music plays in the background."
Pos. Enc.
Text Instruction
DiT R1
R1
R1
R1
R2
R2
R2
R2
+
+
+
+
+
+
+
+
References Tokens
Noisy Target Tokens
Figure 2: Our framework utilizes a DiT to synthesize reference-driven conversational scenes. Reference latents (red, blue) are concatenated with noisy target tokens, and identity-aware positional encodings (R1, R2) distinguish speakers. Text conditions the model via cross-attention, and training with a flow-matching loss enables transformation into high-fidelity conversational audio.
3
3.1
Reference Conditioning via Latent Concatenation
Setup. We build on a pretrained audio diffusion transformer [13] that operates on a sequence of latent tokens. A target audio clip is encoded into a latent sequence z0 ∈ RN ×D by the backbone’s variational autoencoder (VAE), where N is the number of tokens and D is the latent dimension. A text prompt c is encoded by a frozen text encoder and conditions generation through cross-attention; reference clips are encoded by the same VAE as the target. Reference injection. Given K ≤ Kmax reference speaker clips {r1 , . . . , rK }, each is encoded into a latent sequence rk ∈ RNk ×D . The full input to the transformer is the concatenation: zinput = [zt ; r1 ; . . . ; rK ],
(1)
where zt is the noised target at timestep t. Only the target is noised; the reference latents rk are passed clean. All tokens participate in the transformer’s self-attention, allowing the model to attend freely between target and reference tokens. Positional encoding. To distinguish the references from one another and from the target, we associate each reference slot k with a learned embedding ek ∈ RD , broadcast across the Nk tokens of rk and added immediately after the linear projection into the transformer hidden dimension: rk ← rk + ek ,
k = 1, . . . , Kmax ,
(2)
while the target zt receives no additive embedding. This adds only a negligible number of parameters and leaves the rest of the architecture unchanged. We ablate the choice of slot encoding in Appendix A.3, comparing this additive embedding against a RoPE-based alternative and a no-positional baseline. Generation through natural language. A single text prompt drives the entire generation. It describes the scene holistically (ambient sounds, speaker turns, content, and affect) and refers to each reference voice by an ordinary textual mention such as reference 1. For example: “Ocean waves crash gently on the shore. Seagulls call in the distance. The speaker from reference 1 takes a deep breath and says: ‘This is exactly what I needed today.’ The speaker from reference 2 hums in agreement: ‘Yeah, no emails out here.’ Another wave rolls in slowly.” This prompt, together with the concatenated reference latents (Eq. 1) and their slot embeddings (Eq. 2), is the entire input to the model. No special tokens, no identity-preserving adapters [50], no per-segment transcripts, and no spatial or temporal supervision [41, 17] are used. The model produces the entire scene in a single forward pass, including overlapping speech, natural turn-taking, and scene-level ambient sound. Existing multi-speaker systems typically achieve these qualities only through external structure or post-hoc concatenation. In S CENA, binding is delegated entirely to training, which, as we show next, requires careful design of the timestep distribution. 3.2
The Reference Shortcut Challenge
Flow matching background. In flow matching [27, 29], training constructs a noised sample zt = (1 − t)z0 + tϵ at timestep t ∈ [0, 1], where ϵ ∼ N (0, I). The model fθ is trained to predict the velocity field v = ϵ − z0 that transports noise to data. The training objective is: L = Et∼p(t), z0 , ϵ ∥fθ (zt , t, c) − v∥2 , (3) where p(t) is the timestep distribution. Standard practice [12] uses a logit-normal distribution t ∼ σ(N (µ, s2 )), often with a mild shift of µ toward higher noise. In all such variants, the bulk of training mass remains concentrated on intermediate timesteps, where the denoising task is most informative for generation quality.
4
The shortcut mechanism. We now describe a shortcut available in our setup that, if not addressed, lets the model bypass the text prompt entirely. Binding a reference through the prompt requires the model to compose information across two attention paths. Through cross-attention, it must associate the phrase “reference k” in the text with the slot embedding ek carried by the k-th reference’s tokens. Then, through self-attention, the target tokens must locate the reference bearing ek and route their queries toward it. By contrast, the shortcut requires nothing more than what self-attention already does by default: when the noised target still carries acoustic traces of z0 , its tokens are most similar to the matching reference’s, and self-attention routes them there in a single step, with no contribution from the text path. Both routes minimize the training loss; the optimizer favors the simpler one whenever it is available. Consider a training example with target z0 , references {r1 , . . . , rK }, and text prompt c mentioning “reference k”. At timestep t, the noised target is zt = (1 − t)z0 + tϵ, interpolating between the clean target at t = 0 and pure noise at t = 1. The shortcut is available whenever each speaker’s segment of zt remains discriminable from non-matching references; if this holds across most of [0, 1], the prompt is rarely needed during training, and the model has little reason to learn the two-path routing. We quantify this with a probe that measures, at each t, whether zt still carries enough of z0 to identify its source reference. The probe is a binary classifier that mirrors the implicit denoising-time choice: given zt and two candidate references (one matching, one from a different sample), predict the match. We use the first 8 of the 48 transformer blocks of our backbone with a small classification head; full setup is in Appendix A.1.
probe accuracy (%)
Probe accuracy vs target noise
100 99% 98% 97% 96% 95% 93% 92% 91% 90% 88% 87% 85% 83% 81% 78% 75% 71% 67% 100% 99% 99% 98% 97% 96% 94% 92% 90% 87% 83% 79% 75% 70% 65% 62% 56% 59% 52% 45% 95 38% 30% 49% 41% 22% 90 15% 32% 85 8% 20% 80 3% 75 7% 70 0% 65 60 Probe accuracy Beta(5,1)+uniform 55 Shifted logitnormal 50 0.0
0.1
0.2
0.3
0.4 0.5 0.6 target noise level t
0.7
0.8
0.9
1.0
Figure 3: Reference-shortcut probe (§A.1). Blue: probe binary-classification accuracy on the held-out set, measured at 50 evenly spaced target noise levels t; the gray dashed line marks chance (50%). Light curves show two training-time timestep distributions on the same t axis—our Beta+Uniform mixture (red) and a logit-normal distribution centered around t = 0.8 (green)—each independently rescaled to a fixed peak height. The small numbers above (red) and below (green) each blue marker give the fraction of training samples drawn at or above that noise level under each distribution, i.e. P[T ≥ t].
Accuracy stays at ≥ 98% for t ≤ 0.58, remains ≥ 90% through t ≈ 0.86, and is still 75% at t = 0.96. Only as t approaches 1 does it collapse to chance. Standard logit-normal distributions and their mildly shifted variants place the majority of training mass squarely in this high-accuracy regime, making the shortcut the easiest strategy for reducing the training loss. At inference, generation starts from t = 1 (pure noise) and iteratively denoises. The initial denoising steps, which establish the global structure of the output (including which speaker speaks where), occur at high t, where the shortcut is unavailable. The model, however, has not learned to use text either. The result is a model that fails to bind references to text, producing outputs where speakers are confused, ignored, or applied inconsistently. 5
3.3
High-Noise-Biased Timestep Distribution
A naïve remedy is to re-center the logit-normal further toward t = 1. The probe suggests this is insufficient. The regime where the shortcut is solvable is wide enough that any reasonable shift still leaves the bulk of training mass inside it. Pushing further into a narrow high-noise band fixes the shortcut but starves the model of the lower-noise signal it still needs for fine-detail synthesis. We therefore replace the logit-normal with a mixture: p(t) = (1 − λ) Beta(t; α, 1) + λ Uniform(t; ϵ, 1).
(4)
The Beta(α, 1) component, with density αtα−1 , pushes substantial mass into the high-noise tail where the shortcut is least available; the Uniform component retains coverage across the rest of the range. 3.4
Multi-Reference Dataset
We construct a multi-reference training dataset in which each example consists of a target clip, K reference clips, and a caption describing how the full audio scene of the target is related to the references. Each target is a multi-speaker conversational audio clip containing both dialog and non-speech sounds, while each reference is a distinct single-speaker clip. This structure enables supervision in which a single natural-language description specifies how multiple reference speakers are realized within a shared conversational scene. To construct this dataset, we employ a multi-stage pipeline comprising the following components: Matching references to target speakers. For each audio clip, we extract speaker embeddings for each segment using a diarization pipeline. We then use embedding similarity to search for a separate reference clip in which each speaker appears. If a reference clip is found for every speaker, the original clip is defined as a target clip. This process ensures alignment between the speakers in the target clip and its references. Intermediate target caption. We first caption the target clip independently, without conditioning on the references. This intermediate caption provides a reliable description of the scene and reduces hallucinations compared to directly generating a reference-conditioned caption. It serves as a grounding signal for the final caption generation step. Multi-reference caption generation. Finally, we generate a caption for the target conversational scene conditioned on the reference clips. The caption describes both the dialog and non-speech sounds, while grounding speaker identities in the references. The captioning prompt is constructed by combining the reference clips, the intermediate target caption, and speaker-level timestamps obtained from the diarization process. The intermediate caption provides a grounded description of the scene, reducing hallucinations, while the timestamps, aligned with the corresponding reference speakers, provide explicit temporal supervision over speaker turns. Together, these signals improve speaker attribution and help the captioner produce captions that faithfully describe the full conversational scene.
4
Experiments
4.1
Experimental Setup
Audio-only backbone. We adapt LTX-2 [13], a dual-stream audio-video diffusion transformer with separate audio and video streams coupled by bidirectional cross-modal attention. We use the LTX-2.3 release, retain only the audio stream, and remove all video-to-audio cross-attention layers, yielding a standalone audio-only diffusion transformer. The variational autoencoder (VAE), text encoder, and prompt-embedding adapter of LTX-2 are reused without modification; references are encoded by the same VAE as the target. The model supports audio durations of up to 20 s.
6
Training. We fine-tune the backbone to consume up to Kmax = 3 speaker references per training example. Reference clips can be of any length up to the model’s 20 s maximum, with only the target noised during training (§3.1). Training proceeds for 20,000 steps at a global batch size of 128 on 16× NVIDIA GB200 GPUs, taking approximately 24 hours. We optimize with AdamW (β1 =0.9, β2 =0.95, ϵ=10−8 , weight decay 0.01) at a peak learning rate of 1×10−4 , reached via a 1,000-step linear warmup and held constant thereafter. We maintain an exponential moving average (EMA) of the parameters with decay 0.9999. The training objective is the rectified-flow velocity prediction [29, 27]. Evaluations are performed on the EMA weights. Auxiliary training-time augmentations. Two augmentations further tighten reference-to-text binding on top of the timestep distribution. Adversarial reference injection attacks the shortcut from a complementary angle. At training time we fill empty reference slots with extra “distractor” references not mentioned in the prompt. The only way to satisfy the prompt is then to bind by text rather than copy from whatever happens to be in the sequence. No new loss term is introduced; the standard rectified-flow objective is unchanged, only the input. Slot-shuffle augmentation reduces positional bias on the reference slots. At each step we permute the order of the references in the self-attention input sequence and rewrite the prompt’s reference k tokens to match. The model thus cannot rely on, e.g., “slot 1 is always the first speaker”. Our default uses both adversarial reference injection and slot-shuffle, with shuffle introduced as a curriculum: no shuffle for the first 10,000 steps (so the model first learns the basic mapping under a fixed slot order), shuffle thereafter. We ablate both in Appendix A.4. Evaluation set: C OVO M IX 2-D IALOGUE -20 S. We evaluate on the public CoVoMix2 dialog test set [54], which pairs 1,000 DailyDialog [24] two-speaker transcripts with reference clips drawn from LibriSpeech test-clean [31]. We restrict to the 291 dialogs whose target fits the model’s 20 s budget, and call this subset C OVO M IX 2-D IALOGUE -20 S. The retained samples preserve the same speaker-gender mix and reference-similarity distribution as the full test set. For every dialog, the two LibriSpeech prompt clips serve as r1 and r2 , and the DailyDialog transcript is rendered into our standard reference 1/reference 2 prompt format. Reference robustness set: C OVO M IX 2-D IALOGUE -W ILD R EF. C OVO M IX 2-D IALOGUE W ILD R EF probes realistic conditions beyond studio-clean LibriSpeech. We sample 50 dialogs from C OVO M IX 2-D IALOGUE -20 S and re-pair them with 30 in-the-wild English reference clips (crowd noise, background music, street ambience, wind, cartoon voices, and similar). This yields 100 examples, with each wild clip used at least three times. Holding the dialogs fixed isolates the effect of the reference distribution. Metrics. We report seven metrics. Three are standard: WER (Whisper-large-v3 [35]), UTMOS [36], and SQUIM [22]. The remaining four are multi-speaker variants central to our analysis. cpWER [43] is a speaker-aware WER, computed post-diarization with best-permutation alignment so that attribution errors count. SIM-O and cpSIM are cosine similarities between WavLM-ECAPA [4] speaker embeddings, with cpSIM the strict per-speaker variant most sensitive to reference-speaker binding. ACC is the fraction of words whose generated speaker (MMS forced alignment [34] + persegment WavLM-ECAPA assignment) matches the prompt label. Diarization uses pyannote.audio [2]. 4.2
Comparison with Multi-Speaker Dialog Baselines
We compare S CENA against current multi-speaker / dialog TTS systems on C OVO M IX 2-D IALOGUE 20 S: MOSS-TTSD [55], VibeVoice-1.5B and VibeVoice-7B [33], ZipVoice-Dialog [56], and Dia (Nari Labs) [30]. All baselines are run with their public default settings. As shown in Table 1, S CENA obtains the best cpWER, cpSIM, and ACC (the binding-aware metrics), together with the best WER and best (tied) SIM-O. The two naturalness estimators diverge on S CENA: SQUIM places it at 4.32 (within 0.02 of the leaders), while UTMOS reads 3.44 versus 3.76 for MOSS-TTSD. We attribute the UTMOS gap to the LTX-2.3 backbone, which is trained on in-the-wild video soundtracks rather than studio speech and inherits an acoustic profile that UTMOS scores more conservatively than SQUIM does.
7
Table 1: Comparison with multi-speaker dialog baselines on C OVO M IX 2-D IALOGUE -20 S. cpWER ↓
cpSIM ↑
ACC ↑
WER ↓
SIM-O ↑
UTMOS ↑
SQUIM ↑
MOSS-TTSD [55] VibeVoice-7B [33] VibeVoice-1.5B [33] ZipVoice-Dialog [56] Dia (Nari Labs) [30]
0.232 0.206 0.212 0.176 0.303
0.547 0.527 0.503 0.538 0.339
0.855 0.821 0.830 0.847 0.757
0.109 0.044 0.050 0.032 0.133
0.443 0.451 0.423 0.446 0.312
3.76 3.58 3.56 3.57 2.69
4.28 4.28 4.27 4.34 4.09
S CENA
0.145
0.567
0.866
0.020
0.451
3.44
4.32
System
Table 2: Comparison on C OVO M IX 2-D IALOGUE -W ILD R EF (50 dialogs paired with 30 in-the-wild reference clips). cpWER ↓
cpSIM ↑
ACC ↑
WER ↓
SIM-O ↑
UTMOS ↑
SQUIM ↑
MOSS-TTSD [55] VibeVoice-7B [33] VibeVoice-1.5B [33] ZipVoice-Dialog [56] Dia (Nari Labs) [30]
0.156 0.172 0.202 0.173 0.272
0.390 0.386 0.365 0.396 0.278
0.844 0.841 0.826 0.825 0.752
0.059 0.045 0.089 0.038 0.086
0.295 0.317 0.293 0.315 0.256
3.45 2.56 2.33 3.20 2.45
4.21 2.91 2.85 4.19 3.92
S CENA
0.167
0.424
0.819
0.022
0.348
3.30
4.28
System
Robustness to in-the-wild references. Table 2 repeats the comparison on C OVO M IX 2-D IALOGUE W ILD R EF, where the studio-clean LibriSpeech prompts are replaced by 30 in-the-wild reference clips (50 dialogs, see §4.1). S CENA retains the best cpSIM, SIM-O, WER, and SQUIM, and remains close to the leader on cpWER and ACC, where MOSS-TTSD edges ahead. On the wild references, every baseline’s cpSIM drops by roughly 0.15 absolute, falling below 0.40, while S CENA stays above 0.42. The smaller open-source baselines (VibeVoice, Dia) fall off under the harder reference distribution. 4.3
Human Evaluation
We ran a side-by-side A/B preference test against the four baselines from §4.2, on items drawn from both C OVO M IX 2-D IALOGUE -20 S and C OVO M IX 2-D IALOGUE -W ILD R EF; the protocol is in Appendix A.5. S CENA is preferred over every baseline at conventional significance (Table 3). 4.4
Noise Schedule Ablation
The probe illustrates that the shortcut is available; this experiment tests whether closing it at training time is what actually unlocks binding. We compare our Beta+Uniform mixture (§3.3) against three logit-normal distributions whose mass spans the range used in standard flow-matching practice, while keeping all other training settings fixed (§4.1). The first row (µ=0.17, σ=0.75) is closest to typical audio FM training; the remaining two progressively shift their mass toward higher noise. Figure 4 shows the density of each timestep distribution (bottom) and the corresponding metrics (top). All three binding-aware metrics (cpWER, cpSIM, ACC) improve monotonically as the schedule shifts toward higher noise. S CENA leads on every binding metric while remaining competitive on the general audio-quality column.
Table 3: Side-by-side A/B preference results on a mix of C OVO M IX 2-D IALOGUE -20 S and C OVO M IX 2-D IALOGUE -W ILD R EF items. S CENA preferred is S CENA’s win rate among nontie ratings. Significance is from a two-sided binomial test (∗ p<0.05, ∗∗ p<0.01, ∗∗∗ p<0.001). Opponent
S CENA preferred 84.6%∗∗∗ 74.2%∗∗∗ 68.3%∗∗ 59.8%∗
ZipVoice-Dialog [56] Dia [30] VibeVoice-7B [33] MOSS-TTSD [55]
8
Binding System LogitNormal µ=0.17, σ=0.75 LogitNormal µ=0.62, σ=0.75 LogitNormal µ=0.77, σ=1.0 S CENA (Beta+Uniform)
density
3
Standard
cpWER ↓
cpSIM ↑
ACC ↑
WER ↓
SIM-O ↑
UTMOS ↑
SQUIM ↑
0.167 0.158 0.154 0.145
0.503 0.500 0.549 0.567
0.830 0.850 0.859 0.866
0.019 0.019 0.018 0.020
0.402 0.402 0.438 0.451
3.64 3.65 3.54 3.44
4.29 4.30 4.30 4.32
Beta+Uniform (ours) LogitNormal = 0.77, = 1.0 LogitNormal = 0.62, = 0.75 LogitNormal = 0.17, = 0.75
2 1 0 0.00
0.25
0.50 timestep t
0.75
1.00
Figure 4: Noise schedule ablation on C OVO M IX 2-D IALOGUE -20 S. Top: results table, with row tints matching the curves below. Bottom: density of each sampler distribution; all variants share a 10% uniform mix. 4.5
Qualitative Capabilities Beyond Per-Utterance Dialog
The quantitative comparisons in §4.2 hold the input format fixed to two-speaker dialog transcripts, the regime baselines were built for. Our text-prompt format also enables generation modes that are difficult for per-utterance systems: overlapping speech (unison readings, talk-overs, brief interjections), ambient and non-speech audio generated jointly with the dialog, spontaneous paralinguistic events (laughter, sighs, gasps, breaths) attributed in-line to named speakers, and multiple references per speaker bound to a single voice for richer acoustic characterization. We refer the reader to the project page for audio examples and to Appendix A.2 for spectrograms of selected scenarios.
5
Discussion
Contemporary multi-speaker systems typically route the binding problem through structured supervision: per-turn tags, multi-stream transcripts, identity encoders, or spatial layouts. Our results show none of this is necessary. A standard flow-matching transformer with concatenated reference latents and a single learned slot vector per index suffices, provided the timestep distribution is chosen to defeat the reference shortcut. This minimalism is what unlocks the qualitative regimes in §4.5; no system with a structured interface can attempt them. The shortcut argument itself is not specific to audio: it requires only (i) one or more clean references concatenated with a noised target, (ii) the noised target retaining enough information for similaritybased selection at moderate noise, and (iii) training mass concentrated outside the high-noise tail. These conditions are satisfied by any reference-conditioned flow-matching model. We expect the diagnosis and the high-noise-biased fix to transfer to image and video reference conditioning, though direct cross-modality verification is future work.
6
Limitations
We inherit two practical limits from the backbone: a 20 s generation cap and Kmax = 3 reference speakers, beyond which the self-attention sequence grows linearly with K. Both are softer than they appear: in LTX-2, audio tokens are roughly an order of magnitude fewer than video tokens, so an audio-only configuration has substantial headroom, enough to extend duration with modest fine-tuning and to fit additional references (or to add a learned reference-side compressor) without redesigning the model. A third limit is inherent to the flow-matching paradigm: generation duration must be set before sampling, so FM systems resort to either heuristics or user input.
9
References [1] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, Mingqing Gong, Peisong Huang, Qingqing Huang, Zhiying Huang, Yuanyuan Huo, Dongya Jia, Chumin Li, Feiya Li, Hui Li, Jiaxin Li, Xiaoyang Li, Xingxing Li, Lin Liu, Shouda Liu, Sichao Liu, Xudong Liu, Yuchen Liu, Zhengxi Liu, Lu Lu, Junjie Pan, Xin Wang, Yuping Wang, Yuxuan Wang, Zhen Wei, Jian Wu, Chao Yao, Yifeng Yang, Yuanhao Yi, Junteng Zhang, Qidi Zhang, Shuo Zhang, Wenjie Zhang, Yang Zhang, Zilin Zhao, Dejian Zhong, and Xiaobin Zhuang. Seed-TTS: A family of high-quality versatile speech generation models, 2024. [2] Hervé Bredin. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In Proc. INTERSPEECH 2023, pages 1983–1987, 2023. [3] Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, and Julian Weber. XTTS: A massively multilingual zero-shot text-to-speech model. In Proc. INTERSPEECH, 2024. [4] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518, 2022. [5] Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. VALL-E 2: Neural codec language models are human parity zero-shot text to speech synthesizers, 2024. [6] Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-TTS: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024. [7] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [8] Yufan Deng, Yuanyang Yin, Xun Guo, Yizhi Wang, Jacob Zhiyuan Fang, Shenghai Yuan, Yiding Yang, Angtian Wang, Bo Liu, Haibin Huang, and Chongyang Ma. MAGREF: Masked guidance for any-reference video generation with subject disentanglement, 2025. [9] Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. CosyVoice 2: Scalable streaming speech synthesis with large language models, 2024. [10] Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, Keyu An, Guanrou Yang, Yabin Li, Yanni Chen, Zhifu Gao, Qian Chen, Yue Gu, Mengzhe Chen, Yafeng Chen, Shiliang Zhang, Wen Wang, and Jieping Ye. CosyVoice 3: Towards in-the-wild speech generation via scaling-up and post-training, 2025. [11] Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, Yanqing Liu, Sheng Zhao, and Naoyuki Kanda. E2 TTS: Embarrassingly easy fully non-autoregressive zero-shot TTS. In IEEE Spoken Language Technology Workshop (SLT), 2024. [12] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning (ICML), 2024. [13] Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, Eitan Richardson, Guy Shiran, Itay Chachy, Jonathan Chetboun, Michael Finkelson, Michael Kupchick, Nir Zabari, Nitzan Guetta, Noa Kotler, Ofir Bibi, Ori Gordon, Poriya Panet, Roi Benita, Shahar Armon, Victor Kulikov, Yaron Inger, Yonatan Shiftan, Zeev Melumian, and Zeev Farbman. LTX-2: Efficient joint audio-visual foundation model, 2026. 10
[14] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via Min-SNR weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. [15] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. Simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning (ICML), 2023. [16] Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, and Jingren Zhou. In-context LoRA for diffusion transformers, 2024. [17] Sangwon Jang, Jaehyeong Jo, Kimin Lee, and Sung Ju Hwang. Identity decoupling for multisubject personalization of text-to-image models, 2024. [18] Ziyue Jiang, Yi Ren, Ruiqi Li, Shengpeng Ji, Boyang Zhang, Zhenhui Ye, Chen Zhang, Jionghao Bai, Xiaoda Yang, Jialong Zuo, Yu Zhang, Rui Liu, Xiang Yin, and Zhou Zhao. MegaTTS 3: Sparse alignment enhanced latent diffusion transformer for zero-shot speech synthesis. arXiv preprint arXiv:2502.18924, 2025. [19] Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang-Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and Sheng Zhao. NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. In International Conference on Machine Learning (ICML), 2024. [20] Zeqian Ju, Dongchao Yang, Jianwei Yu, Kai Shen, Yichong Leng, Zhengtao Wang, Xu Tan, Xinyu Zhou, Tao Qin, and Xiangyang Li. MoonCast: High-quality zero-shot podcast generation. arXiv preprint arXiv:2503.14345, 2025. [21] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2022. [22] Anurag Kumar, Ke Tan, Zhaoheng Ni, Pranay Manocha, Xiaohui Zhang, Ethan Henderson, and Buye Xu. TorchAudio-Squim: Reference-less speech quality and intelligibility measures in TorchAudio. In ICASSP 2023 – 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023. [23] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voicebox: Text-guided multilingual universal speech generation at scale. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [24] Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (IJCNLP), 2017. [25] Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, and Nima Mesgarani. StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. [26] Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. PhotoMaker: Customizing realistic human photos via stacked ID embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. [27] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023. [28] Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. AudioLDM 2: Learning holistic audio generation with self-supervised pretraining, 2023. [29] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations (ICLR), 2023. [30] Nari Labs. Dia: A TTS model capable of generating ultra-realistic dialogue in one pass. GitHub repository, 2024. Open-weights release. https://github.com/nari-labs/dia. 11
[31] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, 2015. doi: 10.1109/ ICASSP.2015.7178964. [32] Puyuan Peng, Shang-Wen Li, Abdelrahman Mohamed, and David Harwath. VoiceStar: Robust zero-shot autoregressive TTS with duration control and extrapolation, 2025. [33] Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, Shaohan Huang, Yan Xia, and Furu Wei. VibeVoice technical report, 2025. [34] Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1–52, 2024. [35] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023. [36] Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. UTMOS: UTokyo-SaruLab system for VoiceMOS challenge 2022. In Proc. INTERSPEECH 2022, pages 4521–4525, 2022. [37] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021. [38] Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, Jeff Wang, Ivan Cruz, Bapi Akula, Akinniyi Akinyemi, Brian Ellis, Rashel Moritz, Yael Yungster, Alice Rakotoarison, Liang Tan, Chris Summers, Carleigh Wood, Joshua Lane, Mary Williamson, and Wei-Ning Hsu. Audiobox: Unified audio generation with natural language prompts, 2023. [39] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers, 2023. [40] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. InstantID: Zero-shot identity-preserving generation in seconds, 2024. [41] Xierui Wang, Siming Fu, Qihan Huang, Wanggui He, and Hao Jiang. MS-Diffusion: Multisubject zero-shot image personalization with layout guidance, 2024. [42] Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, Weizhen Bian, Zhen Ye, Sitong Cheng, Ruibin Yuan, Zhixian Zhao, Xinfa Zhu, Jiahao Pan, Liumeng Xue, Pengcheng Zhu, Yunlin Chen, Zhifei Li, Xie Chen, Lei Xie, Yike Guo, and Wei Xue. Spark-TTS: An efficient LLM-based text-to-speech model with single-stream decoupled speech tokens, 2025. [43] Shinji Watanabe, Michael Mandel, Jon Barker, Emmanuel Vincent, Ashish Arora, Xuankai Chang, Sanjeev Khudanpur, Vimal Manohar, Daniel Povey, Desh Raj, David Snyder, Aswin Shanmugam Subramanian, Jan Trmal, Bar Ben Yair, Christoph Boeddeker, Zhaoheng Ni, Yusuke Fujita, Shota Horiguchi, Naoyuki Kanda, Takuya Yoshioka, and Neville Ryant. CHiME-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings. In Proceedings of the 6th International Workshop on Speech Processing in Everyday Environments (CHiME 2020), 2020. [44] Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, and Qian He. Less-to-more generalization: Unlocking more controllability by in-context generation, 2025. [45] Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, and Song Han. FastComposer: Tuning-free multi-subject image generation with localized attention, 2023. [46] Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. OmniGen: Unified image generation, 2024. [47] Hanke Xie, Dake Guo, Chengyou Wang, Yue Li, Wenjie Tian, Xinfa Zhu, Xinsheng Wang, Xiulin Li, Guanqiong Miao, Bo Liu, and Lei Xie. DialoSpeech: Dual-speaker dialogue generation with LLM and flow matching, 2025. 12
[48] Hanke Xie, Haopeng Lin, Wenxiao Cao, Dake Guo, Wenjie Tian, Jun Wu, Hanlin Wen, Ruixuan Shang, Hongmei Liu, Zhiqi Jiang, Yuepeng Jiang, Wenxi Chen, Ruiqi Yan, Jiale Qian, Yichao Yan, Shunshun Yin, Ming Tao, Xie Chen, Lei Xie, and Xinsheng Wang. SoulX-Podcast: Towards realistic long-form podcasts with dialectal and paralinguistic diversity, 2025. [49] Kun Xie, Feiyu Shen, Junjie Li, Fenglong Xie, Xu Tang, and Yao Hu. FireRedTTS-2: Towards long conversational speech generation for podcast and chatbot, 2025. [50] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models, 2023. [51] Fan Yu, Tao Wang, You Wu, Lin Zhu, Wei Deng, Weisheng Han, Wenchao Wang, Lin Hu, Xiangyu Liang, Xiaodong He, Yankun Huang, Yu Gu, Yuan Liu, Yuxuan Wang, Zhangyu Xiao, Ziteng Wang, Boya Dong, Feng Dang, Jinming Chen, Jingdong Li, Jun Wang, Yechen Jin, Yuan Zhang, Zhengyan Sheng, and Xin Wang. JoyVoice: Long-context conditioning for anthropomorphic multi-speaker conversational synthesis, 2025. [52] Bowen Zhang, Congchao Guo, Geng Yang, Hang Yu, Haozhe Zhang, Heidi Lei, Jialong Mai, Junjie Yan, Kaiyue Yang, Mingqi Yang, Peikai Huang, Ruiyang Jin, Sitan Jiang, Weihua Cheng, Yawei Li, Yichen Xiao, Yiying Zhou, Yongmao Zhang, Yuan Lu, and Yucen He. MiniMaxSpeech: Intrinsic zero-shot text-to-speech with a learnable speaker encoder. arXiv preprint arXiv:2505.07916, 2025. [53] Leying Zhang, Yao Qian, Long Zhou, Shujie Liu, Dongmei Wang, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He, Sheng Zhao, and Michael Zeng. CoVoMix: Advancing zero-shot speech generation for human-like multi-talker conversations. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [54] Leying Zhang, Yao Qian, Xiaofei Wang, Manthan Thakker, Dongmei Wang, Jianwei Yu, Haibin Wu, Yuxuan Hu, Jinyu Li, Yanmin Qian, and Sheng Zhao. CoVoMix2: Advancing zero-shot dialogue generation with fully non-autoregressive flow matching, 2025. [55] Yuqian Zhang, Donghua Yu, Zhengyuan Lin, Botian Jiang, Mingshu Chen, Yaozhou Jiang, Yiwei Zhao, Yiyang Zhang, Yucheng Yuan, Hanfu Chen, Kexin Huang, Jun Zhan, Cheng Chang, Zhaoye Fei, Shimin Li, Xiaogui Yang, Qinyuan Cheng, and Xipeng Qiu. MOSS-TTSD: Text to spoken dialogue generation, 2026. [56] Han Zhu, Wei Kang, Liyong Guo, Zengwei Yao, Fangjun Kuang, Weiji Zhuang, Zhaoqing Li, Zhifeng Han, Dong Zhang, Xin Zhang, Xingchen Song, Lingxuan Ye, Long Lin, and Daniel Povey. ZipVoice-Dialog: Non-autoregressive spoken dialogue generation with flow matching, 2025. [57] Han Zhu, Wei Kang, Zengwei Yao, Liyong Guo, Fangjun Kuang, Zhaoqing Li, Weiji Zhuang, Long Lin, and Daniel Povey. ZipVoice: Fast and high-quality zero-shot text-to-speech with flow matching. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2025.
13
A
Supplementary
A.1
Reference Shortcut Probe Setup
The probe is trained on a binary classification task: given the noised target zt and two candidate references (one drawn from the same sample as the target, the other from a different sample), predict which of the two matches the target. We take the first 8 of the 48 transformer blocks of our backbone (discarding the rest) and attach a small two-layer MLP classification head on the pooled output. Inputs are routed exactly as in the full model. The target caption enters through the text cross-attention. The noised target zt and the two references form the self-attention sequence, with the references shuffled per example. The timestep t is supplied through the backbone’s standard conditioning. The head outputs a single logit predicting which of the two reference slots belongs to the target. We train the probe for 10,000 steps at batch size 128, sampling t uniformly in [0, 1], and evaluate accuracy on 256 held-out examples at each of 50 evenly spaced timesteps in [0, 1]. A.2
Qualitative Spectrograms
Figure 5: Rapid-fire quiz closed by a non-speech buzzer.
Figure 6: Scene-level prompt with a mid-utterance ambient interruption and paralinguistic events.
A.3
Reference Slot Encoding
We compare three slot-encoding choices. The first is our additive slot embedding (Eq. 2). The second is a RoPE-based alternative that augments the rotary positional encoding [37] with an additional dimension. The new dimension takes value k for tokens belonging to reference rk and value 0 for tokens of the target zt . The original temporal RoPE continues to index time within each segment, while the new dimension distinguishes the target from each reference: RoPE(x) = RoPEtemporal (τ ) ⊗ RoPEref (k),
(5)
where ⊗ denotes concatenation along the per-dimension rotations, τ is the within-segment time of token x, and k is its slot index. The third is a no-positional baseline that concatenates references in the self-attention sequence without any additional encoding to distinguish them from the target or from each other. Adding a new RoPE dimension changes the positional statistics seen by the pretrained backbone. For a fair comparison, we precede RoPE fine-tuning with a 5,000-step warmup on our general-audio corpus, with no references but the extra RoPE dimension already in place. This lets the backbone adapt to the modified positional encoding before reference conditioning is introduced. The additive and no-positional variants require no such adaptation and start fine-tuning directly from the backbone. 14
Table 4: Reference slot encoding ablation on C OVO M IX 2-D IALOGUE -20 S. The RoPE variant uses an additional 5,000-step warmup on general audio with the extra RoPE dimension in place. System S CENA (no-pos) S CENA (RoPE) S CENA (additive)
cpWER ↓
cpSIM ↑
ACC ↑
WER ↓
SIM-O ↑
UTMOS ↑
SQUIM ↑
0.232 0.181 0.145
0.403 0.547 0.567
0.513 0.835 0.866
0.018 0.020 0.020
0.333 0.449 0.451
3.60 3.58 3.44
4.28 4.32 4.32
Table 5: Training-recipe ablation on C OVO M IX 2-D IALOGUE -20 S. no adversarial: same recipe as ours but without random distractor references. always-shuffle: reference-slot shuffle augmentation from step 0. no-shuffle: never shuffle. Our default uses both adversarial distractors and a shuffle curriculum (no shuffle for the first 10 k steps, shuffle thereafter). System S CENA (no adversarial) S CENA (always-shuffle) S CENA (no-shuffle) S CENA
cpWER ↓
cpSIM ↑
ACC ↑
WER ↓
SIM-O ↑
UTMOS ↑
SQUIM ↑
0.157 0.232 0.131 0.145
0.467 0.402 0.491 0.567
0.859 0.502 0.886 0.866
0.018 0.019 0.018 0.020
0.368 0.334 0.380 0.451
3.52 3.68 3.68 3.44
4.29 4.26 4.28 4.32
Table 4 reports results on C OVO M IX 2-D IALOGUE -20 S. The additive embedding leads on every binding-aware metric (cpWER, cpSIM, ACC); the RoPE variant follows closely. The no-positional baseline collapses on binding: ACC falls to 0.513 (essentially chance for two speakers) and cpSIM drops by 0.16. WER and naturalness are largely unchanged, since the model can still produce coherent speech when freed from the binding constraint. This indicates that some explicit slot signal is necessary, but binding does not hinge on its precise form. A.4
Training Recipe Ablations
Table 5 ablates the two auxiliary augmentations described in §4.1 (adversarial reference injection and slot-shuffle augmentation) on C OVO M IX 2-D IALOGUE -20 S. Removing adversarial references (no adversarial) drops cpSIM by ∼0.10 and SIM-O by ∼0.08, with WER and naturalness essentially unchanged, confirming that the augmentation specifically tightens reference-speaker fidelity rather than generation quality. Reference-shuffle behaviour is more nuanced: always-shuffle from step 0 collapses (ACC 0.50, near chance), because the model never gets a chance to anchor a stable reference-to-slot mapping before the augmentation starts moving slots around. No-shuffle at all is competitive on cpWER and ACC, but our curriculum retains a clear lead on the strict speaker-fidelity metrics (cpSIM, SIM-O, SQUIM) where positional bias would surface in adversarial / wild-reference conditions. A.5
Subjective Evaluation
Table 6 shows the instructions presented to evaluators for the side-by-side preference test. Each evaluator is given two reference recordings of the original speakers (natural, noisy recordings) followed by two synthesized versions of a dialog between them, denoted (a) and (b). For every comparison the evaluator selects (a), (b), or “About the same”. Within a session, our system is paired against one randomly chosen competitor per question, with the (a)/(b) order randomized per item; evaluators are blind to system identity.
15
Table 6: Side-by-side A/B Preference Evaluation Instructions. Instruction You’ll hear recordings of two original speakers (natural recordings with background noise; ignore the noise). Then you’ll hear 2 synthesized versions of a dialog between them. Pick the version where the voices sound more like the original speakers and the conversation feels more natural. Which one is better? (a): Dialog (a) is better (b): Dialog (b) is better About the same: Cannot tell which is better
16