Amap Voice Alibaba Group
Alibaba Group, 2026
Amap Voice · Technical Report · 2026
PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis Bowen Li1,∗ Shaotong Guo1,∗ Zhen Wang1,∗ Yang Xiang1 Mingli Jin1 Yihang Lin1,2 Jiahui Zhao1 Weibo Xiong1 Dongrui Li1 Keming Chen1 Yunze Gao1 Yuze Zhou1 Zeyang Lin1,† Yue Liu1,†
arXiv:2605.27258v1 [cs.SD] 26 May 2026
1 Amap, Alibaba Group
2 The Chinese University of Hong Kong, Shenzhen
Abstract Building state-of-the-art text-to-speech (TTS) systems typically demands millions of hours of proprietary data and complex multi-stage architectures, creating substantial barriers for resource-constrained research teams. In this report, we present PilotTTS, a lightweight autoregressive TTS system that achieves competitive performance through minimalist architecture and rigorous data engineering. PilotTTS is trained on only 200K hours of data processed entirely with open-source tools. Specifically, our contributions are: (1) a reproducible multi-stage data processing pipeline covering quality assessment, label annotation, and filtering, and (2) a compact model architecture that employs Q-Former-based conditioning to decouple speaker identity from speaking style via cross-sample paired training. Within a unified framework, PilotTTS supports zero-shot voice cloning, emotion synthesis (11 categories), paralinguistic synthesis (4 categories), and Chinese dialect synthesis (14 dialects). On the Seed-TTS Eval benchmark, PilotTTS achieves the lowest WER of 1.50% on test-en, a CER of 0.87% on test-zh, and the highest speaker similarity on both test sets (0.862 and 0.815), outperforming systems trained on significantly larger datasets. We release the complete data pipeline recipe, pretrained weights, and code at https://github.com/AMAPVOICE/PilotTTS.
1 Introduction With the rapid advancement of generative neural networks, text-to-speech (TTS) synthesis has achieved remarkable progress in naturalness, expressiveness, and speaker fidelity [1–3]. A particularly transformative development is large-scale zero-shot TTS, where models trained on extensive multi-speaker corpora can clone arbitrary voices from only a few seconds of reference audio. Recent systems built upon large language model (LLM) architectures [3–6] or non-autoregressive flow-based generation [7–9] have demonstrated impressive quality, establishing new standards for the field. However, this progress has been accompanied by a persistent trend toward ever-larger datasets (from hundreds of thousands to millions of hours [10, 11]) and increasingly complex architectures involving multi-codebook tokenizers, multi-stage training pipelines, and multiple specialized sub-models. This trajectory, while yielding state-of-the-art results, poses significant challenges for the broader research community. First, acquiring and processing millions of hours of speech data requires proprietary infrastructure and substantial computational resources, with few standardized open-source tools available for constructing high-quality training sets. Second, architectural complexity, such as multi-codebook residual quantization, hierarchical prediction modules, and streaming-specific designs, increases engineering effort and deployment difficulty. Third, advanced controllability features (emotion synthesis, paralinguistic synthesis, dialect synthesis) are typically developed as separate specialized systems, ∗ Equal contribution. † Corresponding author: [email protected], [email protected]
1
PilotTTS
Alibaba Group, 2026
Figure 1: Overview of PilotTTS.
further fragmenting the required expertise and resources. While these directions undoubtedly push the upper bound of speech synthesis expressiveness and represent valid research goals, most commercial deployment scenarios call for a less complex alternative that still meets practical requirements. In such cases, efforts toward extreme expressiveness can paradoxically become an obstacle to rapid deployment and significantly inflate application cost. These barriers collectively restrict resource-constrained teams from building competitive TTS systems or from extending and refining the approaches they already have access to. In this report, we present PilotTTS, an autoregressive TTS framework rapidly assembled to meet pressing practical demand, achieving competitive performance by combining well-established opensource components with meticulous data engineering, rather than through architectural novelty or data scale. The system employs a Qwen3 language model [12] as the autoregressive backbone with a Q-Former-based [13] conditioning module for speaker and style representation, and generates speech through a Conditional Flow Matching (CFM) [14] decoder with a Diffusion Transformer (DiT) [15] backbone followed by a HiFi-GAN [16] vocoder. Trained on approximately 200K hours of speech data collected from public sources and processed using a pipeline built entirely from publicly available tools, PilotTTS delivers performance competitive with systems trained on an order of magnitude more data. Our main contributions are as follows: 1. A reproducible data processing pipeline built upon publicly available modules. We design a multi-stage pipeline covering quality assessment and enhancement, label annotation, and quality filtering, assembled entirely from publicly available tools. This pipeline transforms raw internet audio into clean, richly annotated training data at substantially reduced cost. Since every constituent module is openly accessible, any team can readily obtain the underlying components and rapidly assemble their own data processing pipeline, lowering the barrier for TTS data preparation. 2. A compact autoregressive TTS architecture with decoupled conditioning. We propose a Q-Formerbased speaker-style conditioning mechanism paired with a cross-sample training strategy that disentangles static speaker identity from dynamic speaking style. This design achieves the highest speaker similarity (0.862) and highly competitive content accuracy (CER 0.87%) on the Seed-TTS benchmark, using only 200K hours of training data. 3. Multi-dimensional controllability under low-resource constraints. Within the same framework, PilotTTS demonstrates strong performance in controllable synthesis across dialect, paralinguistic, and emotional dimensions, supporting emotion control across 11 categories, paralinguistic synthesis across 4 categories (laughter, breathing, crying, and coughing), and Chinese dialect synthesis across 14 Chinese dialects, all achieved through targeted post-training.
2
PilotTTS
Alibaba Group, 2026
The remainder of this report is organized as follows. Section 2 describes our data processing pipeline. Section 3 details the model architecture and training strategy. Section 4 presents experimental results across multiple evaluation dimensions. Section 5 concludes with a discussion of limitations and future directions.
2 Data Processing Pipeline
Figure 2: Overview of the three-stage data processing pipeline. Raw audio is first assessed and enhanced (left), then annotated with textual, temporal, and speaker-related labels (middle), and finally filtered through multi-dimensional quality checks to produce approximately 200K hours of training data (right).
High-quality training data is critical for competitive speech synthesis, yet existing large-scale TTS systems often rely on proprietary data processing pipelines that limit reproducibility. To address this, we design a multi-stage data processing pipeline built entirely upon publicly available modules, composed of three sequential stages: quality assessment and enhancement, label annotation, and quality filtering (Figure 2). Starting from raw waveform data collected from heterogeneous real-world sources, the pipeline progressively refines the audio through acoustic preprocessing, structured annotation, and reliability-based sample selection, ultimately producing a clean and richly labeled dataset for downstream model training. 2.1 Quality Assessment and Enhancement The first stage evaluates and improves the acoustic quality of the input audio. Since raw recordings from diverse sources may exhibit substantial variability in recording conditions, background noise, speaker transitions, and signal quality, this stage combines audio standardization, speech structure analysis, and quality-aware enhancement. We first standardize all recordings to a unified format and sampling rate to reduce source-level heterogeneity. The standardized audio is then segmented by a speech activity detection (SAD) module together with a speaker change detection (SCD) module to identify valid speech regions and detect speaker transitions [17, 18].
3
PilotTTS
Alibaba Group, 2026
Next, three quality-related attributes are estimated in parallel. We use DNSMOS [19] to predict perceptual speech quality, a speech/non-speech classifier1 to determine whether a segment contains valid speech rather than music, environmental noise, or other non-speech events, and an SNR estimation module to measure the signal-to-noise ratio. Based on these indicators, each segment is assigned a quality status. Segments with low predicted perceptual quality (e.g., predicted MOS ≤ 3.5), non-speech labels, or insufficient SNR are regarded as acoustically deficient. For segments identified as low quality, we apply a denoising and enhancement module2 to suppress background noise and improve speech intelligibility. This dual assessment-and-enhancement design provides cleaner inputs for subsequent annotation. 2.2 Label Annotation The second stage generates structured annotations for each processed speech segment, including textual, temporal, speaker-related, and prosody-related information. Each audio segment is first transcribed by multiple automatic speech recognition (ASR) systems to obtain textual content. We collect transcription results from several open-source systems, including Paraformer [20], FireRedASR [21], and Whisper [22], as well as internal ASR models. Multi-system transcriptions enable cross-system consistency checking, improving the reliability of the final text labels. Next, each audio-text pair is processed by an overlapping speech detection (OSD) module3 and a forced alignment module. The OSD component identifies segments containing simultaneous speech from multiple speakers, while the aligner establishes temporal correspondence between the transcribed text and the acoustic signal. Based on the alignment results, we further apply a prosody annotation system built upon Qwen3-Force-Alignment [23] to annotate hierarchical prosodic structures. We then perform speaker tagging using 3D-Speaker-Toolkit [24] to assign speaker identity metadata to each segment, facilitating the construction of speaker-consistent training samples. In addition, we perform spectral rolloff analysis to identify low-bandwidth recordings with insufficient high-frequency content, preventing audio with excessively low effective sampling rates from entering the training set. 2.3 Quality Filtering The final stage removes unreliable or undesirable samples based on the quality signals and annotations accumulated in the previous stages. We first apply a truncation detector to identify utterances with incomplete beginnings or endings. Such samples are typically caused by inaccurate segmentation or abrupt clipping and can adversely affect model training. We then employ a synthetic speech detector to identify segments that are likely to be artificially generated rather than naturally spoken. This step is particularly important for large-scale crawled data, where synthesized speech may be mixed with authentic recordings. After these dedicated detection steps, all available metadata and quality indicators are aggregated in a final filtering module. This module performs sample selection by jointly considering acoustic quality, speech validity, transcription reliability, overlap condition, speaker consistency, truncation risk, synthesis likelihood, and spectral quality. Only samples that satisfy all filtering criteria are retained. Rather than discarding excluded samples, we preserve all processed data together with their quality tags and annotation metadata, enabling the flexible construction of datasets with different quality requirements for future tasks. After filtering, we retain approximately 200,000 hours of Chinese and English speech data for pre-training. All constituent modules used in this pipeline are openly accessible, allowing any team to readily obtain them and rapidly assemble a comparable pipeline tailored to their own data preparation needs. 1 https://www.modelscope.cn/models/iic/SenseVoiceSmall 2 https://github.com/resemble-ai/resemble-enhance 3 https://huggingface.co/pyannote/segmentation-3.0
4
PilotTTS
Alibaba Group, 2026
3 Method
Figure 3: Overall architecture of PilotTTS. Left: the full pipeline comprising a Speaker Feature Extraction module (CAMPPlus), a Semantic Content Adapter (Q-Former), a Text Tokenizer, the Qwen3 language model for autoregressive semantic token prediction, a CFM decoder, and a HiFi-GAN vocoder. Right: detailed structure of the Semantic Content Adapter, where learnable query vectors attend to w2v-BERT embeddings through cross-attention with a Conformer block and linear projection.
3.1 Overview The overall architecture of PilotTTS is illustrated in Figure 3. The system consists of four components: (1) a speech tokenizer that converts continuous speech into discrete semantic tokens (Section 3.2), (2) an autoregressive text-to-semantic module built upon Qwen3 [12] that generates semantic token sequences conditioned on text and reference audio (Section 3.3), (3) a Conditional Flow Matching (CFM) [14] decoder with a Diffusion Transformer (DiT) [15] backbone that synthesizes mel-spectrograms conditioned on the predicted semantic tokens (Section 3.4) and reference information, and (4) a HiFi-GAN [16] vocoder that converts mel spectrograms into waveforms. During inference, the target text is first encoded into text tokens using the Qwen3 tokenizer. Simultaneously, the reference audio is processed through two conditioning pathways: a Q-Former-based [13] Semantic Content Adapter that compresses the reference speech into a fixed set of condition tokens, and a frozen CAMPPlus [25] speaker encoder that extracts a global speaker identity embedding. These conditioning representations, together with the text tokens, form the input to the Qwen3 language model, which autoregressively predicts discrete semantic codes. The predicted codes, the speaker embedding, and the reference mel spectrogram are then passed to the CFM decoder to produce the target mel spectrogram, which is finally converted into a waveform by HiFi-GAN. Every component builds on established open-source modules; the system achieves competitive performance through careful integration and data curation rather than architectural novelty. 3.2 Speech Tokenizer We directly adopt the single-codebook speech tokenizer of CosyVoice 3 [26], which is based on Finite Scalar Quantization (FSQ). Unlike multi-codebook residual vector quantization (RVQ) approaches that require hierarchical prediction modules, this design offers a favorable trade-off between simplicity and information density. 5
PilotTTS
Alibaba Group, 2026
In the FSQ module, an intermediate representation 𝐻 is projected into a 𝐷-dimensional low-rank space, where each dimension is independently quantized into [−𝐾, 𝐾] via a bounded rounding operation: 𝐻˜ = ROUND(Proj𝑑𝑜𝑤𝑛 (𝐻)), ˜ 𝐻ˆ = Proj (𝐻), 𝑢𝑝
(1) (2)
yielding a discrete codebook of size (2𝐾 + 1)𝐷 = 6,561. The token index 𝜇𝑖 for each frame is computed as: 𝜇𝑖 =
𝐷−1 Õ
ℎ̃ 𝑖,𝑗 · (2𝐾 + 1) 𝑗 ,
(3)
𝑗=0
where ℎ̃ 𝑖,𝑗 denotes the 𝑗-th quantized dimension of frame 𝑖. The tokenizer operates at 25 Hz, producing one discrete token per 40 ms of speech. To obtain semantically rich representations, CosyVoice 3 [26] additionally employs multi-task training across five objectives: automatic speech recognition (ASR), language identification (LID), speech emotion recognition (SER), audio event detection (AED), and speaker analysis (SA). This diverse supervision encourages the tokens to encode not only linguistic content but also paralinguistic attributes such as emotion, speaker characteristics, and acoustic events. We reuse this pretrained tokenizer in PilotTTS without further modification. 3.3 Autoregressive Text-to-Semantic Module 3.3.1 Decoupled Speaker and Style Encoding
The autoregressive module generates discrete semantic tokens from text input conditioned on reference audio. A key design question is how to extract speaker-related information from the reference signal. Existing approaches follow two paradigms: audio token continuation [1, 4] and speaker embedding. The former directly leverages fine-grained acoustic cues for high-fidelity cloning, but its quality degrades with noisy or short prompts, and long prompts substantially increase inference cost; the latter improves robustness by compressing the reference into a fixed-dimensional vector, but inevitably discards timbral details and fails to capture dynamic style features such as speaking rate and prosodic contours. To address the limitations of both paradigms, our conditioning mechanism is designed to (1) extract speaker information robustly from reference audio of arbitrary length, quality, and accent, and (2) jointly capture global timbre and dynamic speaking style. To this end, we adopt a Q-Former-based [13, 27] conditioner to extract style condition through crossattention. Specifically, the input to the conditioner is the output of a frozen -Bert 2.0 Encoder [28], which is pretrained on large-scale data that produces semantically rich representations. To supplement global timbre information, we additionally introduce a frozen CAMPPlus [25] encoder that extracts a static speaker embedding. Since the speaker embedding already carries speaker identity, it allows the Q-Former conditioner to focus on extracting dynamic speaking style. 3.3.2 Autoregressive Generation
We formulate text-to-semantic prediction as a conditional autoregressive task with input sequence: x = [s, c, 𝑒BT , |lang|, |emo|, eText , 𝑒ET , 𝑒BA , eAudio , 𝑒EA ],
(4)
where s is speaker embedding from CAMPPlus, c = {c𝑖 }32 is style condition from the Q-Former 𝑖=1 conditioner, 𝑒BT /𝑒ET and 𝑒BA /𝑒EA are boundary markers for text and audio regions respectively, |lang| and |emo| are control tags specifying target language and emotion, and eText /eAudio are the text and audio token embeddings. The model predicts: 𝑝(eAudio | x<Audio ) =
𝑁𝑠 Ö
𝑝(eAudio,𝑖 | x<Audio , eAudio,<𝑖 ).
𝑖=1
6
(5)
PilotTTS
Alibaba Group, 2026
To disentangle speaker-related attributes from linguistic content, we adopt a cross-sample paired training strategy: for each training sample, a different utterance from the same speaker is used as the reference for extracting the speaker embedding s and style condition c. This strategy forces the conditioner to encode only speaker-related attributes which are independent of content. This disentanglement also serves as the foundation for downstream emotion control and dialect synthesis. 3.3.3 Emotion Control
The proposed conditioner captures rich speaker-related attributes from the reference audio, including emotional characteristics. The pretrained language model backbone further contributes implicit emotion inference from textual context. However, implicit control lacks precision and stability. We therefore introduce explicit emotion control via post-training on approximately 2,200 hours of emotion-labeled data. The unified label system supports seven primary categories, namely happy, sad, angry, fear, contempt, serious, and surprise, together with four extended categories: concern, blue (melancholy), disgust, and psychology (inner monologue). 3.3.4 Paralinguistic Generation
In our setting, we focus on four paralinguistic phenomena that we consider most important for our natural speech generation: laughter, breathing, crying, and coughing. Our model supports these four behaviors plus a wrapped laughter mode (LAUGH_SPAN), in which laughter is temporally coupled with speech throughout an utterance. This design enables more precise control of when and how laughter co-occurs, thereby improving synthesis accuracy and consistency. Unlike prior approaches limited to fixed realization patterns, our model operates in two modes. In implicit mode, it infers appropriate paralinguistic behavior from textual context, producing varied laughter types (restrained chuckle, soft giggle, hearty laughter) depending on the setting. In explicit mode, users specify behaviors through textual onomatopoeia. Both modes support zero-shot synthesis with arbitrary speaker prompts. These capabilities are realized through supervised fine-tuning on approximately 200 hours of paralinguistic data from the data processing outputs, and internal collections. 3.3.5 Dialect Synthesis
Dialect synthesis generates target-dialect speech using Mandarin audio as the speaker prompt and a dialect tag (|lang|) to specify the variety. The central challenge is data scarcity: dialect recordings are far less abundant than Mandarin. We observe that after bilingual pre-training, the model generates Mandarin robustly even when the prompt contains dialectal features. Exploiting this, we construct parallel data by synthesizing three Mandarin utterances per dialect speaker using the pretrained model, yielding large-scale “dialect– Mandarin” pairs that alleviate sparsity. During dialect fine-tuning on approximately 16,000 hours of dialect data, we employ mixed-prompt sampling: the target is always a dialect utterance, while the conditioning prompt is drawn with equal probability from either a Mandarin or dialect utterance of the same speaker. This forces the model to extract speaker identity from stylistically diverse prompts and map it to the target dialect, rather than copying prompt style directly. 3.4 Speech Decoder The speech decoder converts discrete semantic codes into mel spectrograms using Conditional Flow Matchtgt ing (CFM) [14] with a DiT [15] backbone of 300M parameters. The conditioning input is [Mref , s, eAudio ], comprising the reference mel spectrogram, CAMPPlus speaker embedding, and predicted semantic features. Through 10-step iterative denoising, the decoder progressively transforms a noise sample into the target mel spectrogram conditioned on these inputs. The generated mel spectrogram is then converted to a waveform by HiFi-GAN [16].
7
PilotTTS
Alibaba Group, 2026
Method
test-zh
test-en
CER (%)↓
SIM↑
WER (%)↓
SIM↑
Seed-TTS [3] F5-TTS [7] FireRedTTS-2 [6] CosyVoice-3-0.5B [26] VoxCPM-0.5B [29] Qwen3-TTS-25Hz-0.6B [10] MiniMax-Speech [11] VibeVoice-1.5B [30]
1.12 1.56 1.14 1.16 0.93 1.18 0.83 1.16
0.796 0.741 0.736 0.780 0.772 – – 0.744
2.25 1.83 1.95 2.02 1.85 1.64 1.65 3.04
0.762 0.647 0.655 0.718 0.729 – – 0.689
PilotTTS (Ours)
0.87
0.862
1.50
0.815
Table 1: Zero-shot speech generation results on the Seed-TTS Eval benchmark [3]. CER and WER are evaluated using Paraformer-zh [20] and Whisper [22], respectively. SIM is the cosine similarity between speaker embeddings. Best results are in bold and second-best are underlined. “–” indicates unreported scores.
4 Experiments
4.1 Experimental Setup Training Data. Pre-training uses approximately 200,000 hours of Chinese and English speech data collected from publicly available sources and processed using the pipeline described in Section 2. Posttraining data is organized by capability: (1) Emotion: approximately 2,200 hours of emotion-labeled speech (1,000 hours high-quality and 1,200 hours augmented), drawn from open-source datasets, internal annotations, and model-augmented sources; (2) Paralinguistic: approximately 200 hours from the outputs of the data processing pipeline and internal collections; (3) Dialect: 16,000 hours covering 14 Chinese dialects, all derived from dialectal ASR corpora. The autoregressive module is built upon Qwen3-0.6B [12], and the CFM decoder (Section 3.4) contains approximately 300 million parameters. Evaluation Metrics. For zero-shot speech generation, we evaluate on the Seed-TTS Eval benchmark [3], reporting character error rate (CER) for Chinese and word error rate (WER) for English to measure content accuracy, along with speaker similarity (SIM) to assess voice cloning fidelity. CER is computed using Paraformer-zh [20], WER using Whisper [22], and SIM measured by calculating the cosine similarity between speaker embeddings. For emotion control, paralinguistic synthesis, and dialect synthesis, we conduct human evaluations with criteria detailed in the respective subsections. Baselines. For zero-shot evaluation, to ensure fair comparison, we primarily select baselines with a parameter scale around 0.6B and without post-training techniques. We compare against eight systems spanning diverse architectures and data scales: Seed-TTS [3], F5-TTS [7], FireRedTTS-2 [6], CosyVoice-30.5B [26], VoxCPM-0.5B [29], Qwen3-TTS-25Hz-0.6B [10], MiniMax-Speech [11], and VibeVoice-1.5B [30]. For emotion control, we compare with four systems that support explicit emotion conditioning: VoxCPM [29], Fish-Speech S2 [31], IndexTTS [32], and CosyVoice 3 [26]. For paralinguistic synthesis, we compare with CosyVoice 3 [26] and Fish-Speech S2 [31], the two most advanced open-source systems that support paralinguistic generation. For Chinese dialect synthesis evaluation, we find that the results are predominantly influenced by human subjectivity. Therefore, to prevent uncontrolled bias stemming from subjective judgments, we exclude comparison baselines in this setting.
8
PilotTTS
Alibaba Group, 2026
Category
VoxCPM
Fish-Speech S2
IndexTTS
CosyVoice3
PilotTTS
Primary Emotions Happy Sad Fear Angry Contempt Serious Surprise
14.5 21.8 18.2 45.5 32.7 20.0 29.1
41.8 67.3 50.9 40.0 61.8 61.8 96.4
23.6 7.3 27.3 25.5 – – 10.9
81.8 96.4 80.0 80.1 88.2 90.9 69.1
86.4 90.5 83.2 89.0 81.2 93.2 93.2
Extended Emotions Blue Concern Disgust Psychology
58.2 67.3 20.0 23.6
32.7 81.8 34.5 92.7
49.1 – 47.3 –
86.4 83.6 52.7 98.2
79.1 82.9 65.5 98.2
Avg. (Primary) Avg. (All)
26.0 31.9
60.0 60.2
– –
83.8 82.5
88.1 85.7
Table 2: Emotion control evaluation results (success rate, %). A sample is counted as successful only when both speaker timbre is preserved and the target emotion is clearly perceivable. Best results per category are in bold and second-best are underlined. “–” indicates unsupported categories. Condition
VoxCPM
Fish-Speech S2
IndexTTS
CosyVoice3
PilotTTS
Without emotion control With emotion control
0.4982 0.3361
0.5727 0.5731
0.7680 0.4233
0.7963 0.6940
0.8101 0.7329
Table 3: Speaker similarity under different emotion control settings. Higher is better. Best results are in bold and second-best are underlined.
4.2 Zero-Shot Speech Generation Table 1 presents the zero-shot speech generation results on the Seed-TTS Eval benchmark. PilotTTS achieves the highest speaker similarity on both test sets (0.862 on test-zh and 0.815 on test-en), substantially outperforming all baselines with reported SIM scores. On content accuracy, PilotTTS attains a CER of 0.87% on test-zh, ranking second only to MiniMax-Speech (0.83%) with a margin of merely 0.04%, and achieves the lowest WER of 1.50% on test-en. We note that Qwen3-TTS and MiniMax-Speech do not publicly report SIM scores, preventing a complete comparison on speaker similarity for these systems. These results are particularly notable given that PilotTTS uses only approximately 200,000 hours of training data, substantially less than several competing systems that leverage larger-scale proprietary corpora. We attribute this data efficiency to two factors: (1) the rigorous data processing pipeline ensures high training data quality, and (2) the decoupled conditioning mechanism based on Q-Former and CAMPPlus effectively leverages reference audio for speaker modeling. The strong SIM improvements (+0.066 over the second-best Seed-TTS on test-zh, and +0.053 on test-en) suggest that the dual-pathway conditioning design captures both stable speaker identity and dynamic speaking style more effectively than single-pathway approaches. 4.3 Emotion Control We evaluate emotion control using 51 speaker prompts: 15 expressive voices drawn from anime and film characters, and 36 ordinary speakers (18 male, 18 female). Human evaluators listen to three utterances for each test case: the original speaker prompt, a neutral synthesis (without emotion control), and an emotion-controlled synthesis. A sample is counted as successful only when both conditions are met: (1) the speaker timbre remains consistent with the original prompt, and (2) the target emotion is clearly
9
PilotTTS
Alibaba Group, 2026
Method PilotTTS (Ours) CosyVoice 3 [26] Fish-Speech S2 [31]
LAUGH
COUGH
BREATH
Overall
LAUGH_SPAN
CRY
97.6 83.3 54.8
64.3 59.5 64.3
81.0 95.2 83.3
85.1 80.4 64.3
94.6 – –
61.9 – –
Table 4: Paralinguistic synthesis success rates (%). LAUGH_SPAN and CRY are unique capabilities of PilotTTS not supported by the baseline systems. Best results per category are in bold. “–” indicates unsupported categories. Method PilotTTS (Ours)
Same-Dialect
Mandarin-to-Dialect
Cross-Dialect
91.80
86.46
85.38
Table 5: Dialect synthesis accuracy (%) across three evaluation scenarios. A synthesized sample is considered a failure case if the proportion of Mandarin pronunciation exceeds 10%.
recognizable in the synthesized speech. Table 2 compares emotion control success rates across five systems. On primary emotions, PilotTTS achieves the highest average success rate of 88.1%, surpassing CosyVoice 3 (83.8%). On the overall average across all eleven categories, CosyVoice 3 leads marginally (81.4% vs. 80.2%), largely due to stronger performance on extended emotion categories. Notably, IndexTTS supports only seven of the eleven categories, and its timbre variation under emotion conditioning further lowers its success rates under our joint evaluation criterion. Table 3 evaluates speaker similarity under emotion control. PilotTTS achieves the highest speaker similarity both without emotion control (0.8101) and with emotion control (0.7329), and exhibits the smallest drop between the two conditions among all systems. This indicates that our decoupled conditioning design effectively modulates emotional expressiveness while preserving speaker timbre. 4.4 Paralinguistic Synthesis We construct a dedicated test set covering four categories of paralinguistic phenomena: laughter (LAUGH), breathing (BREATH), crying (CRY), and coughing (COUGH), along with wrapped laughter (LAUGH_SPAN). For each category, we use 21 distinct speaker prompts to reflect diverse speaker timbres and speaking styles for zero-shot synthesis. Human evaluators judge whether the target paralinguistic behavior is successfully generated in each sample. We compare against CosyVoice 3 [26] and Fish-Speech S2 [31] on the three commonly supported categories. As shown in Table 4, PilotTTS achieves an overall success rate of 85.1% across the three common categories, outperforming CosyVoice 3 (80.4%) and Fish-Speech S2 (64.3%). On LAUGH, PilotTTS attains 97.6%, substantially exceeding both CosyVoice 3 (83.3%) and Fish-Speech S2 (54.8%). For BREATH, CosyVoice 3 achieves the highest success rate (95.2%), while PilotTTS reaches 81.0%. Among the three categories, COUGH is the most difficult for all systems, with success rates remaining around 60%. This is due to the substantial acoustic variability of cough events and their limited presence in the training data. Beyond the three commonly supported categories, PilotTTS uniquely supports LAUGH_SPAN and CRY, not available in baseline systems. LAUGH_SPAN achieves a success rate of 94.6%, demonstrating the model’s ability to maintain coherent speech while simultaneously generating natural laughter throughout the utterance. CRY reaches 61.9%, providing a paralinguistic capability that is largely absent from existing open-source TTS systems. These additional modes extend the expressiveness of the synthesis system and are enabled by the targeted SFT training strategy described in Section 3.3.4. 4.5 Dialect Synthesis To comprehensively evaluate dialect synthesis, we construct a test set covering three scenarios of increasing difficulty: (1) Same-Dialect, where both the reference audio and the target speech belong to the same dialect; (2) Mandarin-to-Dialect, where standard Mandarin audio serves as the speaker prompt to generate 10
PilotTTS
Alibaba Group, 2026
Content Accuracy (%)↓ Test Set
Full
test-zh (CER) 1.130 test-en (WER) 1.940 test-hc (CER) 7.830
w/o spk w/o both 1.022 1.860 8.866
1.412 2.710 10.623
Speaker Similarity↑ Full
w/o spk w/o both
0.8626 0.8157 0.8470
0.8594 0.8143 0.8355
0.8617 0.8027 0.8435
Table 6: Ablation study on the conditioning components of the autoregressive module. Full uses both the CAMPPlus speaker embedding (spk) and the Q-Former condition tokens (conds); w/o spk removes the speaker embedding while keeping condition tokens; w/o both removes both, leaving only the text input. All models are trained for the same number of steps (200K) on a cleaned 60K-hour subset of the full 200K-hour dataset. We report CER on test-zh, WER on test-en, and CER on the Seed-TTS hard-case subset (test-hc), together with speaker similarity (SIM).
target dialect speech; and (3) Cross-Dialect, where the reference audio is in dialect A while the target output is in dialect B. For each dialect, male and female speakers are randomly selected to form the test set. We adopt dialect control accuracy as the primary metric through subjective listening evaluation, where a synthesized sample is counted as a failure case if the proportion of non-target-dialect pronunciation exceeds 10%. As shown in Table 5, PilotTTS achieves 91.8% accuracy in the Same-Dialect scenario, demonstrating reliable dialect generation when the reference and target share the same dialect. For the more challenging Mandarin-to-Dialect setting, the system attains 86.46%, confirming that the mixed-prompt training strategy (Section 3.3.5) effectively enables cross-lingual style transfer from standard Mandarin prompts. The Cross-Dialect scenario reaches 85.38%, indicating robust speaker identity extraction even when the source and target dialects differ. These results validate the effectiveness of our parallel data construction and mixed-prompt sampling approach in alleviating dialect data scarcity. 4.6 Ablation Study on Conditioning Components To assess the individual contribution of each conditioning component in the autoregressive module (Section 3.3.1), we conduct an ablation study on a cleaned 60K-hour subset of the full 200K-hour dataset. Three settings are compared, each trained for the same number of optimization steps (200K) to ensure fairness: (i) Full, the proposed dual-pathway design with both the CAMPPlus speaker embedding (s) and the Q-Former condition tokens (c); (ii) w/o spk, which removes s while keeping c; and (iii) w/o both, which removes both conditioning signals, leaving the model conditioned only on text. We evaluate on three test sets: the Chinese (test-zh) and English (test-en) subsets of Seed-TTS Eval, and the Seed-TTS hard-case subset (test-hc) that contains acoustically challenging samples. Content accuracy (CER/WER) and speaker similarity (SIM) are reported in Table 6. Condition tokens are indispensable for content accuracy. Removing the Q-Former condition tokens leads to a substantial degradation in pronunciation accuracy across all three test sets. The effect is most pronounced on test-hc, where CER rises from 7.83% to 10.62% (a relative increase of approximately 35%). On test-zh and test-en, content errors also increase from 1.13% to 1.41% and from 1.94% to 2.71%, respectively. These results indicate that the fine-grained content- and prosody-related cues carried by the Q-Former condition tokens are essential for stable autoregressive generation, particularly under challenging acoustic conditions, and therefore cannot be removed without significantly compromising synthesis quality. Speaker embedding plays a complementary role. The effect of the CAMPPlus speaker embedding is more nuanced. Under a fixed training budget, removing s slightly reduces content errors on test-zh and test-en, as the model can allocate more capacity to exploiting condition tokens; we observe that this gap narrows further as training progresses. In contrast, the speaker embedding consistently improves speaker similarity across all three test sets, with the largest gain observed on test-hc (0.8355 → 0.8470). Beyond 11
PilotTTS
Alibaba Group, 2026
directly enriching timbre information, s also encourages the Q-Former condition tokens to specialize in timbre-independent prosodic and stylistic cues, which is consistent with the stronger robustness observed on the hard-case subset. Overall, the dual-pathway design strikes a desirable balance between content accuracy, speaker fidelity, and robustness.
5 Conclusion and Future Work We have presented PilotTTS, a text-to-speech system built upon publicly available modules with a design philosophy that prioritizes integration and data engineering over architectural novelty. A data processing pipeline assembled entirely from publicly available tools enables the system to achieve competitive zero-shot performance with only approximately 200,000 hours of training data. On the Seed-TTS Eval benchmark, PilotTTS attains the highest speaker similarity on both Chinese and English test sets and the lowest word error rate on the English set. The decoupled conditioning mechanism further enables emotion control, paralinguistic generation, and cross-dialect synthesis, all through targeted post-training. Despite these results, several limitations remain and point to future directions: 1. Insufficient explicit style modeling. Our architecture does not include a dedicated style modeling module with strong representational capacity; instead, it relies on the Q-Former condition to implicitly capture style-related factors. As a result, the synthesis quality may be limited in the granularity of expressive details. To address this limitation, we are developing a representation module that jointly models global style and local (fine-grained) style, aiming to enhance the expressiveness of generated audio. 2. Single-codebook quantization ceiling. Although the single-codebook FSQ tokenizer is architecturally simpler and is generally sufficient for speech, it possesses a lower information-capacity ceiling than multi-codebook residual vector quantization or continuous latent representations. As a result, it becomes more challenging to extend the model to more complex scenarios such as singing and background music. 3. Lossy mel-spectrogram reconstruction. The decoding pipeline relies on mel spectrograms and a separate vocoder, an indirect reconstruction process that may introduce additional distortion compared to end-to-end waveform generation. Future work will focus on introducing an explicit style encoder, exploring higher-capacity quantization strategies, and adopting end-to-end waveform generation architectures. All code and model weights will continue to be released as open source.
References [1] Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing, 33:705–718, 2025. [2] Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, Jiang Bian, et al. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In International conference on learning representations, volume 2024, pages 698–722, 2024. [3] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024. [4] Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue
12
PilotTTS
Alibaba Group, 2026
Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024. [5] Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10117, 2024. [6] Kun Xie, Feiyu Shen, Junjie Li, Fenglong Xie, Xu Tang, and Yao Hu. Fireredtts-2: Towards long conversational speech generation for podcast and chatbot. arXiv preprint arXiv:2509.02020, 2025. [7] Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, JianZhao JianZhao, Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6255–6271, 2025. [8] Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du, Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, et al. Ditar: Diffusion transformer autoregressive modeling for speech generation. In International Conference on Machine Learning, pages 27255–27270. PMLR, 2025. [9] Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. In International Conference on Machine Learning, pages 22605–22623. PMLR, 2024. [10] Hangrui Hu, Xinfa Zhu, Ting He, Dake Guo, Bin Zhang, Xiong Wang, Zhifang Guo, Ziyue Jiang, Hongkun Hao, Zishan Guo, et al. Qwen3-tts technical report. arXiv preprint arXiv:2601.15621, 2026. [11] Bowen Zhang, Congchao Guo, Geng Yang, Hang Yu, Haozhe Zhang, Heidi Lei, Jialong Mai, Junjie Yan, Kaiyue Yang, Mingqi Yang, et al. Minimax-speech: Intrinsic zero-shot text-to-speech with a learnable speaker encoder. arXiv preprint arXiv:2505.07916, 2025. [12] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [13] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023. [14] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In 11th International Conference on Learning Representations, ICLR 2023, 2023. [15] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023. [16] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: generative adversarial networks for efficient and high fidelity speech synthesis. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 17022–17033, 2020. [17] Alexis Plaquet and Hervé Bredin. Powerset multi-class cross entropy loss for neural speaker diarization. In 24th Interspeech Conference (INTERSPEECH 2023), pages 3222–3226. ISCA, 2023. [18] Hervé Bredin. pyannote. audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In 24th Interspeech Conference (INTERSPEECH 2023), pages 1983–1987. ISCA, 2023. [19] Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6493–6497. IEEE, 2021. 13
PilotTTS
Alibaba Group, 2026
[20] Zhifu Gao, ShiLiang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. In Proc. Interspeech 2022, pages 2063–2067, 2022. [21] Kai-Tuo Xu, Feng-Long Xie, Xu Tang, and Yao Hu. Fireredasr: Open-source industrial-grade mandarin speech recognition models from encoder-decoder to llm integration. arXiv preprint arXiv:2501.14350, 2025. [22] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023. [23] Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, et al. Qwen3-asr technical report. arXiv preprint arXiv:2601.21337, 2026. [24] Yafeng Chen, Siqi Zheng, Hui Wang, Luyao Cheng, Tinglong Zhu, Rongjie Huang, Chong Deng, Qian Chen, Shiliang Zhang, Wen Wang, et al. 3d-speaker-toolkit: An open-source toolkit for multimodal speaker verification and diarization. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025. [25] Hui Wang, Siqi Zheng, Yafeng Chen, Luyao Cheng, and Qian Chen. Cam++: A fast and efficient network for speaker verification using context-aware masking. In Proc. Interspeech 2023, pages 5301–5305, 2023. [26] Zhihao Du, Changfeng Gao, Yuxuan Wang, Fan Yu, Tianyu Zhao, Hao Wang, Xiang Lv, Hui Wang, Chongjia Ni, Xian Shi, et al. Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training. arXiv preprint arXiv:2505.17589, 2025. [27] Sungkyung Kim, Adam Lee, Junyoung Park, Andrew Chung, Jusang Oh, and Jay-Yoon Lee. Towards efficient visual-language alignment of the q-former for visual reasoning tasks. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15155–15165, 2024. [28] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250. IEEE, 2021. [29] Yixuan Zhou, Guoyang Zeng, Xin Liu, Xiang Li, Renjie Yu, Ziyang Wang, Runchuan Ye, Weiyue Sun, Jiancheng Gui, Kehan Li, et al. Voxcpm: Tokenizer-free tts for context-aware speech generation and true-to-life voice cloning. arXiv preprint arXiv:2509.24650, 2025. [30] Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, et al. Vibevoice technical report. arXiv preprint arXiv:2508.19205, 2025. [31] Shijia Liao, Yuxuan Wang, Songting Liu, Yifan Cheng, Ruoyi Zhang, Tianyu Li, Shidong Li, Yisheng Zheng, Xingwei Liu, Qingzheng Wang, et al. Fish audio s2 technical report. arXiv preprint arXiv:2603.08823, 2026. [32] Wei Deng, Siyi Zhou, Jingchen Shu, Jinchao Wang, and Lu Wang. Indextts: An industrial-level controllable and efficient zero-shot text-to-speech system. arXiv preprint arXiv:2502.05512, 2025.
14