Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions Oluwanifemi Bamgbose* , Simon Rosen, Jash Shah, Lindsay Devon Brin, Hoang H Nguyen, Anke Koelzer, Rachel Hansen, Tara Bogavelli, Fanny Riols ServiceNow nifemi.bamgbose@servicenow.com

arXiv:2608.09930v1 [cs.SD] 10 Aug 2026

Abstract Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (AudioLLMs) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLMs judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLMs judges show selective, promptdependent detection that does not generalise across all dimensions. Neither class reliably captures a breadth of linguistically structured speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.

1

Introduction

Text-to-speech (TTS) systems are increasingly deployed in production contexts where the naturalness of synthesized speech directly affects user experience and comprehension, with potential consequences for user engagement (Tan et al., 2021; Xie et al., 2025). Listeners are sensitive to a multidimensional set of failures; a TTS system may stress the wrong syllable in a word, break a sentence at an unnatural point, or deliver a question with the flat intonation of a statement, any of which would be immediately human-perceptible (Geneva et al., 2023; Gutierrez et al., 2021; Lee et al., 2026). The gold standard for measuring TTS quality across these dimensions remains human perceptual judgment (Baughan et al., 2023; Ulgen et al., 2026). * Corresponding author.

Table 1: Capability comparison with recent metaevaluation resources. ETE: EmergentTTS-Eval (Manku et al., 2025); ITE: InstructTTSEval (Huang et al., 2025); SJ: SpeechJudge (Zhang et al., 2025); SS: SpeakerSleuth (Lee et al., 2026). ✓ present, ~ partial, ✗ absent; partial = human labels only validate a model judge (ETE/ITE), or controlled difficulty without injected linguistic errors (SS). Only ours pairs expert, aspect-level human labels with a linguistically-grounded taxonomy and controlled error injection, auditing both MOS predictors and Audio-LLM judges. Capability Human-annotated ground truth Dimension-level annotation per sample Linguistically-grounded aspect taxonomy Covers MOS predictors & Audio-LLM judges

Ours

ETE

ITE

SJ

SS

✓ ✓ ✓ ✓

~ ✗ ✗ ✗

~ ✗ ✗ ✗

✓ ✗ ✗ ✓

✓ ✗ ✗ ✓

As deployment scales, however, relying solely on human judgment becomes expensive and time consuming. Two main paradigms have emerged to automatically evaluate speech quality at scale: neural Mean Opinion Score (MOS) predictors (Mittag et al., 2021; Saeki et al., 2022), trained to output individual, holistic quality scores that reflect human MOS; and Audio Large Language Model (AudioLLM) Judges (Zhang et al., 2025; Manakul et al., 2026), which prompt audio-capable LLMs to rate or compare audio samples. A good speech evaluator should respond to all aspects of perceptual quality to which humans attend, yet it is not clear whether models of either type do so (i.e whether individual output scores align with quality along any or all human-perceptible dimensions). For AudioLLMs in particular, it additionally remains an open question whether they even have the underlying capability, and/or can be prompted, to attend to each dimension. To better understand and identify the aspects of speech to which MOS models and Audio-LLMs actually attend, we construct the first meta-evaluation benchmark for automated TTS evaluators that has annotations based on linguistically-grounded, human-perceptible speech dimensions. Our contributions are as follows:

Figure 1: Dataset Construction Pipeline. Sentences from three sources (Harvard Sentences, EmergentTTS-Eval, synthetic text) are fed through three error-generation pathways: LLM-modified plain text, LLM-modified IPA transcriptions, and direct acoustic manipulation. All pathways pass through the TTS model to produce clean (Score 1) and error-targeted samples. Samples are then annotated by three linguists on all ten dimensions. After linguist annotation, majority filtering and downsampling produce a balanced final dataset of 860 samples.

• Annotation Schema. A linguistically grounded evaluation schema decomposing naturalness into 10 phonologically and phonetically grounded dimensions across word, prosodic, and paralinguistic levels. This enables future evaluation to be aligned with human perception of speech quality. • Meta-Evaluation Dataset. A per-dimension annotated dataset of 860 utterances, labeled by trained linguist raters across the 10 dimensions. • Dimension-Level Audit. An evaluation of four neural MOS models as well as four AudioLLMs judges across four prompting conditions, with and without reference transcripts. This reveals uneven phonetic, prosodic and paralinguistic sensitivity, with neither MOS predictors nor Audio-LLMs judges reliably detecting errors across these dimensions under any condition tested. Our benchmark enables researchers and practitioners to attribute TTS system failures to specific interpretable dimensions rather than opaque holistic scores. This provides a diagnostic foundation that prior evaluation frameworks have lacked.

2

Related Works

Automated TTS Evaluation. The dominant paradigm for scalable TTS evaluation has been MOS predictors: learning to approximate human naturalness scores from raw waveforms without a reference signal. Through the VoiceMOS Challenge series (2022–2024) (Huang et al., 2022, 2024), Semi-supervised Learning (SSL)-based MOS predictors (Patton et al., 2016; Reddy et al., 2021; Saeki et al., 2022; Baba et al., 2024) have demonstrated strong system-level correlation with human judgments, approaching the practical ceil-

ing of aggregate naturalness scoring. However, these methods produce holistic quality signals without grounding them in independently interpretable linguistic attributes, leaving no diagnostic path when a system performs poorly. More recently, researchers have explored leveraging AudioLLMs as richer judges for TTS evaluation, evaluating speaking style (Chiang et al., 2025), speech quality (Zhang et al., 2025; Manakul et al., 2026; Monjur and Nirjon, 2025), and speaker consistency (Lee et al., 2026). Yet, LLM-based judges inherit the same limitation: scores remain holistic, and the perceptual dimensions driving any given rating are opaque. TTS Evaluation Benchmarks. Several benchmarks have moved beyond scalar MOS towards richer TTS evaluation (Manku et al., 2025; Huang et al., 2025), yet share a fundamental limitation: evaluation dimensions are organized around predefined scenario categories rather than a principled speech quality schema. EmergentTTSEval (Manku et al., 2025) introduces a modelas-a-judge benchmark spanning six pre-defined scenario types (emotions, paralinguistics, foreign words, syntactic complexity, complex pronunciation, and questions) and demonstrates that AudioLLMs judges can distinguish broad scenario-level variation. InstructTTS-Eval (Huang et al., 2025) takes a complementary direction, centralizing evaluations on expressive and style-following capability of TTS systems across pre-defined dimensions such as speaking rate, pitch, and emotional expressiveness. However, neither provides humanannotated ground truth at the attribute level, making it impossible to audit whether automated evalua-

tors track the aspects listeners actually perceive. We bridge this gap with a human-annotated dataset grounded in 10 interpretable aspects across three linguistically-motivated quality dimensions, paired with a calibrated annotation framework, enabling direct attribution of TTS system failures to specific and interpretable aspects. More information related to other benchmarks can be found at Table 1

3

Benchmark Design

3.1 Evaluation Schema Current TTS evaluation conflates perceptually distinct failure types under a single quality judgment. Terms such as “naturalness,” “fluency,” and “clarity” are used interchangeably in MOS elicitation, yet listeners explain their ratings by invoking factors ranging from pronunciation accuracy to affective delivery (Kirkland et al., 2023), aspects that are not only perceptually distinct but defined at different levels of linguistic structure. Drawing on Crystal’s distinction between linguistic and paralinguistic features (Crystal, 1969), we organize evaluation around three levels corresponding to where each property is linguistically specified: the lexicon, the utterance, and the speaker. We propose a ten-dimension schema in which each attribute is operationally defined and perceptually separable (see Table 5). 3.1.1 Word level. Quality at the word level is determined by whether each word contains the correct sounds and carries stress on the correct syllable. We organize this level around fixed properties of lexical entries, including both phoneme identity and stress position. Errors at this level (whether phonetic or stress-related) constitute word identity errors regardless of sentence context (Jesse et al., 2017). Phonetic accuracy: Whether the sounds in each word fall within the acceptable range of realization for their lexical targets. Lexical stress: Whether primary stress falls on the correct syllable of each word, including appropriate vowel reduction in unstressed syllables. 3.1.2 Prosodic level. Quality at the prosodic level is determined by whether suprasegmental properties are appropriate for the syntactic and semantic properties of the utterance. Prosody is a perceptually separable dimension that listeners isolate from overall speech quality (Gutierrez et al., 2021) and one where current systems remain measurably below

natural speech (Vallés-Pérez et al., 2021). We define the dimensions below based on linguistic function. Intonation: The pitch contour over phrases and sentences, encoding utterance type (e.g. question, statement) and discourse structure (e.g. topic boundaries, turn-taking signals). Prosodic stress: Prominence on specific words or phrases relative to the rest of the utterance, determined by information structure. Prosodic boundary appropriateness: The chunking of the utterance into prosodic phrases aligned with syntactic structure, realized through boundary tones, lengthening, and pause. Speech rate appropriateness: The overall pace of delivery, which can fail without any corresponding failure in pitch or prosodic structure. 3.1.3 Paralinguistic level. Quality at the paralinguistic level is determined by whether the speech conveys speaker characteristics appropriate to the intended speaker and content. Although paralinguistic dimensions share audio properties with prosody, such as pitch and timing, they convey information about the speaker rather than the linguistic message. Following Crystal (1969), we distinguish speaker-stable voice quality features from affective features that modulate in response to content and context (see Appendix 6.4). Where no external reference speaker exists, dimensions are assessed relative to the system’s own output. Emotional appropriateness: The degree to which the emotional tone matches the affective content of the utterance. Expressiveness: Natural variation in delivery style and vocal energy, assessed relative to the system itself. Human plausibility: The degree to which vocal qualities fall within the range of physically realizable human production. Speaker identity consistency: Stability of perceived vocal characteristics across the utterance, assessed relative to the system itself. 3.2 Dataset Construction Our dataset consists of 860 samples of annotated and majority-voted audios. We generated synthetic speech using Cartesia Sonic-3 and introduced controlled errors through text modification and acoustic manipulation. Below we describe our source corpora, annotation scale, error generation methods, and human annotation procedure.

Table 2: Error generation method, sample counts, and inter-annotator agreement per dimension. n is the balanced evaluation set after majority-class downsampling to a 50/50 positive/negative split (positives = negatives = n/2 for every dimension). Krippendorff’s α is reported on this set with 95% bootstrap CIs. Total: n = 860. Level

Dimension

Method

Text Source

Manipulation

n

Krippendorff’s α

Word

Phonetic Accuracy Lexical Stress

LLM-altered IPA LLM-altered IPA

Harvard Sentences Harvard Sentences

Phoneme substitution Stress mark transposition

110 90

0.767 [0.660, 0.852] 0.821 [0.728, 0.907]

Prosodic

Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness

Acoustic manipulation LLM-altered plain text LLM-altered plain text Acoustic manipulation

EmergentTTS-Eval Harvard Sentences Synthetically generated Harvard Sentences

F0 contour manipulation Function word quoting Structurally targeted sentences Duration scaling

80 102 88 128

0.469 [0.332, 0.611] 0.658 [0.542, 0.766] 0.727 [0.604, 0.834] 0.707 [0.605, 0.801]

Text–emotion mismatch via Cartesia tag F0 manipulation, loudness compression, vowel shortening Cross-speaker fading Inserted audio distortions and glitches

46

0.596 [0.400, 0.767]

94

0.460 [0.332, 0.597]

66 56

0.713 [0.577, 0.836] 0.710 [0.565, 0.849]

Paralinguistic

Emotional Appropriateness

API-enforced emotion

Synthetically generated

Expressiveness

Acoustic manipulation

Harvard Sentences

Speaker Identity Consistency Human Plausibility

Acoustic manipulation Acoustic manipulation

Harvard Sentences Harvard Sentences

Total

Source corpora. To systematically cover the annotation schema dimensions, we selected utterances from two existing corpora and synthetically generated additional texts where needed (Table 2). Harvard Sentences (IEEE Subcommittee on Subjective Measurements, 1969) provide 720 phonetically balanced sentences covering the English phoneme inventory broadly, wellsuited to word-level and some prosodic dimensions. EmergentTTS-Eval (Manku et al., 2025) provides sentences with complex syntactic structures and varied question types (useful for prosodic boundary and intonation dimensions) alongside sentences with explicitly defined affective content, used for emotional appropriateness. For dimensions lacking appropriate utterances in these sources, we synthetically generated sample texts (Appendix 7.4). Scale design. All dimensions are rated as binary (0/1), where 0 means any error is present and 1 means the audio is perfect, following the annotation schema. Error generation pipeline. For each source sentence, we generated utterances at the intended quality level by introducing targeted errors via one of three pathways (see Figure 1): • LLM-altered IPA. Used for Phonetic Accuracy and Lexical Stress. An LLM (GPT-5) produces a modified IPA transcription instantiating the intended error, passed to Cartesia Sonic-3 (Cartesia AI, 2025) for IPA-driven synthesis (e.g., phoneme substitution: seed /sid/ → /sib/; stress transposition: harvest /"hAr.vIst/ → /hAr."vIst/). • LLM-altered plain text. Used for Prosodic Stress and Prosodic Boundary Placement. For stress, quotation marks around LLM-selected function words induce atypical prominence (e.g., Type out three lists “of” orders); for bound-

860

aries, commas split LLM-selected syntactic constituents (e.g., I dropped my, keys). • API-enforced emotion. Used for Emotional Appropriateness. Sentences with a predetermined emotion are synthesized via Cartesia with deliberately mismatched emotion tags (e.g., My beloved dog passed away quietly in his sleep tonight. with tag euphoric). • Acoustic Manipulation. Used for Intonation, Speech Rate, Expressiveness, Speaker Identity Consistency, and Human Plausibility. Praatbased manipulation (Boersma, 2001) via Parselmouth (Jadoul et al., 2018) introduces targeted perturbations to a clean synthesis baseline: F0 contour manipulation for intonation; duration scaling for speech rate; combined F0, loudness, and vowel duration modification for expressiveness; cross-speaker fading for speaker identity; and inserted distortions for human plausibility. We treat these as controlled upper-bound cases: a judge that cannot detect a deliberately introduced perturbation cannot detect subtler naturally occurring deviations of the same type. The prompts and details of the acosutic modification are provided in Appendix 7.5 and Appendix 7.2 3.3 Human Annotation Raters. Each sample was annotated by three professional linguists with experience in analytical work with natural language data. Three raters is the practical minimum for stable Krippendorff’s α estimation; sample counts per dimension are reported in Table 2. Pilot and guideline refinement. To reduce guideline ambiguity (Uma et al., 2022), we conducted a pilot annotation (n = 50 samples) with all raters and iteratively refined the guidelines after collec-

tive review of disagreements. Procedure. Raters are initially presented with the audio only, but have the option to toggle the transcript. Every sample is rated on all applicable dimensions, not only the dimension it was generated to target. This yields cross-dimensional contamination data and allows us to quantify perceptual spillover: when a sample was generated to elicit a failure on dimension X, what do raters observe on dimension Y? (See Appendix 2) Inter-rater reliability. Table 2 reports Krippendorff’s α (Krippendorff, 2011) on the constructaligned subset (n = 46-128 per dimension), after class balancing, with 95% bootstrap confidence intervals (1117 resamples, percentile method, seed 42). 3.4 Dataset Finalization Aggregation and downsampling. Ground truth annotations for each sample, for each dimension, were determined by majority vote. Each intended (analysis) dimension was downsampled to a balanced dataset of positive and negative samples, randomly sampling from the larger class with priority for consensus annotations. Final dataset statistics, including per-dimension sample counts, are reported in Table 2.

4

Experiments

4.1 Judges Evaluated We evaluate two classes of automated judge that differ in output type and evaluation mechanism. MOS predictors. We include UTMOSv2 (Baba et al., 2024), DNSMOS-Pro (Cumlin et al., 2024), NISQA (Mittag et al., 2021), and AudioboxAesthetics (Tjandra et al., 2025). Each produces either a single continuous scalar per audio or dimension-wise scores. Audio-LLMs judges. We evaluate a set of opensource and closed audio large language models: Gemini 3.5 Flash , Gemini 3 Flash, Qwen3Omni-30B-A3B and Step-Audio-R2-Mini, AudioLLMs judges are prompt-sensitive, motivating the systematic prompting conditions described in Section 4.2. 4.2 Prompting Strategy Conditions We define four prompting conditions varying in the amount of schema guidance provided and the granularity of the expected output. In all conditions, models produce binary labels matching the groundtruth annotation format. Each judge is queried three

times per sample per condition, with the majority label taken as the final prediction. All conditions are run with and without the reference transcript. Condition 1: Underspecified MOS-style prompt. The model is prompted to rate the "naturalness" of the sample without a detailed schema, producing a single holistic score analogous to a neural MOS predictor. This condition establishes what the judge attends to in the absence of detailed task guidance, and serves as the primary baseline. Condition 2a: Schema-guided prompt, single score. The model is prompted with the full 10dimension schema but instructed to produce a single overall score. Since this yields one score per sample, we correlate it against human labels independently for each dimension, treating the same predicted score as the candidate signal for each. This condition reveals whether schema exposure alone is sufficient to direct the judge toward the right perceptual dimensions. Condition 2b: Schema-guided prompt, per-dimension scores. The model is prompted with the full schema and instructed to produce a score for each dimension, enabling direct per-dimension correlation against ground-truth labels. This condition establishes whether models can score all dimensions simultaneously when given the a detailed schema. Condition 3: Isolated per-dimension prompts. The model is prompted for each dimension individually. This condition reveals which dimensions the judge can reliably detect without interference from the full schema. It tests dimension-specific sensitivity in its purest form. Full prompt text for all conditions is provided in Appendix 6.7. 4.3 Metrics For each (rating dimension × model) pair, we evaluate how well model scores track binary human ground-truth labels. Per-dimension effect size and significance. Kendall’s τb serves as a unified comparison metric for per-dimension effect sizes across both model types: for continuous models (MOS predictors) it measures rank-order correlation against binary labels, and for binary models (Audio-LLMs) it is algebraically equivalent to Pearson’s ϕ. This equivalence enables consistent comparison of continuous and binary models within the same evaluation framework.

Table 3: Per-dimension Kendall’s τ against human ground truth per evaluation dimension. AUDIO -LLM S judge rows use the C1 naturalness prompt (single holistic score, closest MOS analogue). Model

Word Phon.

Prosodic

Lex.

Inton.

Pros.S

Paralinguistic

Pros.B Rate

Emo.

Expr.

Spk.

Hum.

MOS Models AudioBoxCE AudioBoxCU AudioBoxPC AudioBoxPQ DNSMOS-ProBVCC DNSMOS-ProVCC NISQA UTMOSv2

-0.051 -0.052 -0.058 -0.088 -0.112 0.005 0.027 0.000

0.161 0.015 -0.019 0.038 -0.055 -0.121 -0.047 0.087

∗∗∗

0.371 0.098 -0.099 0.272∗∗ 0.092 0.026 0.068 0.289∗∗

-0.241∗∗ 0.068 0.337∗∗∗ -0.169∗ 0.121 -0.048 0.142 0.200∗

-0.036 0.048 0.152 -0.079 0.179 -0.038 0.099 -0.024

0.550∗∗∗ 0.485∗∗∗ 0.020 0.447∗∗∗ -0.147∗ 0.071 0.236∗∗ 0.390∗∗∗

0.050 0.187∗ -0.025 0.418∗∗∗ -0.050 0.326∗∗∗ 0.015 0.304∗∗∗ -0.061 -0.072 0.128 0.406∗∗∗ -0.015 0.347∗∗∗ -0.062 0.099

-0.270∗∗ 0.606∗∗∗ -0.177 0.511∗∗∗ -0.142 -0.615∗∗∗ -0.379∗∗∗ 0.606∗∗∗ -0.196 0.450∗∗∗ -0.092 0.397∗∗∗ -0.036 0.377∗∗∗ 0.050 0.313∗∗

AudioLLM Models †

∗∗∗

∗

Gemini 3 Flash 0.323 0.251 Gemini 3 Flash△ -0.022 0.089 Gemini 3.5 Flash† 0.412∗∗∗ 0.312∗∗ △ Gemini 3.5 Flash -0.091 0.082 Qwen3 Omni† 0.000 -0.124 Qwen3 Omni△ 0.173 0.192 Step Audio 2 Mini† -0.018 0.022 Step Audio 2 Mini△ 0.019 0.023

0.125 -0.225∗ 0.083 -0.026 -0.113 0.026 -0.128 0.100

0.125 0.099 0.227∗ 0.111 0.039 -0.103 -0.080 0.040

-0.081 0.123 0.089 0.274∗∗ -0.104 0.091 0.000 -0.050 0.276∗ 0.116 0.255 0.170 0.230 0.000 0.045 0.135 0.064 0.214∗ 0.105 0.061 0.262 0.300∗∗∗ 0.088 0.138 0.116 -0.096 -0.132 0.065 -0.111 -0.078 -0.039 -0.021

0.036 0.036 0.254∗ 0.068 0.293∗ 0.228 0.129 0.095

0.202 0.121 0.238 0.219 0.000 0.237 0.073 0.036

Model Subscripts:: CE = Content Enjoyment; CU = Content Usefulness; PC = Production Complexity; PQ = Production Quality. DNSMOS-Pro subscripts: BVCC = trained on BVCC corpus; VCC = trained on VCC2018 corpus. AUDIO LLM rows use the C1 naturalness prompt. Columns: Phon. = Phonetic Accuracy; Lex. = Lexical Stress; Inton. = Intonation; Pros.S = Prosodic Stress; Pros.B = Prosodic Boundary; Rate = Speech Rate; Emo. = Emotional Appropriateness; Expr. = Expressiveness; Spk. = Speaker Identity; Hum. = Human Plausibility.ç Transcript: † = with transcript; △ = without transcript. Bold = significant (p < .05). ∗ p < .05, ∗∗ p < .01, ∗∗∗ p < .001 (two-sided).

To test significance, for MOS predictors (continuous output), we report Mann-Whitney U , which tests for distributional separation between score distributions of positive and negative labels. For Audio-LLMs(binary output), we also report McNemar’s test, which tests for marginal symmetry between model predictions and ground truth. For MOS predictors, we additionally report AUROC, which is more naturally interpretable for continuous output and is equivalent to a normalized Mann-Whitney U statistic. AUROC rankings are consistent with effect sizes reported by Kendall’s τb across predictors (Spearman’s ρ = 0.995). 4.4 Main Findings MOS Predictors and AudioLLM Judges show sensitivity to different dimensions. Under a basic naturalness prompt (C1), MOS predictors and Audio-LLMs judges attend to fundamentally different aspects of speech quality. Neither class aligns reliably with human judgments across the full set of perceptual dimensions. We observe that quality degradations in many dimensions go unnoticed by these models. Every MOS predictor reaches significance on at least one dimension, with their attention concentrating on paralinguistic dimensions generated via acoustic manipulation, while remaining insensitive to word-level and prosodic dimensions (see

Table 3). Human Plausibility, constructed via inserted glitches, is the most consistently detected dimension, reaching significance across all tested models. Speech Rate Appropriateness, constructed via duration scaling, is also significant across 6 of the 8 models. Both patterns are consistent with MOS predictors tracking signal-level degradation, reflecting their origins in telephony-era voice quality assessment where such artifacts were the primary target. Expressiveness also reaches significance for six of the eight models, though whether this reflects genuine sensitivity to expressive delivery or a response to shared low-level acoustic properties across the Expressiveness, Speech Rate, and Human Plausibility stimuli remains unclear. In contrast, no MOS models show significant detection of Phonetic Accuracy, Lexical Stress, and Prosodic Boundary Placement, dimensions where TTS systems produce linguistically meaningful errors that human listeners reliably detect, but MOS predictors do not. Some significant correlations are negative. We observe a misalignment between some predictors and human perceptual judgments. AudioBoxPC shows a strong negative relationship with Human Plausibility; negative effects also appear across AudioBoxCE , AudioBoxPQ and DNSMOS ProBVCC on Speaker Identity Consistency,

Table 4: Per-dimension Kendall’s τ for AUDIO -LLM S judges using C2a, C2b, and C3 prompts (with and without a provided schema, single or per-dimension outputs). Undefined τ (model predicted same value for all samples) indicated by (.) Model

Word Phon.

Lex.

Prosodic

Paralinguistic

Inton. Pros.S Pros.B Rate

Emo.

Expr.

Spk.

Hum.

C2a — Multi-dimension schema; single holistic score †

0.292 0.206∗ 0.486∗∗∗ 0.396∗∗∗ 0.097 0.194∗ 0.000 0.041

†

∗∗∗

∗∗

Gemini 3 Flash Gemini 3 Flash△ Gemini 3.5 Flash† Gemini 3.5 Flash△ Qwen3 Omni† Qwen3 Omni△ Step Audio 2 Mini† Step Audio 2 Mini△

0.037 0.327∗∗ 0.294∗∗ 0.438∗∗∗ 0.146 0.212∗ 0.186 0.000

0.197 0.141 0.064 0.000 -0.160 0.000 0.283∗ 0.172 0.119 0.216∗ 0.064 0.000 -0.052 0.064 -0.104 0.102 0.052 -0.101 0.192 . . -0.141 0.000 . . -0.101 0.000 0.000 0.160 -0.039 -0.111 -0.221∗

0.264 0.142 0.177 0.135 0.129 0.211∗ 0.124 0.000 0.396∗∗ -0.031 0.122 0.146 0.099 0.170 0.296∗ 0.115 . 0.182 0.435∗∗∗ 0.000 . . -0.124 0.135 0.213 -0.147 0.413∗∗∗ 0.135 0.000 -0.165 0.061 -0.153

C2b — Multi-dimension schema; one score per dimension Gemini 3 Flash Gemini 3 Flash△ Gemini 3.5 Flash† Gemini 3.5 Flash△ Qwen3 Omni† Qwen3 Omni△ Step Audio 2 Mini† Step Audio 2 Mini△

0.397 0.194∗ 0.514∗∗∗ 0.210∗ . . . .

0.106 0.186 0.290∗∗ 0.106 . . . .

. . .

. . . 0.113 . . . . . . . 0.095 .

0.196 .

. .

0.137 0.137 . . . .

0.149 . 0.309∗ 0.149

0.126 0.089 . . . .

. . . 0.149

0.104 . 0.104 . 0.211∗ 0.177 0.147 0.218 . . -0.104 . . . . .

. . 0.135 0.135 . . . .

C3 — Single-dimension focus; one dimension at a time †

Gemini 3 Flash 0.366∗∗∗ 0.000 Gemini 3 Flash△ 0.261∗∗ 0.267∗ † Gemini 3.5 Flash 0.355∗∗∗ 0.178 △ Gemini 3.5 Flash 0.194∗ 0.170 Qwen3 Omni† . . Qwen3 Omni△ 0.136 0.243∗ † Step Audio 2 Mini . . Step Audio 2 Mini△ . -0.108

0.083 . 0.000 . . . -0.155 0.100 . . . . . . . 0.100

0.319∗ 0.089 0.055 0.021 0.243 0.181∗ 0.276∗ 0.232∗∗ 0.137 -0.184∗ 0.298∗ 0.050 . . . .

0.213 0.061 0.213 -0.054 0.459∗∗ -0.060 0.303∗ 0.043 . 0.026 -0.149 -0.214∗ . 0.104 . 0.069

. .

. 0.286∗ 0.218

. . . .

0.135 0.135 -0.135 0.165 0.139 0.000 0.038

Columns: Phon. = Phonetic Accuracy; Lex. = Lexical Stress; Inton. = Intonation; Pros.S = Prosodic Stress; Pros.B = Prosodic Boundary; Rate = Speech Rate; Emo. = Emotional Appropriateness; Expr. = Expressiveness; Spk. = Speaker Identity; Hum. = Human Plausibility. Transcript: † = with transcript; △ = without transcript. Bold = significant (p < .05). ∗ p < .05, ∗∗ p < .01, ∗∗∗ p < .001 (two-sided).

Prosodic Stress and Speech Rate. We hypothesize that these predictors assign higher scores to the controlled acoustic distortions applied to the paralinguistic dimensions (see Appendix 7.3), rating degraded samples more favorably than human listeners do. Generic naturalness prompts yield sparse, inconsistent sensitivity. In contrast to MOS models, Audio-LLMs judges prompted for a generic naturalness rating (C1) show sparse and scattered attention to human-perceptible dimensions (see Table 3). Gemini 3.5 Flash with transcript shows the broadest coverage of any Audio-LLMs condition, reaching significance on dimensions spanning the word-level, prosodic, and paralinguistic tiers, while every other model and condition reaches at most two or three significant dimensions. Beyond the Gemini family, sensitivity under C1 is sparse to

absent, with no dimension reliably detected across more than one model, and the few significant effects are directionally inconsistent. Taken together, the C1 baseline shows selective sensitivity across the Gemini models, while the other models show little evidence of reliable sensitivity to any dimension under an underspecified naturalness prompt. Schema guidance helps selectively. We find that schema guidance recovers word-level sensitivity absent under the unguided naturalness prompt (C1): providing the full 10-dimension schema while maintaining a single aggregated score (C2a) increases sensitivity to word-level dimensions for models that showed none under C1, suggesting that this capacity was present but not elicited without schema guidance (see Table 4). The clearest gain appears in the no-transcript condition: both Gemini variants show no signifi-

cant word-level correlations under the basic naturalness prompt (C1) but achieve statistically significant correlations when provided the detailed schema (C2a). This supports schema guidance improving model sensitivity on word-level dimensions, and underscores how under-specified a bare naturalness prompt is for surfacing sensitivity to which the model evidently has access. However, this benefit does not extend uniformly across all dimensions: some correlations significant under C1 lose significance under C2a, while new ones emerge. Paralinguistic dimensions remain largely resistant across all conditions, with significant correlations sparse and inconsistent. Output collapse (where a model predicts the same score for every sample, leaving correlation undefined) is also observed under C2a; for example, Step Audio 2 Mini assigns the maximum score to every sample regardless of true label. When models are provided the schema and also required to score each dimension independently (C2b), output collapse becomes the dominant failure mode. Two of the four models assign the maximum score across nearly all dimensions and both transcript conditions, leaving correlation undefined in almost all cells. Transcript access does not consistently increase sensitivity to any dimension. Transcript availability produces inconsistent effects across conditions and models, with no clean directional pattern. The models that show sensitivity to word-level dimensions under the unguided naturalness prompt C1 (Gemini 3 Flash and Gemini 3.5 Flash) only do so when the transcript is provided. However, once the schema is provided (C2a), this dependency does not persist; both Gemini variants become significant on the two word-level dimensions without the transcript, although transcript provision is associated with stronger effect sizes. When models are prompted for each dimension independently (C3), both Gemini variants remain significant on Phonetic Accuracy with stronger effect sizes when the transcript is provided. For paralinguistic dimensions, Gemini 3.5 Flash also reaches significance on Emotional Appropriateness in both transcript conditions. However, the pattern runs the opposite way for paralinguistic dimensions for two other models; Qwen3 Omni and Step Audio 2 Mini show significant Speaker Identity sensitivity under C2a only with the transcript, and show no sensitivity to the other paralinguistic dimensions in

either condition. The absence of a consistent transcript benefit suggests that current Audio-LLMs judges are not consistent in their ability to leverage textual grounding to improve dimension-specific detection, even for dimensions such as Phonetic Accuracy where the reference transcript is directly informative. Audio-LLMs sensitivity remains inconsistent under per-dimension prompting. Under C3 (perdimension prompting), sensitivity remains sparse and inconsistent, with a pattern that largely differs from C1 and C2a. An exception is Gemini’s increased sensitivity to Phonetic Accuracy under transcript provision, which holds across prompt variations. Negative correlations still exist for some models on isolated dimensions. Step Audio 2 Mini shows no significant correlation under any transcript condition in C3, consistent with its performance across all other conditions and suggesting a fundamental sensitivity limitation rather than a prompting artifact.

5

Conclusion

We present the first dimension-level metaevaluation benchmark for TTS, decomposing naturalness into 10 linguistically grounded perceptual attributes annotated by trained linguists. Across four MOS predictors and four Audio-LLMs judges, we find that current automated evaluators exhibit systematic but distinct blind spots: MOS predictors align strongly with signal-level artifacts but fail on word-level and prosodic dimensions, whereas Audio-LLMs judges show selective, prompt-dependent detection that never generalises across the full dimension set. We further show that how Audio-LLMs judges are prompted in practice substantially affects their measured performance: querying dimensions in isolation consistently improves alignment with human judgments, while asking models to score all dimensions jointly degrades output reliability through prediction collapse. Taken together, our findings suggest that naturalness cannot be treated as a single scalar construct for modern TTS evaluation and that robust evaluation will require dimension-aware benchmarks.

Limitations Ecological validity. Due to the difficulty of sourcing naturally occurring TTS failures at scale across specific perceptual dimensions, errors were elicited through TTS prompting and Praat manipulation.

Praat-manipulated stimuli are thus best understood as controlled upper-bound conditions rather than representative deployment samples. Stimulus complexity ceiling. Harvard Sentences average 7–8 words with SVO structure, limited discourse context, and few idioms or proper nouns. This is ideal for eliciting clear word-level failures, but limits generalisability to more complex naturalistic speech where prosodic and syntactic interactions are richer. Single architecture for IPA-controlled generation. Phoneme-level input is currently only supported by Cartesia Sonic among the evaluated providers; all IPA-driven samples consequently reflect one TTS architecture. We identify multisystem evaluation as a priority for future work. Annotation scale granularity. Binary scales do not capture full fine-grained quality gradients. In contrast, ordinal scales provide granularity but also the potential for response biases and inconsistent boundary interpretation between levels. In this context, binary scales were a deliberate scope choice; the benchmark targets error detection, not full perceptual grading. Annotation reliability. Majority-class downsampling to achieve class balance reduces effective sample sizes, most consequentially for Emotional Appropriateness (n=46) and Lexical Stress (n=90), where agreement estimates carry wider confidence intervals.

Ethics Statement Our proposed benchmark is intended solely to support more targeted and interpretable TTS evaluation research and is not intended for discriminatory applications or commercial ranking of TTS providers. All annotators are first-language English speakers and trained linguists who participated voluntarily at standard rate compensation. No sensitive or personally identifiable data was collected. All of our speech samples are synthetically generated or acoustically manipulated; no real human speech recordings were collected. Source corpora and APIs are used in accordance with their respective licenses and terms of service (see Appendix 7.5). Regarding Large Language Model (LLM) usage in manuscript preparation, we utilize them solely to refine the language used in paper to improve clarity and correctness, without generating any substantial content or claims.

References Kaito Baba, Wataru Nakata, Yuki Saito, and Hiroshi Saruwatari. 2024. The t05 system for the VoiceMOS Challenge 2024: Transfer learning from deep image classifier to naturalness MOS prediction of highquality synthetic speech. In IEEE Spoken Language Technology Workshop (SLT). Amanda Baughan, Xuezhi Wang, Ariel Liu, Allison Mercurio, Jilin Chen, and Xiao Ma. 2023. A mixedmethods approach to understanding user trust after voice assistant failures. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–16. Paul Boersma. 2001. Praat, a system for doing phonetics by computer. Glot International, 5(9/10):341– 345. Cartesia AI. 2025. Sonic-3: Streaming textto-speech model. https://docs.cartesia.ai/ build-with-cartesia/tts-models/latest. Accessed: 2025. Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, and Lijuan Wang. 2025. Audio-aware large language models as judges for speaking styles. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 467–480, Suzhou, China. Association for Computational Linguistics. David Crystal. 1969. Prosodic Systems and Intonation in English. Cambridge University Press, Cambridge. Fredrik Cumlin, Xinyu Liang, Victor Ungureanu, Chandan K. A. Reddy, Christian Schüldt, and Saikat Chatterjee. 2024. DNSMOS Pro: A Reduced-Size DNN for Probabilistic MOS of Speech. In Interspeech 2024, pages 4818–4822. Diana Geneva, Georgi Shopov, Kostadin Garov, Maria Todorova, Stefan Gerdjikov, and Stoyan Mihov. 2023. Accentor: An Explicit Lexical Stress Model for TTS Systems. In Interspeech 2023, pages 4848–4852. Elijah Gutierrez, Pilar Oplustil-Gallegos, and Catherine Lai. 2021. Location, location: Enhancing the evaluation of text-to-speech synthesis using the rapid prosody transcription paradigm. In Proc. SSW 2021, pages 25–30. Kexin Huang, Qian Tu, Liwei Fan, Chenchen Yang, Dong Zhang, Shimin Li, Zhaoye Fei, Qinyuan Cheng, and Xipeng Qiu. 2025. Instructttseval: Benchmarking complex natural-language instruction following in text-to-speech systems. arXiv preprint arXiv:2506.16381. Wen Chin Huang, Erica Cooper, Yu Tsao, Hsin-Min Wang, Tomoki Toda, and Junichi Yamagishi. 2022. The voicemos challenge 2022. In Proc. Interspeech 2022, pages 4536–4540.

Wen-Chin Huang, Szu-Wei Fu, Erica Cooper, Ryandhimas E Zezario, Tomoki Toda, Hsin-Min Wang, Junichi Yamagishi, and Yu Tsao. 2024. The voicemos challenge 2024: Beyond speech quality prediction. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 803–810. IEEE. IEEE Subcommittee on Subjective Measurements. 1969. IEEE recommended practice for speech quality measurements. IEEE Std 297-1969. Appendix C: 1965 Revised List of Phonetically Balanced Sentences (Harvard Sentences). Yannick Jadoul, Bill Thompson, and Bart de Boer. 2018. Introducing Parselmouth: A Python interface to Praat. Journal of Phonetics, 71:1–15. Alexandra Jesse, Katja Poellmann, and Ying-Yee Kong. 2017. English listeners use suprasegmental cues to lexical stress early during spoken-word recognition. Journal of Speech, Language, and Hearing Research, 60(1):190–198. Ambika Kirkland, Shivam Mehta, Harm Lameris, Gustav Eje Henter, Eva Szekely, and Joakim Gustafson. 2023. Stuck in the MOS pit: A critical analysis of MOS test methodology in TTS evaluation. In 12th ISCA Speech Synthesis Workshop (SSW2023), pages 41–47. Klaus Krippendorff. 2011. Computing Krippendorff’s alpha-reliability. Departmental papers, University of Pennsylvania, Annenberg School for Communication, Philadelphia, PA. Jonggeun Lee, Junseong Pyo, Gyuhyeon Seo, and Yohan Jo. 2026. Speakersleuth: Evaluating large audio-language models as judges for multi-turn speaker consistency. arXiv preprint arXiv:2601.04029. Potsawee Manakul, Woody Haosheng Gan, Michael J Ryan, Ali Sartaz Khan, Warit Sirichotedumrong, Kunat Pipatanakul, William Barr Held, and Diyi Yang. 2026. Audiojudge: Understanding what works in large audio model based speech evaluation. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3644–3663. Ruskin Raj Manku, Yuzhi Tang, Xingjian Shi, Mu Li, and Alexander Smola. 2025. Emergenttts-eval: Evaluating tts models on complex prosodic, expressiveness, and linguistic challenges using model-as-ajudge. 38. Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. 2021. Nisqa: A deep cnn-selfattention model for multidimensional speech quality prediction with crowdsourced datasets. arXiv preprint arXiv:2104.09494. Mahathir Monjur and Shahriar Nirjon. 2025. Speechqualityllm: Llm-based multimodal assessment of speech quality. arXiv preprint arXiv:2512.08238.

Brian Patton, Yannis Agiomyrgiannakis, Michael Terry, Kevin Wilson, Rif A. Saurous, and D. Sculley. 2016. Automos: Learning a non-intrusive assessor of naturalness-of-speech. In NIPS 2016 End-to-end Learning for Speech and Audio Processing Workshop. Chandan KA Reddy, Vishak Gopal, and Ross Cutler. 2021. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6493–6497. IEEE. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab system for voicemos challenge 2022. In Proc. Interspeech 2022, pages 4521–4525. Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. 2021. A survey on neural speech synthesis. arXiv preprint arXiv:2106.15561. Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, and 1 others. 2025. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139. Ismail Rasim Ulgen, Zongyang Du, Junchen Lu, Philipp Koehn, and Berrak Sisman. 2026. Objective evaluation of prosody and intelligibility in speech synthesis via conditional prediction of discrete tokens. IEEE Open Journal of Signal Processing. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2022. Learning from disagreement: A survey. J. Artif. Int. Res., 72:1385–1470. Iván Vallés-Pérez, Julian Roth, Grzegorz Beringer, Roberto Barra-Chicote, and Jasha Droppo. 2021. Improving Multi-Speaker TTS Prosody Variance with a Residual Encoder and Normalizing Flows. In Interspeech 2021, pages 3131–3135. Tianxin Xie, Yan Rong, Pengfei Zhang, Wenwu Wang, and Li Liu. 2025. Towards controllable speech synthesis in the era of large language models: A systematic survey. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 764–791. Xueyao Zhang, Chaoren Wang, Huan Liao, Ziniu Li, Yuancheng Wang, Li Wang, Dongya Jia, Yuanzhe Chen, Xiulin Li, Zhuo Chen, and 1 others. 2025. Speechjudge: Towards human-level judgment for speech naturalness. arXiv preprint arXiv:2511.07931.

6 Appendix Glossary of Terms

boundaries, and syntactic relationships influence prominence placement and intonational contour shape.

Linguistic Theory Acoustic.Of or relating to the physical properties of sound as a mechanical wave. In speech science, acoustic analysis involves measurements of frequency, amplitude, duration, and spectral characteristics derived from the audio signal.

Paralinguistic.Pertaining to communicative cues that accompany linguistic expression without constituting part of its lexical, grammatical, or phonological structure. Such cues may convey affect, stance, or speaker identity. Although paralinguistic features may be vocal, visual, gestural, or otherwise embodied, this glossary uses the term in the narrower context of TTS evaluation to refer only to vocal cues.

Lexical.Of or relating to the words or vocabulary of a language, as distinct from its grammatical structure. In speech, lexical properties include word identity, lexical stress assignment, and phonological form. Linguistic.Of or relating to language or linguistics. In speech evaluation, pertaining to features conventionally encoded in the grammar of a language, including segmental and prosodic structure. Phoneme.The smallest contrastive unit of sound capable of distinguishing meaning. Phonemes are abstract categories realized by phonetically similar variants (allophones) whose distribution is governed by phonological context. Phonetics.The branch of linguistics concerned with the physical and perceptual properties of speech sounds. Phonology.The branch of linguistics concerned with the abstract, rule-governed organization of sound systems. Phonology describes the systematic patterns governing the distribution and combination of phonemes. Segmental.Pertaining to individual consonant and vowel segments comprising the phonemic inventory of a language. Segmental quality encompasses place and manner of articulation, voicing, vowel quality, and allophonic processes such as aspiration and flapping. Semantic.Of or relating to meaning in language. In TTS evaluation, semantic considerations include whether prosodic realization—particularly intonation and focus—accurately reflects information structure, such as given/new status and contrastive emphasis. Suprasegmental.Pertaining to phonological features spanning units larger than a single segment: stress, pitch accent, intonation, rhythm, and boundary placement. Also termed prosodic features. Syntactic.Of or relating to the grammatical structure and word order of sentences. Syntactic constituent boundaries frequently align with prosodic

Notational f0 (Fundamental Frequency) The lowest frequency component of a voiced speech waveform, corresponding to the rate of vocal fold vibration. f0 is the primary acoustic correlate of perceived pitch and a key parameter of intonation and stress. IPA (International Phonetic Alphabet) A standardized phonetic notation system maintained by the International Phonetic Association, providing a unique symbol for every sound in any human language. Used for both broad (phonemic) and narrow (phonetic) transcription. Speech & Language Technology LALM (Large Audio Language Model) A multimodal extension of large language models incorporating audio as an input and/or output modality. LALMs can process and generate spoken language, in many architectures subsuming TTS and ASR within a unified model. LLM (Large Language Model) A neural language model trained on large-scale text corpora via self-supervised objectives, capable of generation, classification, and reasoning across a wide range of text-based tasks. TTS (Text-to-Speech) A class of speech synthesis technology converting written text to spoken audio. Modern neural TTS systems leverage sequence-tosequence architectures and neural vocoders trained at scale to produce high-fidelity synthetic speech. MOS (Mean Opinion Score) A perceptual evaluation metric in which listeners rate stimulus quality on a scale of 1–5; the MOS is the arithmetic mean of collected ratings. Originally standardized for telephony (ITU-T P.800), MOS and its variants are standard in TTS evaluation.

Table 5: The ten-aspect evaluation schema organized by dimension. Full annotation criteria and worked examples are provided in Table 6. Level

Dimension

Failure condition

Word

Phonetic accuracy Lexical stress

Sound outside acceptable range of lexical target Stress assigned to wrong syllable

Intonation

Prosodic boundary placement Speech rate appropriateness

Pitch contour inappropriate to utterance type or pragmatic meaning Post-lexical prominence misplaced relative to information structure Phrase boundaries inconsistent with syntactic structure Tempo inappropriate to context

Expressiveness Emotional appropriateness Speaker identity consistency Human plausibility

Affective range insufficient or excessive for content Emotional tone inconsistent with content Voice characteristics vary within the utterance Vocal qualities outside physically realizable range

Prosodic

Paralinguistic

Prosodic stress

6.1 Data Scale 6.1.1 Scale Collapse For most ternary dimensions, collapsed α is comparable to raw ordinal α; for expressiveness, collapsed α exceeds raw α , indicating that the middle level adds noise rather than signal. Three-way splits on ternary dimensions (each rater assigning a distinct level) occur only on the 1-versus-2 boundary and are irresolvable by majority vote; these samples are excluded from the final dataset for that dimension.

Table 6: Annotation rubric for all ten speech quality dimensions. Each dimension is described in full in §3.3.

Dimension Phonetic Accuracy (§6.2.1)

Score

Meaning

1

One or more consonants or vowels are wrong and the error(s) are very noticeable. You might mishear the word entirely. One or more consonants or vowels are slightly off, but you can still tell what word was intended without much effort. All consonants and vowels sound correct. No perceivable phonetic errors.

2 3

Lexical Stress (§6.2.2)

0 1 1

Intonation (§6.2.3) 2 3 Prosodic Stress (§6.2.4)

0 1

Prosodic Boundary Placement (§6.2.5)

1

2 3 Speech Rate Appropriateness (§6.2.6)

1 2 3

Emotional Appropriateness (§6.2.7) Expressiveness (§6.2.8)

0 1 1 2 3

At least one word has stress on the wrong syllable. Even one error earns a 0. Every multi-syllable word has stress on the correct syllable with appropriate vowel quality. No errors detected. One or more pitch contour errors cause high negative impact. The intonation may confuse the intended sentence type or signal an unintended attitude. One or more errors cause low negative impact. Something about the melody sounds slightly unnatural but does not cause confusion. No perceivable errors. The intonation sounds natural and appropriate for the sentence and its context. Prominence falls on words that do not warrant it, or words that should be prominent are not. The choice of prominent words sounds predictable and appropriate. New or important information is highlighted; given material is de-emphasized. A boundary is missing where it is needed or present where it should not be, with severe impact. You cannot recover the intended sentence structure while listening. A boundary is in the wrong position and noticeably hurts the experience, but you can still follow the sentence structure. Words are grouped into phrases that align naturally with the sentence’s syntactic and semantic structure. Rate is clearly inappropriate: so fast words blur together, or so slow the pace feels labored. Rate is slightly too fast, too slow, or too uniform, but you can follow the content without real difficulty. Tempo feels natural and varies appropriately. Complex material gets more time; parentheticals move briskly. The emotional tone is wrong for the content. The mismatch is clearly perceivable. The emotional tone is a plausible match for the text and context. Delivery is extremely flat and robotic, or excessively exaggerated, to the point of seriously detracting from the experience. Delivery is slightly too flat or slightly too dramatic for the content, but not very distracting. Delivery has natural, appropriate variation; sounds engaged and dynamic without drawing attention to itself. (Continued on next page)

(Continued from previous page)

Dimension

Score

Meaning

Speaker Identity Consistency (§6.2.9)

0

The voice changes in a way that makes it sound like a different person, either within or across sentences. The voice is recognizably consistent throughout. It sounds like a single individual, even with natural variation in expressiveness or loudness.

1

Human Plausibility (§6.2.10)

0

1

The audio contains at least one moment that could not have been produced by a human speaker: robotic or metallic artifacts, impossible pitch jumps, audio glitches, or unnaturally cut-off syllables. The entire audio stream sounds like it could have been produced by a human speaker.

6.2 Human Annotations Annotator Background All annotators are firstlanguage English speakers and trained linguists. Annotators participated voluntarily at the standard rate compensation. The annotation task involved listening to synthesized speech samples and marking perceptual quality dimensions based on the linguistic-verified guidelines; no sensitive or personal data was collected. Annotator Iteration For dimensions that contain inherent human subjectivity, we initially tested capturing a severity gradient with a ternary scale (1-3). However, annotators could not reliably distinguish between non-perfect rating levels, so we collapsed these to binary, trading nuance for dataset reliability. Since not all generations reliably exhibit the intended failure, we generated an excess of samples and relied on human annotation to establish groundtruth labels (see Table 9). Annotation is conducted through a purpose-built interface presenting the audio, the dimension definition, the revealable transcript, and the response options in a single screen, minimizing cognitive load and reducing interface-induced variance between raters. Annotation Guide The annotation guide provides detailed explanations and rubric ratings on how to rate synthetic (TTS) speech on each metric in the evaluation schema. For every metric, you will find: a plain-language explanation of what the metric captures, what to listen for, a rating scale with clear descriptions of each score, and examples to anchor annotators’ judgments. The metrics are grouped into three dimensions: Word-level (are the right sounds and stress patterns produced for each word?), Prosodic (is the speech organized into phrases with appropriate melody, emphasis, and timing?), and Paralinguistic (does the speech convey appropriate emotion, expressiveness, and speaker characteristics?). The details are provided in Table 6. General Labeling Guideline Listen to each utterance at least twice before scoring. On the first pass, get a general impression. On the second pass, focus on whichever metric you are currently rating. Try not to let a strong error in one metric bias your rating in another: a word might be mispronounced (a Phonetic Accuracy issue) while the prosody is perfectly fine, or vice versa. Rate each metric independently.

Word-Level Metrics Word-level metrics ask: does each individual word sound right? This covers whether the TTS system picked the correct sounds (phonemes) and placed stress on the correct syllable within each word. 6.2.1 Phonetic Accuracy What you are listening for. Does every word contain the right speech sounds? This includes consonants, vowels, and coarticulation. You are checking whether the system produced the correct phonemes for each word, especially for tricky cases like heteronyms, rare words, proper nouns, and loanwords. Example. “She will lead the group to the lead mine.” The first lead should rhyme with feed and the second with red. If the system pronounces both the same way, that is a phonetic accuracy error. How disruptive it is determines whether you score 1 or 2. 6.2.2 Lexical Stress What you are listening for. Is the stress on the right syllable within each word? For example, the noun record is stressed on the first syllable (RE-cord) while the verb is stressed on the second (re-CORD). This metric is strictly about within-word stress; do not consider sentence-level emphasis here (see Prosodic Stress, Metric 4). Why binary? Lexical stress errors are relatively unambiguous, and even a single error can substantially impair word recognition. Lexical vs. prosodic stress. Lexical stress is a fixed dictionary property (which syllable within a word). Prosodic stress (Metric 4) is a speaker-level choice about which words in a sentence receive prominence. Prosodic Metrics Prosodic metrics ask: is the speech organized and delivered in a way that communicates the structure and meaning of the sentence? Prosodic errors can change what a listener understands even when every individual word sounds correct. Because pitch, duration, loudness, and pauses serve multiple prosodic functions, a single acoustic event can be relevant to more than one metric. Rate each metric for its own question, not the acoustic signal in isolation. Table 7 summarizes which channels each metric draws on. 6.2.3 Intonation What you are listening for. Intonation is the pitch melody across phrases and sentences. It signals sentence type (questions typically rise; statements

Table 7: Acoustic channels for each prosodic metric. Metric Intonation Prosodic Stress Boundary Placement Speech Rate

Pitch ✓ ✓ ✓

Dur.

✓ ✓ ✓

Loud.

Pause

✓ ✓ ✓

fall), speaker attitude, and discourse structure. This metric concerns pitch only; do not factor in rate or pauses. 6.2.4 Prosodic Stress What you are listening for. Which words in the sentence receive extra prominence? Speakers signal focus via higher pitch, greater loudness, and longer duration. Do not confuse with lexical stress (Metric 2), which concerns syllables within a word. 6.2.5 Prosodic Boundary Placement What you are listening for. Speakers group words into phrases using pitch movements, final-syllable lengthening, and pauses. Misplaced boundaries disrupt parsing even when no alternative meaning is at stake. Focus on whether the grouping aligns with plausible syntactic structure and produces a pleasant listening experience. Example. Boundary errors go in both directions. A spurious boundary (“It was a | beautiful day”) sounds choppy; a missing boundary (“When the dog bites the man screams”) creates a parsing failure. In both cases the question is whether the boundary made the speech harder or less pleasant to follow. 6.2.6 Speech Rate Appropriateness What you are listening for. Is the overall tempo appropriate, and does it vary naturally? Human speakers slow for complex or emotionally weighty material and speed up through predictable or parenthetical content. This metric is distinct from Boundary Placement (where phrase breaks occur). Paralinguistic Metrics Paralinguistic metrics ask: beyond the linguistic message, does the speech convey appropriate speaker characteristics? This covers emotion, expressiveness, voice consistency, and whether the audio could have been produced by a human. 6.2.7 Emotional Appropriateness What you are listening for. Does the emotional tone of the voice match the content? A sentence about exciting news should not sound bored, and a condolence message should not sound cheerful. This metric does not ask whether the emotion is

strong enough (see Expressiveness, Metric 9); it asks only whether the type of emotion is correct. 6.2.8 Expressiveness What you are listening for. Expressiveness captures the degree of natural variation in delivery. TTS speech can be flat (monotone, robotic) or overdone (theatrical, exaggerated). Both extremes are errors. Rate expressiveness relative to what is appropriate for the content: informational material calls for more measured delivery; emotionally charged content warrants greater vocal variation. 6.2.9 Speaker Identity Consistency What you are listening for. Does the voice maintain a consistent perceived identity (stable pitch range, vocal quality, perceived age, perceived gender, and accent) both within a single sentence and across sentences in a set? Inconsistencies can occur across sentences or partway through a single sentence. 6.2.10 Human Plausibility What you are listening for. This metric is independent of linguistic correctness. The question is whether a human being could have physically produced this audio. It targets digital artifacts: robotic buzzing, clicks, metallic resonance, impossible pitch jumps, abruptly cut-off syllables, or anything outside the range of human vocal production. Do not penalize linguistic errors here; a mispronounced word is a Phonetic Accuracy issue but may still sound physically human. Quick Reference Table 8: All metrics at a glance. Dim.

Metric

Scale

Summary

Word

Phonetic Accuracy Lexical Stress

1–3 0/1

Right sounds produced? Right syllable stressed?

Intonation

1–3

Prosodic Stress

0/1

Boundary Placement

1–3

Speech Rate

1–3

Pitch melody fits sentence? Right words emphasized? Natural phrase grouping? Tempo appropriate?

Emotional Appr.

0/1

Expressiveness

1–3

Speaker Consistency

0/1

Human Plausibility

0/1

Pros.

Para.

Emotion matches content? Delivery dynamic/fitting? Same person throughout? Could a human produce this?

Remember: Rate each metric independently. A single utterance can score 3 on Phonetic Accuracy but 1 on Intonation, or 0 on Human Plausibility but 1 on Lexical Stress. Do not let a strong impression from one dimension bleed into another.

6.3

Acoustic Overlap in the Prosodic and Paralinguistic Tiers The prosodic dimensions share acoustic channels: f0 , for instance, carries intonation, prosodic stress, and boundary tones simultaneously. Crucially, however, sharing an acoustic channel does not entail these dimensions cannot fail independently. A misplaced prosodic boundary may disrupt the pitch contour without any corresponding failure in accent placement, and a stress error may occur with an otherwise well-formed intonation contour. This independence motivates rating each dimension separately rather than holistically: each stimulus is rated on all applicable dimensions independently, so cases where a single acoustic anomaly produces failures across multiple categories are captured explicitly rather than collapsed into a single quality judgment. The paralinguistic tier presents an analogous case. These four dimensions share acoustic channels with prosody but convey information about the speaker rather than the linguistic message. A synthesis artifact may simultaneously affect perceived speaker identity consistency and human plausibility, yet the two dimensions target distinct speaker properties and can fail independently of one another. Figure 2 further demonstrates the idea of correlations. 6.4

Paralinguistic Framework and Neural TTS Crystal’s original framework distinguishes voice quality features, which are speaker-stable properties and remain constant across an utterance, from affective paralinguistic features, which modulate in response to content and context. The four paralinguistic dimensions in our schema map directly onto this distinction: speaker identity consistency and human plausibility reflect speaker-stable properties, while expressiveness and emotional appropriateness reflect content-driven modulation. Human plausibility warrants particular note. Modern neural TTS can produce voices that are internally consistent and expressively adequate but physically implausible (a failure mode not anticipated by Crystal’s framework but characteristic of neural synthesis). We treat it as a distinct paralinguistic dimension rather than subsuming it under overall quality, since it represents a novel category of synthesis failure with no natural analog in human speech perception research.

6.5 Intended versus observed labels. Table 9 reports the agreement between intended quality levels (design variable) and majority-vote annotator labels (ground truth). Of 741 samples intended to contain the targeted failure, annotators majority-labeled 239 (32.3%) as having no failure: approximately 1 in 3 designed errors was not perceived as such. Misses are highest for emotional appropriateness (46.2%), intonation (40.0%), and lexical stress (38.4%), and lowest for prosodic boundary placement (22.4%) and phonetic accuracy (21.2%). The false-alarm rate on intendedclean items is only 3.5%, indicating that raters systematically under-call failures rather than over-call them which is a pattern consistent with the difficulty of eliciting perceptually salient errors on higher-level dimensions. 6.6 AUROC Analysis for MOS Models As demonstrated in Table 10, MOS models exhibit striking performance gap across dimensions. Most presented models achieve strong alignment with human judgments on human plausibility (AUC 0.72 − 0.92) and speech rate (AUC 0.40 − 0.89), yet systematically fail on word-level and lower prosodic dimensions. Phonetic accuracy and lexical stress yield near-chance performance across all models (AUC 0.41 − 0.61), while intonation and prosodic stress show modest, inconsistent correlation (AUC 0.33 − 0.74). 6.7

Prompt Details

Condition 1: Underspecified MOS-style prompt [ROLE] You are a speech quality evaluator assessing text-to-speech audio. [TASK] % if show_text %A synthesized speech sample is attached above, along with the text it was generated from. Evaluate the overall naturalness of the speech. Intended text: "original_text"% else %A synthesized speech sample is attached above. Evaluate its overall naturalness based on the audio alone.% endif % [CRITERIA] Rate the naturalness on a binary scale (0 = failure present, 1 = no failure): 0: One or more issues affecting naturalness are perceivable, whether minor or severe. 1: Completely natural, indistinguishable from human speech. [PROCEDURE] Follow these steps: 1. Attend to the attached audio sample. % if show_text %2. Compare the spoken output to the intended text. 3. Identify any issues that affect naturalness. 4. Decide the most appropriate score.% else %2. Identify any issues that

Table 9: Intended versus observed label distributions on targeted samples. Caught = majority vote agrees with intended failure. Missed = majority vote labels intended failure as clean. Tie = no majority among the first 3 valid binary votes (only possible when fewer than 3 raters scored the dimension). Miss rate = missed / intended-fail. Bottom panel: of 376 intended-clean items, 363 (96.5%) were observed clean; false-alarm rate = 3.5%. Tier

Dimension

N

Int. fail

Caught

Missed

Tie

Miss rate

Word-level

Phonetic accuracy Lexical stress

120 113

80 73

63 45

17 28

0 0

21.2% 38.4%

Prosodic

Intonation Prosodic stress Prosodic boundary placement Speech rate

120 124 95 168

80 79 58 128

44 53 45 93

32 26 13 35

4 0 0 0

40.0% 32.9% 22.4% 27.3%

Paralinguistic

Emotional appropriateness Expressiveness Speaker identity consistency Human plausibility

58 100 104 115

39 70 64 70

21 50 32 49

18 20 31 19

0 0 1 2

46.2% 28.6% 48.4% 27.1%

Total

1117

741

495

239

7

32.3%

Intended-clean items (n = 376): observed clean 363 (96.5%); false-alarm rate 3.5%; ties 0.

Table 10: Construct-aligned AUC for MOS models per evaluation dimension. Model

Word

Prosodic

Phon. Lex. Inton. AudioBoxCE AudioBoxCU AudioBoxPC AudioBoxPQ DNSMOS-ProBVCC DNSMOS-ProVCC NISQA UTMOSv2

0.46 0.46 0.46 0.44 0.42 0.50 0.52 0.50

0.61 0.51 0.49 0.53 0.46 0.41 0.47 0.56

∗∗∗

0.76 0.57 0.43 0.69∗∗ 0.56 0.52 0.55 0.70∗∗

Pros.S ∗∗

0.33 0.55 0.74∗∗∗ 0.38∗ 0.59 0.47 0.60 0.64∗

Paralinguistic

Pros.B Rate 0.47 0.53 0.61 0.44 0.63 0.47 0.57 0.48

Emo. Expr. ∗∗∗

0.89 0.84∗∗∗ 0.51 0.81∗∗∗ 0.40∗ 0.55 0.67∗∗ 0.77∗∗∗

0.53 0.48 0.47 0.51 0.46 0.59 0.49 0.46

∗

0.63 0.79∗∗∗ 0.73∗∗∗ 0.71∗∗∗ 0.45 0.79∗∗∗ 0.74∗∗∗ 0.57

Spk.

Hum. ∗∗

0.31 0.38 0.40 0.23∗∗∗ 0.36 0.44 0.47 0.53

0.92∗∗∗ 0.86∗∗∗ 0.07∗∗∗ 0.92∗∗∗ 0.82∗∗∗ 0.78∗∗∗ 0.76∗∗∗ 0.72∗∗

AudioBox subscripts: CE = Content Enjoyment; CU = Content Usefulness; PC = Production Complexity; PQ = Production Quality. DNSMOS-Pro subscripts: BVCC = trained on BVCC corpus; VCC = trained on VCC2018 corpus. Column key: Phon. = Phonetic Accuracy; Lex. = Lexical Stress; Inton. = Intonation; Pros.S = Prosodic Stress; Pros.B = Prosodic Boundary; Rate = Speech Rate; Emo. = Emotional Appropriateness; Expr. = Expressiveness; Spk. = Speaker Identity; Hum. = Human Plausibility. Bold = significant (p < .05). ∗ p < .05, ∗∗ p < .01, ∗∗∗ p < .001 (two-sided Mann–Whitney U ). affect naturalness. 3. Decide the most appropriate score.% endif % [OUTPUT] Respond in this exact JSON format: "evaluation_steps": "<your step-by-step reasoning>", "score": <0 or 1> Condition 2a: Schema-guided prompt, single [ROLE] You are an expert speech quality evaluator with training in phonetics and prosodic phonology. [TASK] % if show_text %A synthesized speech sample is attached above, along with the text it was generated from. Evaluate the overall quality of the speech, considering all dimensions in the evaluation schema below. Intended text: "original_text"% else %A synthesized speech sample is attached above. Evaluate the overall quality of the speech, considering all dimensions in the evaluation schema below.% endif % [CRITERIA] Consider the following ten dimensions when forming your overall judgment:

WORD-LEVEL: - Phonetic Accuracy: Does every word contain the right speech sounds? This includes consonants, vowels, and the way adjacent sounds blend together (coarticulation). Check whether the system produced the correct phonemes for each word, especially for heteronyms, rare words, proper nouns, and loanwords. - Lexical Stress: Is the stress on the right syllable within each word? Every multi-syllable English word has a predictable stress pattern. Incorrect lexical stress often changes vowel quality too. This is strictly about within-word stress. PROSODIC: - Intonation: The melody of speech: the way pitch rises and falls across phrases and sentences. It signals sentence type, speaker attitude, and discourse structure. Concerns pitch only. - Prosodic Stress: Which words in a sentence receive extra prominence via higher pitch, greater loudness, and longer duration, signaling which information is new, important, or contrastive. This is about entire words within a sentence, not syllables within a word. - Prosodic Boundary Placement: Speakers group words

into phrases using pitch movements, slight lengthening of the final syllable, and pauses. Misplaced boundaries disrupt the listening experience. Focus on whether grouping aligns with plausible syntactic structure. - Speech Rate Appropriateness: Is the overall speed appropriate, and does it vary in natural ways? Human speakers slow down for complex material and speed up through predictable material. PARALINGUISTIC: - Emotional Appropriateness: Does the emotional tone of the voice match the content of the text? This does not ask whether the emotion is strong enough (that is Expressiveness). It asks only whether the type of emotion is correct. - Expressiveness: Natural variation in delivery style, pace, and vocal energy. Human speech is dynamic. TTS speech can be flat or overdone. Rate relative to what would be appropriate for the content. - Speaker Identity Consistency: Does the voice maintain a consistent perceived identity in terms of pitch range, vocal quality, perceived age, gender, and accent? Check for shifts both within and across sentences. - Human Plausibility: Could a human have physically produced this audio? Targets digital artifacts and glitches: robotic buzzing, clicks, unnatural metallic resonance, impossible pitch jumps, abrupt syllable cutoffs. Independent of linguistic correctness. Considering ALL of the above, rate overall quality on a binary scale (0 = failure present, 1 = no failure): 0: One or more issues detected across the ten dimensions, whether minor or severe. 1: No perceivable issues detected across all ten dimensions. [PROCEDURE] Follow these steps: 1. Attend to the attached audio sample. 2. For each of the three dimension groups (word-level, prosodic, paralinguistic), note whether any issues are present. 3. Identify which specific dimensions (if any) show failures. 4. Decide the most appropriate score. [OUTPUT] Respond in this exact JSON format: "word_level_issues": "<any word-level issues noted>", "prosodic_issues": "<any prosodic issues noted>", "paralinguistic_issues": "<any paralinguistic issues noted>", "score": <0 or 1> Condition 2b: Schema-guided prompt, per-dimension scores

Intended text: "original_text"% else %A synthesized speech sample is attached above. Evaluate the speech independently on each of the ten dimensions below, using the specified scale for each.% endif % [CRITERIA] Rate each metric independently. Do not let a strong impression from one dimension bleed into another. ALL DIMENSIONS use a binary scale (0 = failure present, 1 = no failure): Lexical Stress: Is the stress on the right syllable within each word? This is strictly about within-word stress, not sentence-level emphasis. 0: At least one word has stress on the wrong syllable. 1: Every multi-syllable word has stress on the correct syllable. Prosodic Stress: Which words in the sentence receive extra prominence via higher pitch, greater loudness, and longer duration? 0: Prominence falls on words that do not warrant it, or words that should be prominent are not. 1: The choice of which words receive prominence sounds predictable and appropriate. Emotional Appropriateness: Does the emotion of the voice match the content of the text? If the text is neutral, the emotion should be neutral as well. 0: The emotion is wrong for the content. 1: The emotion is a plausible match for the text. Speaker Identity Consistency: Does the voice maintain a consistent perceived identity? 0: The voice changes in a way that makes it sound like a different person. 1: The voice is recognizably consistent throughout. Human Plausibility: Could a human have physically produced this audio? 0: The audio contains at least one moment that could not have been produced by a human speaker. 1: The entire audio stream sounds like it could have been produced by a human speaker. Phonetic Accuracy: 0: One or more consonants or vowels are outside the range of possible expected sounds for the utterance spoken in general american english. 1: All consonants and vowels fall within the range of possible expected sounds for the utterance spoken in general american english. Intonation: 0: Perceivable errors in the pitch contour, regardless of severity. 1: No perceivable errors in the pitch contour.

[ROLE] You are an expert speech quality evaluator with training in phonetics and prosodic phonology.

Prosodic Boundary Placement: 0: Misplaced boundary disrupts sentence structure, noticeably or severely. 1: Phrase grouping aligns naturally with syntactic structure.

[TASK] % if show_text %A synthesized speech sample is attached above, along with the text it was generated from. Evaluate the speech independently on each of the ten dimensions below, using the specified scale for each.

Speech Rate Appropriateness: 0: Speech rate is too slow or too fast during parts or the whole of the utterance. 1: Speech rate feels natural and varies appropriately.

Expressiveness: 0: Delivery is too flat or too dramatic, regardless of severity. 1: Delivery has natural, appropriate variation. [PROCEDURE] For each dimension: 1. Attend to the attached audio sample focusing on that dimension. 2. Compare what you hear to the rubric anchors above. 3. Record your reasoning and score. Evaluate each dimension independently. [OUTPUT] Respond in this exact JSON format: { "phonetic_accuracy":{"reasoning": "...", "score": <0 or 1>}, "lexical_stress":{"reasoning": "...", "score": <0 or 1>}, "intonation": {"reasoning": "...", "score": <0 or 1>}, "prosodic_stress":{"reasoning": "...", "score": <0 or 1>}, "prosodic_boundary":{"reasoning": "...", "score": <0 or 1>}, "speech_rate": {"reasoning": "...", "score": <0 or 1>}, "emotional_appropriateness": {"reasoning": "...", " score": <0 or 1>}, "expressiveness":{"reasoning": "...", "score": <0 or 1>}, "speaker_identity":{"reasoning": "...", "score": <0 or 1>}, "human_plausibility":{"reasoning": "...", "score": <0 or 1>}

dimensions originally generated with three severity levels, scores 1 and 2 were collapsed to label 0; score 3 maps to label 1. 7.1 Shared Defaults LLM. Azure OpenAI gpt-5; api_version 2025-03-01-preview; max_completion_tokens = 16384; temperature: API default. TTS (default). Cartesia sonic-3 (Blake: a167e0f3-df7e-4d52-a9c3-f949145efdab, Brooke: e07c00bc-4134-4eae-9ea4-1a55fb45746b) TTS (intonation). ElevenLabs eleven_multilingual_v2. 7.2 Pathway 1: IPA-Controlled TTS gpt-5 produces a modified transcript for each intended error level, passed to Cartesia Sonic-3. Table 11 lists voices and prompts are listed in 7.5; Table 12 lists intended errors. Table 11: IPA pathway: voice per dimension. All use Cartesia Sonic-3.

}

Condition 3: Isolated per-dimension prompts [ROLE] You are an expert speech quality evaluator with training in phonetics and prosodic phonology. [TASK] % if show_text %A synthesized speech sample is attached above, along with the text it was generated from. Evaluate the speech on ONE specific dimension. Intended text: "original_text"% else %A synthesized speech sample is attached above. Evaluate the speech on ONE specific dimension based on the audio alone.% endif %

Dim.

Voice

Phon. Acc. Lex. Str. Pros. Str.

Brooke Blake Blake

Table 12: IPA pathway: intended error per label. Dim.

Label

Intended error

Phon. Acc.

1 0

Canonical IPA Phoneme substitution (within- or cross-class)

Lex. Str.

1 0

Canonical stress Shifted stress + vowel reduction

Pros. Str.

1 0

Emphasis on content word Emphasis on function word (Cartesia quote-markup)

[CRITERIA] dimension_block [PROCEDURE] Follow these steps: 1. Attend to the attached audio sample. 2. Focus exclusively on dimension_name. 3. Compare what you hear to the score anchors above. 4. Select your score. Do not consider any other aspect of speech quality. Evaluate ONLY dimension_name. [OUTPUT] Respond in this exact JSON format: "reasoning": "<your reasoning about dimension_name>", "score": <score_range>

7

Generation Pipeline Details

This appendix documents the model configurations and hyperparameters for the error generation pipeline described in Section 3.2. All audio: 44,100 Hz, 16-bit PCM, mono. All dimensions use binary labels: 1 (acceptable) and 0 (error). For

7.3 Pathway 2: Acoustic Manipulation A clean TTS baseline is synthesised first; Parselmouth/Praat PSOLA or local DSP then introduces a targeted perturbation. Intonation. Source: EmergentTTS-Questions split (declaratives and questions; 4–12 words; no commas/dashes). ElevenLabs voice Brian (nPczCjzI2devNBz1zQrb) serves as the label-1 anchor; label-0 samples are derived via F0 manipulation with the Praat parameters in Table 13. Label 1: verbatim ElevenLabs synthesis. Label 0: F0 contour manipulation ; either terminal-rise compression (last ∼30% of contour compressed to 15–30% of original excursion) or one of 10 highimpact strategies (Table 14), randomly selected.

Table 13: Praat PSOLA parameters (Intonation and Speaker Identity). Parameter Analysis floor Manipulation floor Pitch ceiling Output clip range Control points Time step

Value 75 Hz 50 Hz 600 Hz 60–320 Hz 45 0.01 s

Two strategies may be stacked with probability –stack-rate P (default 0.0). Table 14: F0 strategies for Intonation label 0. All operate within mean ± 70–130 Hz. Strategy

Parameters

Kept

terminal-rise comp.

Last ∼30% of contour compressed to 15–30% of original excursion Reverse contour; scale 1.2– 1.5×; ±25 Hz noise Invert around mean; scale 1.3–1.6×; ±25 Hz noise Linear rise: mean−40 → mean+130 Hz Range ×3.5–5.0; clip at ceiling 3–5 levels; mean ±70 Hz ∼50% frames random; mean ±90 Hz 0.7×mean base; 3–5 bumps 0.5–1.6× 3–5 Hz wobble; ±40– 65 Hz 130→180 Hz drift; ±30 Hz wobble Cumulative; σ ≈35 Hz/step

✓

reverse invert question exaggerate staircase random jumps flatten+spike sine wave monotone drift random walk

modified; label 0: post-processed. Table 15: DSP artifact taxonomy (24 types, 8 categories).

Category

Types (✓ = used in kept dataset)

Compression

mp3 compression ✓, quantization noise ✓ hard clipping ✓, soft clipping ✓, bit crushing ✓ buffer dropout, stutter repeat , granular fragmentation ✓ white noise ✓, pink noise, vinyl crackle ✓, EM hum 50 Hz ✓, EM hum 60 Hz ✓ time-stretch artifact ✓, pitchcorrection artifact ✓, phase cancellation ✓ bit errors ✓, sample-rate mismatch, buffer underrun ✓ echo feedback ✓, resonant comb filter ✓, feedback squeal ✓ FFT smearing ✓, phase distortion, aliasing ✓

Clipping Glitch Noise

Time/Pitch ✓ ✓ ✓

Digital Delay

✓ ✓ ✓ — ✓ — ✓

Speech Rate. Voice: default English. Direction (fast/slow) randomly assigned. Label 1: Cartesia speed=1.0. Label 0: either Cartesia speed=1.4 (fast) or 0.6 (slow), optionally further stretched via librosa time-stretch — factor 0.7 (fast; net ≈2.0×) or 1.3 (slow; net ≈0.78×). Speaker Identity Consistency. Each sample is a pair: label 0 (inconsistent) and label 1 (consistent). Label 0 uses either (a) equal-power crossfade of two renderings by different voices across a target vowel (≥50 ms), or (b) stitch of two utterances with a 400 ms gap. Voice B undergoes PSOLA pitch alignment when F0 mismatch exceeds 0.5 semitones; pairs with >3 semitones are rejected. Voice pairs: Brooke ↔ Emma (feminine), Ronald ↔ Henry (masculine), all Cartesia Sonic-3 SSE. Human Plausibility. gpt-5 selects 1–2 artifact types from Table 15 (always from distinct categories); local DSP applies the recipe to clean Cartesia synthesis (default English voice). Label 1: un-

Spectral

7.4 Pathway 3: Synthetically Generated Text Prosodic Boundary Placement. We created 60 manually curated sentences from Claude Opus 4.8, stratified by error severity and type. The set contains 20 sentences with severe errors (Score: 0), 20 with moderate errors (Score: 0), and 20 with correct boundaries (Score: 1). Within each tier, 10 sentences contain boundary insertion errors and 10 contain boundary deletion errors. The severe-deletion subset uses garden-path sentences (e.g., “the old man the boat”) where a necessary boundary deletion forces readers to reparse syntactic structure. This is a failure mode where both human readers and modern TTS systems consistently underperform. Severe-insertion sentences feature mid-constituent commas that create infelicitous boundaries (e.g., “The quick brown, fox jumps over the lazy dog”). Moderate-deletion errors are produced by deleting spaces around coordinating conjunctions, creating rushed but recoverable runs (e.g., “The rain stoppedso we went for a walk”). Moderateinsertion errors place commas between major constituents that listeners can still regroup (e.g., “He bought, a new car last weekend”). TTS: Cartesia Sonic-3; no LLM phase.

Emotional Appropriateness. 38 items use Cartesia Sonic-3 with a mismatched inline emotion tag (e.g. euphoric on sad text) for label 0. 20 additional items use ElevenLabs voice Gerald (Emotionless, Flat and Clear) for flat delivery of emotionally loaded sentences (label 0). Label 1: matched emotion or natural delivery.

"error_description": "Canonical American English pronunciation.", "transcription": "<<canonical-IPA>>", "sentence": "<sentence with inline transcription>" } ] }

Constraints.

7.5 LLM Prompts Prompts are reproduced exactly as submitted to gpt-5. Per-utterance slots shown as placeholders.

1. Scores 1 and 2 must not produce a real English word. 2. Score 1: target the primary stressed syllable; use large vowel-class or manner shifts.

Phonetic Accuracy Prompt — Phonetic Accuracy

3. Score 2: clearly audible but less dramatic — consonant voicing/manner change, or a one-step vowel shift in a secondary-stressed syllable. Do not swap schwas in fully reduced syllables (inaudible in synthesis).

Task. You are a phonetic transcription assistant. Given a sentence and a target word, produce three versions of the sentence in which the target word is replaced by three distinct IPA transcriptions in Cartesia sonic-3 phoneme notation, representing decreasing levels of segmental accuracy (score 1 = most erroneous; score 3 = canonical).

4. Do not use word-final /z/ ↔ /s/ swaps — the contrast is undetectable in synthesis. 5. Use only phonemes from the inventory above. Do not introduce ÉŽ, Éń, É¿, ÊŤ, length marks, or diacritics absent from the table; write ÉŽ as Éİ (stressed) or ÉŹ|Éź (unstressed).

Phoneme Format. Strings are enclosed in << . . . >> with pipe-separated symbols; stress markers precede the stressed syllable. Example: Cartesia → <<k|ÉŚ|Éź|ËĹ|t|i|ÊŠ|ÉŹ>> Ph.

Ex.

Ph.

Ex.

aI aU b d dZ eI f h i j k l m n oU p s t tS v z

my now bed do jeans day fish hat see yes key lip man no go pen sip top church van zoo

A O OI @ E 3r g I r S U Z " , T D "æ N u w

cot law boy sofa bed bird go bit red ship book vision primary stress secondary stress thin this cat sing two we

Input. Sentence: {{INPUT_UTTERANCE}} Target word: {{TARGET_WORD}} Output Format. Return only valid JSON — no explanation, markdown, or code fences. { "target_word": "<original word>", "versions": [ { "score": 1, "error_description": "<substitution + why highly salient>", "transcription": "<<IPA-with-error>>", "sentence": "<sentence with inline transcription>" }, { "score": 2, "error_description": "<substitution + why moderate >", "transcription": "<<IPA-with-error>>", "sentence": "<sentence with inline transcription>" }, { "score": 3,

6. Score 3 is the unmodified canonical American English transcription.

Lexical Stress Prompt — Lexical Stress Task. You are a phonetic transcription assistant. Given a sentence and a target word, produce two versions of the sentence in which the target word is replaced by a Cartesia sonic-3 phoneme string. The two versions differ only in lexical stress placement. When stress shifts to a different syllable, apply the phonological changes that naturally follow in American English: vowel reduction (newly unstressed full vowels reduce, e.g. /ÉŻ/ → /ÉŹ/); vowel restoration (reduced /ÉŹ/ in newly stressed syllables becomes its full counterpart); and any other stress-conditioned segmental alternations. Phoneme format and inventory: identical to Phonetic Accuracy above (see §7.2). Input. Sentence: {{INPUT_UTTERANCE}} Target word: {{TARGET_WORD}} Output Format. Return only valid JSON — no explanation, markdown, or code fences. { "target_word": "<original word>", "versions": [ { "score": 0, "error_description": "<which syllable incorrectly stressed,

where it should be, and phonological changes applied>", "transcription": "<<IPA-with-wrong-stress>>", "sentence": "<sentence with inline transcription>" }, { "score": 1, "error_description": "Canonical American English pronunciation with correct lexical stress placement.", "transcription": "<<canonical-IPA>>", "sentence": "<sentence with inline transcription>" }

• Complementizers / relativizers: which, who, whom

that,

• Particles: to (infinitival) Ambiguous cases: “that” as demonstrative → function word; “that” in contrastive focus → content word. Wh-words in questions → content words; as relativizers → function words. Contractions → leave unquoted.

]

Rules.

}

1. Preserve original spelling, capitalization, and punctuation exactly. Sentence-final punctuation stays outside any quoted span.

Constraints. 1. Target word must be polysyllabic. If monosyllabic, return an empty versions array. 2. Score 0: move primary stress to a different syllable. If a secondary stress exists, swapping primary and secondary is preferred. Apply all resulting phonological changes. 3. Phoneme changes in score 0 must be a direct consequence of the stress shift — no arbitrary substitutions. 4. Use only phonemes from the inventory (same constraint as Phonetic Accuracy).

2. A single function word gets its own quote pair; two or more adjacent function words share one pair. 3. Do not add, remove, or reorder words. Input. Sentence: {{INPUT_UTTERANCE}} Output Format. Return only valid JSON — no explanation, markdown, or code fences. target_word is always the literal string "stress" (used for folder naming). { "target_word": "stress", "versions": [ { "score": 1, "error_description": "Natural prosody -- content words stressed, function words reduced.", "transcription": "", "sentence": "<original sentence unchanged>" }, { "score": 0, "error_description": "Stress placed on \"<region>\" (<pos>), \"<region>\" (<pos>).", "transcription": "<region1>; <region2>; ...", "sentence": "<sentence with function-word runs in \"...\">" } ]

Prosodic Stress Prompt — Prosodic Stress Task. You are a linguistic annotation tool. Given an English sentence, identify function words (words that should not carry prosodic stress under a neutral broad-focus reading) and output two versions: (1) score 1 — original sentence, unchanged (natural prosody baseline); (2) score 0 — every contiguous run of function words is wrapped in typographic double quotes " . . . ", forcing a pitch accent on the quoted span and producing audibly misplaced prosodic stress. Function-word categories. • Determiners: these, those

a, an, the, this, that,

• Prepositions: in, on, at, to, from, of, for, by, with, about, into, through, over, under, between, among, during, before, after, against, without

}

error_description for score 0: begin with Stress placed on, then list each quoted region as "<region>" (<pos>), with multi-word POS joined by ‘+’, e.g. "on the" (preposition + determiner). Example. Input: The cat sat on the mat. { "target_word": "stress", "versions": [ { "score": 1, "error_description": "Natural prosody -- content words stressed, function words reduced.", "transcription": "", "sentence": "The cat sat on the mat." }, { "score": 0, "error_description": "Stress placed on \"The\" ( determiner), \"on the\" (preposition + determiner).", "transcription": "The; on the", "sentence": "\u201CThe\u201D cat sat \u201Con the\ u201D mat." } ]

• Auxiliaries: is, am, are, was, were, be, been, being, has, have, had, do, does, did, will, would, shall, should, can, could, may, might, must • Pronouns: I, me, you, he, him, she, her, it, we, us, they, them, my, your, his, its, our, their • Conjunctions: and, but, or, nor, so, yet, because, although, if, when, while, that, which, who

}

Failures on each rating dimension, split by intended-dimension ground truth score Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility

emotional_appropriateness

0

10

human_plausibility

0

0

0

0

10

20

20

30

lexical_stress

10

20

30

prosodic_boundary

10

20

30

40

40

speaker_identity_consistency

10

20

Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility

30

Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility

Phonetic Accuracy Lexical Stress Intonation Prosodic Stress Prosodic Boundary Placement Speech Rate Appropriateness Emotional Appropriateness Expressiveness Speaker Identity Consistency Human Plausibility

expressiveness

Intended dimension score score = 0 score = 1

0

0

0

20

10

20

30

40

phonetic_accuracy

20

40

20

40

60

prosodic_stress

0

0

40

intonation

speech_rate

20

40

60

Figure 2: Co-occurrence of perceptual failures across speech quality dimensions. Per-dimension failure counts stratified by intended-dimension ground-truth label. Each subplot corresponds to one intended dimension (column panels); bars show the number of score-0 (failure) observations on each rating dimension (y-axis), split by whether the sample’s ground-truth label on its own intended dimension is a failure (orange, score = 0) or a pass (blue, score = 1). Tall blue bars on dimensions other than the intended one reveal that samples annotated as passing on their intended construct nonetheless exhibit failures on unrelated dimensions.

Human Plausibility Prompt — Human Plausibility Task. You are an audio signal processing expert and evaluation dataset designer. Given a sentence and a target word, produce two versions of the same utterance at controlled quality levels. Both versions use the same clean spoken sentence — the sentence field is identical across both. The difference is a post-processing artifact recipe applied after synthesis. Score 1 is an unmodified clean reference; score 0 has 1–2 digital artifacts applied, making it clearly sound artificial or degraded. Artifact Types. Use these exact identifiers in the type field. Compression: mp3_compression (lossy codec smearing, muffled quality); quantization_noise (low-bit-depth noise/distortion). Clipping & Distortion: hard_clipping (peaks chopped flat; harsh); soft_clipping (warmer harmonic distortion); bit_crushing (lo-fi crunch). Glitch / Stutter: buffer_dropout (sudden silence or frozen chunk); stutter_repeat (rapid looping of a tiny slice); granular_fragmentation (micro-segments scrambled). Noise: white_noise (broadband injection); pink_noise (spectrally weighted); vinyl_crackle (crackle and hiss); em_hum_50hz (50 Hz EM hum); em_hum_60hz (60 Hz EM hum). Time & Pitch: time_stretch_artifact (metallic smearing); pitch_correction_artifact (robotic, over-quantised tuning); phase_cancellation (comb-filtering from offset copies). Digital Errors: bit_errors (random bit flips; sharp pops); sample_rate_mismatch (aliasing or chirping); buffer_underrun (choppy, skipping playback). Delay-Based: echo_feedback (feedback runaway); resonant_comb_filter (resonant comb); feedback_squeal (feedback loop squeal). Spectral: fft_smearing (FFT smearing from over-processed EQ); phase_distortion (heavy-filter phase distortion); aliasing (sample-rate-reduction aliasing). Location Values. target_word : target word’s duration only; sentence_start : first ∼20%; sentence_middle : middle ∼60%; sentence_end : last ∼20%; throughout : full signal. Severity Values. mild : subtle, attentive listening needed; moderate : clearly audible, intelligibility intact; severe : highly noticeable, degrades intelligibility. Input. Sentence: {{INPUT_UTTERANCE}} Target word: {{TARGET_WORD}}

Artifact assignment score 0: {{ARTIFACT_SELECTION}}

for

Output Format. Return only valid JSON — no explanation, markdown, or code fences. { "target_word": "<original target word>", "versions": [ { "score": 1, "sentence": "<original sentence, unchanged>", "error_description": "Clean reference -- no artifacts applied.", "artifacts": [] }, { "score": 0, "sentence": "<original sentence, unchanged>", "error_description": "<listener-perspective description>", "artifacts": [ { "type": "<identifier from list above>", "location": "<value from list above>", "severity": "<mild | moderate | severe>" } ] } ] }

Constraints. 1. sentence must be identical across both versions. 2. Score 1 must have "artifacts": []. 3. Score 0 must use exactly the specified artifact type(s) — no substitutions or additions. 4. At least one score 0 artifact must be moderate or severe; mild-only is insufficient. 5. Use location: target_word for localised glitch artifacts (buffer_dropout, stutter_repeat, bit_errors, granular_fragmentation). Use throughout for diffuse artifacts (noise, compression, hum). 6. Write error_description from a listener’s perspective, e.g. “A sharp digital stutter interrupts the stressed vowel of ‘eclipse’, looping a tiny slice twice before the sentence continues.” 7. Match location and severity to the target word’s phonetics: stutter/dropout are most salient on vowel-heavy or sonorant words; clipping/bit-crushing on fricatives and plosives; noise/compression are phoneme-agnostic.

7.6 Artifacts Licenses We provide documentations of the artifact licenses that were used to assist with our data generation and annotation pipeline together with detailed analyses in Table 16, 17, 18.

Table 16: TTS APIs and source datasets used in this work. Artifact

Version / Model

License

cartesia SDK eleven_multilingual_v2 openai SDK

Commercial Commercial Commercial

IEEE Std 269 bosonai/EmergentTTSEval

IEEE Apache 2.0

TTS APIs / Services Cartesia TTS API ElevenLabs TTS API Azure OpenAI / GPT-5 Source Datasets Harvard Sentences EmergentTTS-Eval

Table 17: Signal processing and audio tools used in this work. Artifact

Version / Notes

License

Signal Processing / Audio Tools Praat (praat-parselmouth) Montreal Forced Aligner SciPy

parselmouth english_us_arpa signal processing

SoundFile / libsndfile

audio I/O

GPL v3 MIT BSD 3Clause BSD 3Clause

7.7 Model Details MOS Predictors NISQA, DNSMOS-Pro, UTMOSv2, and Audiobox-Aesthetics were each served via a custom FastAPI inference endpoint on a single NVIDIA Tesla P100-PCIE-12GB GPU paired with an Intel Xeon Gold 6126 CPU @ 2.60 GHz. Audio-LLMs We evaluate four Audio-LLMs as automatic judges. Provider, endpoint identifier, input modality, and parameter count are summarized in Table 19. Where the vendor has not published an official parameter count, we report it as not disclosed. Table 19 summarises hardware requirements for each LALM judge. The two Gemini models are accessed via the Google API. Qwen3-Omni (30B total / ≈3B active, MoE) requires ≈60 GB VRAM at bfloat16 and is served on 2×80 GB GPUs. Step-Audio-2-mini (8B) requires ≈16 GB VRAM but is served on 1×80 GB GPU to accommodate its custom vLLM backend and audio detokenizer (tensor-parallel-size = 2). All openweight models were served via vLLM on NVIDIA A100/H100 GPUs. Each judge was queried three times per sample per condition; with 860 samples × 4 conditions × 2 transcript settings, this yields up to 20,640 inference calls per model.

Table 18: Python packages used in the pipeline and analysis stages of this work.

Package

Purpose

License

Azure OpenAI / GPT-5 API client Cartesia TTS API client ElevenLabs TTS API client Numerical computation Tabular data manipulation HuggingFace dataset loading HuggingFace model/data access Excel file I/O Environment variable management YAML configuration parsing Audio file I/O

MIT MIT MIT BSD 3-Clause BSD 3-Clause Apache 2.0 Apache 2.0 MIT BSD 3-Clause MIT BSD 3-Clause

Inter-annotator agreement Classification and evaluation metrics Statistical modeling Statistical tests Crowdsourcing aggregation Relative importance analysis Interactive visualization Static visualization Statistical visualization Interactive data exploration UI Static image export for plotly

MIT BSD 3-Clause BSD 3-Clause BSD 3-Clause Apache 2.0 MIT MIT PSF / BSD BSD 3-Clause Apache 2.0 MIT

Pipeline Packages openai cartesia elevenlabs numpy pandas datasets huggingface_hub openpyxl python-dotenv ruamel-yaml soundfile Analysis Packages krippendorff scikit-learn statsmodels pingouin crowd-kit dominance-analysis plotly matplotlib seaborn streamlit kaleido

Table 19: Audio-LLM judges and compute requirements. T = text, I = image, A = audio, V = video. Full model name for row 3: Qwen3-Omni-30B-A3B-Instruct. GPU counts reflect our serving configuration; open-weight models served via vLLM on NVIDIA A100/H100 GPUs. # Judge

Provider Modality

1 Gemini 3 Flash Google 2 Gemini 3.5 Flash Google 3 Qwen3-Omni Alibaba 4 Step-Audio-2-mini StepFun

Parameters

VRAM GPUs

Access

T+I+A+V not disclosed - API T+I+A+V not disclosed - API T+I+A+V 30B/≈3B (MoE) ≈60 GB 2×80 GB open weights T+A 8B ≈16 GB 1×80 GB open weights

Related documents

• ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions#1069855
• Like humans, language models demonstrate face-to-character biases.#469009
• Resonant Structural Emulation: Toward Recursive Coherence in Reflective AI#73147
• Structural Silence: When AI Infrastructure Fails Speakers of Underrepresented Languages#447971
• Distinguishing Concreteness Differences in LLM Representations via Linear Probing#1089928