SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning
KiHyun Nam1
arXiv:2605.15044v1 [cs.SD] 14 May 2026
1
Jungwoo Heo2
Siu Bae1
Ha-Jin Yu2
Joon Son Chung1†
Korea Advanced Institute of Science and Technology (KAIST) 2 University of Seoul [email protected] † Corresponding author
Abstract As audio-first agents become increasingly common in physical AI, conversational robots, and screenless wearables, audio large language models (audio-LLMs) must integrate speaker-specific understanding to support user authorization, personalization, and context-aware interaction. This requires modeling who is speaking, how the voice sounds, and how recording conditions affect speaker cues. Conventional speaker verification systems provide strong scalar scores but little linguistic evidence, while current audio-LLMs and speaker-aware language models have limited ability to organize speaker information beyond binary labels or descriptive profiles. We present SpeakerLLM, a speaker-specialized audio-LLM framework that unifies single-utterance speaker profiling, recording-condition understanding, utterance-pair speaker comparison, and evidence-organized verification reasoning within a natural-language interface. We construct verification-reasoning targets and a decision-composition policy that separate profile-level evidence from the final same-or-different decision and organize recording condition, profile evidence, and the decision into a structured trace. At its core, SpeakerLLM uses a hierarchical speaker tokenizer designed to capture multiple granularities of speaker evidence. Utterance-level speaker embeddings summarize identity and profile-level cues, whereas frame-level speaker features preserve fine-grained acoustic descriptors. Experiments show that SpeakerLLM-Base improves speaker-profile and recordingcondition understanding over general audio-LLMs, while SpeakerLLM-VR preserves strong generated-verdict accuracy and produces decision traces grounded in the supervised verification reasoning schema. We will release the metadataenriched supervision dataset and target-construction code for reproducibility.
1
Introduction
The rapid proliferation of large language models (LLMs) has accelerated a shift toward agentic AI systems that operate through natural interaction channels. In settings such as physical AI, conversational robots, and screenless wearables, voice is often the primary, and sometimes the only, medium for human-agent interaction [1–3]. In such audio-first environments, speaker-specific understanding is needed to support user authorization, personalization, and context-aware interaction. Speaker verification (SV) is therefore no longer merely an auxiliary personalization module; it becomes the identity-facing layer of a broader speaker-aware interface. The shift reframes SV from a backend scoring problem to an audio-language interface problem [4–7]. Conventional SV systems compare two recordings by producing a similarity score from speaker embeddings [8–10] and thresholding it into a same/different decision. While effective as backend modules, such systems do not explain whether a rejection is likely caused by transient noise, reverberation, recording mismatch, or genuine speaker-identity mismatch. A speaker-aware audio-LLM Preprint.
should therefore go beyond a bare verdict and organize speaker evidence in natural language so that the decision is auditable to the user [11, 12]. Existing research remains fragmented with respect to this goal. Conventional SV systems provide strong verification accuracy but expose only scalar scores without linguistic evidence [13]. Explainable SV methods introduce attribute-based concept bottlenecks [14, 15] or explainable voice vectors [16], but remain largely tied to score-based verification. Recent speaker-aware LLMs move toward language-based interaction, but often cast verification as binary text-label classification over speaker embeddings [17, 18]. Conversely, generative speaker profiling models can describe speaker attributes, but descriptive profiles alone are insufficient for identity verification because multiple speakers can share correct descriptors such as gender, age, or regional background [19, 20]. Thus, a unified framework that connects speaker understanding, utterance-pair comparison, and evidence-organized verification reasoning remains missing. We present SpeakerLLM, a speaker-specialized audio-LLM framework for natural-language speaker understanding and comparison. SpeakerLLM targets an LLM-interface setting in which speakerrelated acoustic cues are read, compared, and expressed in natural language. It is built around two design principles. First, speaker evidence is distributed across representation granularities. Utterance-level speaker embeddings summarize identity and profile-level cues, whereas frame-level speaker features preserve fine-grained acoustic evidence such as pitch, timbral brightness, and recording condition. We operationalize this representation-granularity observation with a hierarchical speaker tokenizer that converts both representations into continuous speaker tokens for the language model. Second, speaker cue reading, utterance-pair comparison, and evidence-organized verification reasoning require different supervision structures. We therefore train SpeakerLLM along a twostage trajectory, yielding SpeakerLLM-Base for speaker-profile QA, recording-condition QA, and standard same/different speaker judgment, and SpeakerLLM-VR for speaker verification reasoning. A key component of SpeakerLLM-VR is the verification reasoning target construction policy. Rather than generating free-form rationales [21], speaker verification reasoning uses a three-block format: ENVIRONMENT _ STATUS, PROFILE _ COMPATIBILITY, and DECISION. The target organizes recording condition, profile-level evidence, and the final same/different decision into an evidenceorganized decision trace. Crucially, it separates profile-level evidence from the final decision: two utterances may have similar profiles but come from different speakers, or differ in some profile attributes while belonging to the same speaker. We explicitly include such reversal cases to discourage shortcuts that map profile similarity mechanically to a same-speaker decision. In this work, we operationalize faithfulness in an evidence-grounded sense [22, 23]: generated evidence blocks are evaluated against the supervised schema used to construct the verification reasoning target. Our contributions are as follows: • Speaker-specialized audio-LLM framework. We introduce SpeakerLLM as a unified natural-language interface for single-utterance speaker profiling, recording-condition understanding, utterance-pair speaker comparison, and evidence-organized verification reasoning. • Representation-granularity analysis for speaker evidence. We show that utterance-level speaker embeddings and frame-level speaker features preserve complementary speaker evidence, and operationalize this finding with a hierarchical speaker tokenizer for speakerspecialized audio-LLMs. • Evidence-organized reasoning target construction. We design a verification reasoning target and decision composition policy that separate profile-level evidence from the final same/different decision and organize recording condition, profile evidence, and the decision into an auditable trace, including reversal cases that discourage profile-similarity shortcuts. We also construct a metadata-enriched supervision dataset and will release the dataset and target-construction code.
2
Speaker Understanding and Verification Reasoning Dataset
We construct a metadata-enriched supervision corpus for speaker-specialized audio-LLM training. The corpus connects single-utterance speaker understanding, recording-condition understanding, utterance-pair speaker comparison, and evidence-organized verification reasoning in natural language. 2
Figure 1: QA task inventory for SpeakerLLM training. Single-utterance tasks read speaker-profile and recording-condition cues; utterance-pair tasks include standard SV, attribute compatibility QA, and the three-block verification reasoning target. Table 1: Attribute taxonomy used in SpeakerLLM. Speaker-profile attributes are derived from public metadata or extracted from audio; environment factors are generated online through acoustic simulation. Exact binning boundaries are in Appendix A.1. Attribute
Classes
Source
Gender Age Region
{male, female} {1–7, 8–12, 13–17, 18–25, 26–35, 36–45, 46–55, 56–65, 66–75, 76+} {North American, European, British / Irish, Latin / Hispanic, Oceanian, East / Southeast Asian, Middle Eastern / North African, African} {very low, low, normal, high, very high} (gender-conditioned) {muted, mellow, neutral, bright, brilliant} {clean, mild, moderate, severe, extreme} {minimal, slight, moderate, heavy, extreme}
VoxCeleb, LibriTTS-R VoxCeleb VoxCeleb
Pitch Timbral brightness Noise Reverberation
VoxCeleb, LibriTTS-R VoxCeleb, LibriTTS-R MUSAN SLR28 RIR
Rather than collecting a new corpus from scratch, we combine VoxCeleb [24, 25] and LibriTTSR [26] with public metadata [27, 28], audio-derived descriptors, and online acoustic simulation. The central design principle is to separate speaker-profile attributes from environment factors: the former provide interpretable speaker-intrinsic evidence, whereas the latter describe recording conditions that can affect the reliability of that evidence. The resulting labels define the SpeakerLLM task inventory: single-utterance QA, utterance-pair QA, and verification reasoning targets. Table 1 summarizes the attribute taxonomy; details are provided in Appendices A and B. 2.1
Attribute Supervision
Speaker-profile attributes are constructed from both metadata-derived profile labels and audio-derived descriptors. For VoxCeleb [24, 25], we use VoxCeleb metadata [27] to obtain gender, age, and nationality. Nationality is mapped to coarser regional/linguistic background classes and age is converted into coarse bins, providing stable and interpretable cues instead of optimizing fine-grained demographic prediction from speech. For LibriTTS-R [26], we use only the gender metadata aligned from LibriTTS-P [28]. We additionally extract pitch (mean F0) and timbral brightness (spectral centroid) from both VoxCeleb and LibriTTS-R audio; together they form the voice characteristic target (see Appendix A.3 for extraction and gender-conditioning details). Environment factors consist of background noise and reverberation. Noise is simulated with MUSAN non-speech audio [29] and controlled by SNR, while reverberation is simulated with SLR28 RIRs [30] and bucketed by estimated RT60 (Appendix A.2). Although these environment factors do not define speaker identity, they provide recording-condition evidence that affects the reliability of speaker cues. 2.2
Training Scenarios
Single-utterance QA. Single-utterance QA trains the model to read speaker-profile attributes or environment factors from one speech recording. Speaker-side targets include gender, age, regional/linguistic background, voice characteristic, and full speaker profile. Environment-side targets include noise, reverberation, and joint acoustic profile. 3
Figure 2: Overview of SpeakerLLM. A frozen speaker encoder extracts a speaker embedding e and frame-level features H. The hierarchical speaker tokenizer converts them into embedding-level and sequence-level speaker tokens, which are inserted at fixed prompt slots for LLM conditioning. Utterance-pair QA. Utterance-pair QA uses two recordings as input and supervises relational speaker evidence. This category includes attribute compatibility QA, where the model compares a profile axis across the two utterances, and a standard same/different speaker judgment. Verification reasoning target. For the evidence-organized reasoning variant, we construct a verification reasoning target. The target follows a structured three-block format: ENVIRONMENT _ STATUS → PROFILE _ COMPATIBILITY → DECISION.
The first block describes the recording conditions of the two utterances. The second block summarizes whether the available speaker-profile attributes provide supportive, mixed, or conflicting profile-level evidence. The final DECISION block produces the final verification verdict. Crucially, the decision block is not a direct restatement of the profile-level evidence: two utterances may have similar profiles but come from different speakers, or differ in some profile attributes while belonging to the same speaker. The verification reasoning target is therefore constructed to separate profile-level evidence from the final same/different decision, allowing the target to express both aligned and reversal cases in a linguistically coherent form. Exact target templates, profile-support scoring, and the verification-reasoning target construction policy are provided in Appendix B. Because full speaker-profile descriptions and speaker verification reasoning require all profile attributes, these targets are instantiated only on VoxCeleb utterances, where age and region metadata are available; the ENVIRONMENT _ STATUS block is populated by applying the same online noise and reverberation simulation used for standalone environment QA on LibriTTS-R.
3
Approach
SpeakerLLM is a speaker-specialized audio-LLM that reads speaker-related cues from one or two utterances and maps them to natural-language speaker profiles, recording-condition descriptions, utterance-pair comparisons, and evidence-organized verification reasoning. As shown in Figure 2, the model consists of a frozen speaker encoder, a speaker tokenizer, and a language model. The speaker encoder extracts speaker-discriminative representations from the input waveform, the speaker tokenizer converts them into continuous speaker tokens, and the language model consumes these speaker tokens together with textual prompts to generate task-specific answers. SpeakerLLM is built around two design principles motivated by empirical observations about speaker representations and training dynamics. First, speaker-profile attributes and audio-derived descriptors are not uniformly represented at a single granularity. Utterance-level speaker embeddings provide stable summaries of speaker identity and profile-level cues, whereas fine-grained acoustic evidence such as pitch, timbral brightness, and recording condition is better retained in frame-level representations. Second, speaker cue reading, utterance-pair comparison, and evidence-organized verification reasoning share the same speaker-token conditioning mechanism but require different supervision structures and output formats. We therefore train SpeakerLLM along a sequential trajectory that first 4
builds speaker understanding and then further tunes the same speaker-token conditioning mechanism for verification reasoning. The trajectory yields two model variants. SpeakerLLM-Base is obtained after Stage 1 and is used for speaker-profile QA, recording-condition QA, and standard same/different speaker judgments. SpeakerLLM-VR is obtained by further tuning SpeakerLLM-Base with verification reasoning targets and is used for verification reasoning. Both variants share the same frozen speaker encoder, speaker tokenizer, language-model backbone, and prompt conditioning mechanism. 3.1
Hierarchical Speaker Tokenizer
We use the term speaker tokenizer to denote the waveform-to-token module that converts an input utterance into continuous speaker tokens readable by the language model. It consists of a frozen speaker encoder and a trainable speaker-token adapter. The speaker encoder extracts speaker representations from the input waveform, and the adapter maps the encoder outputs into a sequence of speaker tokens in the LLM hidden space. Thus, the adapter is the learnable mapping component inside the speaker tokenizer, while the speaker tokenizer denotes the full speaker-specialized conditioning module. This distinction is important for speaker-specialized audio-LLMs. A generic audio-to-LLM connector can be viewed largely as a projection from an encoder output to the LLM hidden dimension. For speaker understanding and SV Reasoning, however, dimensional alignment alone is insufficient. The final speaker embedding of a speaker verification encoder provides a compact utterance-level identity summary, but temporal pooling can suppress fine-grained acoustic evidence needed for explanation, such as pitch, timbral brightness, and recording condition. Conversely, frame-level speaker features retain temporal and acoustic detail, but may lack a compact identity-level summary by themselves. The observation motivates our representation-granularity hypothesis: speaker evidence useful for an audio-LLM is distributed across multiple representation levels. To retain these complementary sources of evidence, SpeakerLLM uses a hierarchical speaker tokenizer. Given an input utterance x ∈ RT , the frozen speaker encoder produces a speaker embedding e ∈ Rde and frame-level speaker features H ∈ RL×dh before temporal pooling. The speaker-token adapter processes these two representations with separate branches. The embedding-level branch maps e to N speaker tokens with an MLP, while the sequence-level branch summarizes H into M speaker tokens with a Q-Former [31]: S(emb) = MLP(e) ∈ RN ×d , S = [S
(emb)
(seq)
;S
]∈R
S(seq) = QFormer(H) ∈ RM ×d , (N +M )×d
.
(1) (2)
Here, S is the continuous speaker-token sequence inserted into the LLM prompt. The hierarchical construction avoids compressing all speaker evidence into a single embedding and instead retains both embedding-level and sequence-level speaker evidence in the language-model context. A token-count ablation verifies that this gain is not attributable to using more speaker tokens alone (Appendix D.1). 3.2
Two-Stage Training for SpeakerLLM Variants
SpeakerLLM is trained along a single sequential trajectory that yields SpeakerLLM-Base and SpeakerLLM-VR. The two-stage procedure is not merely a training schedule; it stabilizes and then extends the same speaker-token conditioning mechanism for different interaction modes. Stage 1: Speaker Understanding. Stage 1 trains the model to read speaker-profile attributes, environment factors, and simple utterance-pair relations. It includes single-utterance QA and simple utterance-pair QA such as standard same/different speaker judgment, noise comparison, and reverberation comparison. At the beginning of Stage 1, the speaker encoder and language model are kept frozen, and only the speaker tokenizer is trained with short-form targets. The warm-up phase focuses on predicting the answer value itself instead of generating full sentences, thereby aligning audio-derived speaker representations with the language model input space. The model is then trained with sentenceform targets while the speaker tokenizer remains trainable and the language model is adapted with LoRA [32]. Sentence adaptation then transfers class-style cue reading into natural-language speaker and environment descriptions. The resulting model is used as SpeakerLLM-Base. 5
Stage 2: Verification-Reasoning Tuning. Stage 2 further tunes SpeakerLLM-Base for evidenceorganized verification reasoning. Its key distinction is not the use of paired inputs alone, since Stage 1 already includes simple utterance-pair QA through standard same/different judgment and environment comparison. Instead, Stage 2 trains the model to compose recording condition, profile-level evidence, and the final same/different decision into an evidence-organized decision trace. To this end, Stage 2 focuses on verification reasoning targets and attribute compatibility QA. The verification reasoning target follows the three-block format (ENVIRONMENT _ STATUS, PRO FILE _ COMPATIBILITY, DECISION) defined in Section 2.2. Attribute compatibility QA asks the model to compare profile axes such as gender, age, voice characteristic, regional/linguistic background, and the overall speaker profile. A small fraction of sentence-form understanding tasks is also replayed to maintain the grounded cue-reading ability learned in Stage 1. Stage 2 therefore serves as a tuning stage that adds verification reasoning ability on top of SpeakerLLM-Base, without replacing the speaker-understanding variant. 3.3
Prompt Conditioning and Objective
For each task, the textual prompt is converted into a token-embedding sequence Ptext in the languagemodel hidden space. For the active utterance set {xi }m i=1 , where m ∈ {1, 2}, the speaker tokenizer produces one speaker-token block per utterance, Si ∈ RKspk ×d , Kspk = N + M. These speaker-token blocks are inserted into the designated prompt positions, yielding the prompt-side embedding sequence P = Insert (Ptext , {Si }m i=1 ) . The conditioning mechanism is shared across single-utterance QA and utterance-pair QA: one speakertoken block is inserted for single-utterance tasks, and two blocks are inserted for utterance-pair tasks. Given a target sequence y = (y1 , . . . , yL ), SpeakerLLM is trained with the standard autoregressive next-token cross-entropy objective: L X LCE = − log pθ (yt | y<t , P). t=1
During speaker-tokenizer warm-up, only the speaker tokenizer is updated. During sentence adaptation and Verification-Reasoning Tuning, the speaker tokenizer and LoRA parameters are trained jointly. The speaker encoder remains frozen throughout.
4
Experiments
4.1
Experimental Settings
Data. We train SpeakerLLM on the VoxCeleb1 development set [24] and the LibriTTS-R clean360h training list [26], containing 265k utterances and 530.8 hours of measured audio. We use this controlled base-scale setting for all main comparisons and ablations, so that differences across model variants reflect the speaker tokenizer, training trajectory, and verification reasoning target design, not changes in data scale. We center speaker-profile QA and standard same/different speaker judgment evaluation on VoxCeleb1-O because the VoxCeleb domain provides both speaker-profile metadata/enrichment and established same/different trials, allowing profile evidence and speaker judgments to be evaluated in the same utterance domain. Environment QA is evaluated on LibriTTSR test-clean with controlled noise and reverberation labels. During training, each audio input is randomly cropped to a duration between 3 and 15 seconds. At evaluation, we use a fixed 15-second window when the utterance is longer than 15 seconds, and otherwise use the full utterance. Model. SpeakerLLM uses a frozen ReDimNet-B3 [33] speaker encoder and Qwen2.5-1.5BInstruct [34] as the language-model backbone. The hierarchical speaker tokenizer maps the final speaker embedding to 16 embedding-level speaker tokens via an MLP and the frame-level speaker features to 32 sequence-level speaker tokens via a Q-Former [31], yielding 48 speaker tokens per utterance. The speaker encoder remains frozen throughout training, while the language model is adapted with LoRA [32] after the initial speaker-tokenizer warm-up. LoRA uses rank 16, alpha 32, and dropout 0.05. 6
Table 2: Main results on VoxCeleb1-O (SV and speaker-profile QA) and LibriTTS-R test-clean (environment QA). All values are accuracy (% ↑); bold indicates the best result in each column. “–” denotes unsupported tasks or metrics not comparable under our prompted generated-answer accuracy protocol. Speaker Profile
Environment
Model
LLM Backbone
SV
Gender
Age
Region
Pitch
Bright.
Noise
Reverb
General Audio-LLMs Qwen2.5-Omni-7B [5] Qwen3.0-Omni-30B [37] Audio Flamingo3 [7]
Qwen2.5-7B [34] Qwen3-30B-A3B Qwen2.5-7B [34]
65.2 54.0 54.5
99.8 99.1 99.9
17.5 20.3 23.2
76.0 75.7 59.8
22.7 23.8 22.1
25.0 32.1 38.3
20.4 32.7 21.2
20.2 20.9 21.0
Speaker-Specialized LLMs SA-TinyLLaMA [38] TinyLLaMA-1.1B [39] CoLMbo [19] GPT2-base [40]
– –
– 78.6
– 22.7
– 45.5
– –
– –
– –
– –
Ours SpeakerLLM-Base
96.1
99.9
39.8
83.1
72.4
54.2
52.7
51.7
Qwen2.5-1.5B [34]
Training. Training follows the two-stage trajectory in Section 3.2. Stage 1 first trains only the speaker tokenizer with short-form targets and then adapts the language model with LoRA using sentence-form speaker understanding targets, producing SpeakerLLM-Base. Stage 2 further tunes SpeakerLLM-Base with verification reasoning targets, attribute compatibility QA, and a small replay of sentence-form understanding tasks, producing SpeakerLLM-VR. The three training phases run for 126k, 83k, and 221k steps with AdamW [35], stage-specific peak learning rates of 1×10−4 , 4×10−6 , and 6×10−6 , bf16 mixed precision, and FlashAttention-2 [36]. Evaluation and baselines. We report generated-answer accuracy for all LLM-interface tasks. Generated answers are mapped to task labels with deterministic parsers, and parsing failures are counted as incorrect. For utterance-pair speaker judgment, we distinguish two evaluation modes. SV denotes the standard same/different speaker judgment mode, where the model generates a short verdict. SV-R denotes the evidence-organized reasoning mode, where the model generates a full verification reasoning output; SV-R accuracy is computed from the same/different verdict parsed from the generated DECISION block. SV is evaluated as generated-verdict accuracy rather than score-based EER or minDCF, because our LLM-interface protocol evaluates textual verdicts and evidence traces rather than calibrated continuous trial scores. Metadata is used only for label construction and offline analysis, and is not provided to any model at inference time. We compare against general audio-LLMs, including Qwen2.5-Omni-7B [5], Qwen3.0-Omni-30B [37], and Audio Flamingo3 [7], using prompted evaluation with explicit answer options. We also list SA-TinyLLaMA [38] and CoLMbo [19] for scope comparison; unsupported tasks or metrics that are not comparable under our generated-verdict protocol are marked as “–”. Full experimental details are provided in Appendix C. 4.2
Results
Speaker understanding performance. Table 2 summarizes the main LLM-interface benchmark. We test whether a speaker-specialized audio-LLM can jointly support speaker-profile, recordingcondition, and same/different speaker judgment within a single natural-language interface. General audio-LLMs perform well on attributes that are relatively easy to infer from speech, most notably gender. However, their performance drops substantially on tasks that require speaker-specific acoustic evidence or recording-condition awareness, such as age, pitch, timbral brightness, noise, and reverberation. This suggests that general audio-LLMs can capture speech content and some surface paralinguistic cues, but are not designed to retain and interpret fine-grained speaker-identity evidence or voice-quality differences between speakers. Among speaker-specialized LLMs, CoLMbo supports only limited profiling without utterance-pair judgment, and SA-TinyLLaMA reports score-based EER not directly comparable to our generatedverdict protocol. SpeakerLLM-Base achieves the best performance on most speaker-profile and environment attributes, while also obtaining 96.1% generated-verdict accuracy for standard same/different speaker judgment. The gains are especially large for pitch and timbral brightness, indicating that targeted speaker 7
Table 3: Ablation on speaker-token adapter design. All variants use the same frozen encoder and LLM, trained under Stage 1 speaker-tokenizer-only training. Token-count analysis is in Appendix D.1. Speaker Repr. Adapter
Embed.
Linear MLP Q-Former
✓ ✓
Ours (MLP + Q-Former)
✓
Speaker Profile
Environment
Frames
SV
Gender
Age
Region
Pitch
Bright.
Noise
Reverb
✓
49.1 86.5 84.7
77.3 99.1 99.3
16.3 30.2 32.2
49.5 78.1 74.4
17.3 57.4 70.0
18.1 41.3 51.0
23.9 32.6 48.9
22.8 32.5 45.6
✓
95.6
99.7
39.5
79.8
72.3
53.1
47.7
50.0
supervision is important for voice-characteristic understanding. SpeakerLLM-Base also exceeds 50% accuracy on both noise and reverberation, which are five-way classification tasks with a chance level of roughly 20%. This suggests that the model learns recording-condition cues beyond speech-content understanding. Speaker tokenizer analysis. Table 3 compares the adapter design inside the speaker tokenizer. Linear and MLP consume only the final speaker embedding, Q-Former consumes only frame-level speaker features, and the hierarchical speaker tokenizer combines both representations. All variants use the same frozen speaker encoder and LLM backbone and are evaluated under Stage 1 speakertokenizer-only training. The Linear adapter provides a simple projection lower bound and performs poorly across tasks, indicating that dimension matching alone is insufficient for speaker-specialized audio-LLM conditioning. The MLP adapter retains useful global speaker information from the pooled speaker embedding, but is limited on attributes requiring temporal or acoustic detail, such as pitch and timbral brightness. Conversely, the Q-Former adapter improves these acoustic attributes by attending to frame-level speaker features, but trails in standard same/different speaker judgment. The hierarchical tokenizer achieves the best balance across speaker-profile, recording-condition, and same/different judgment tasks, supporting our representation-granularity hypothesis: speaker identity, profile-level cues, and acoustic voice descriptors benefit from different levels of speaker representation. Appendix D.1 further shows that this gain persists when MLP-only and Q-Former-only variants are given the same total number of speaker tokens, confirming that the benefit comes from combining complementary representation levels, not from token count alone. Verification reasoning quality. We next Table 4: SV-R evidence grounding on VoxCeleb1evaluate SpeakerLLM-VR, the model variO. Format Valid measures three-block schema vaant obtained by further tuning SpeakerLLMlidity; Attr.-level and Prof.-support measure profileBase with verification reasoning targets. evidence agreement at clause and summary levels. SpeakerLLM-VR generates an evidenceAll values are accuracy (% ↑). organized decision trace containing environSchema Profile Evidence Grounding ment status, profile-level evidence, and the final same/different decision. We evaluate this Fmt. Valid Attr.-level Prof.-support output from two complementary perspectives: SpeakerLLM-VR 100.0 72.7 63.6 whether the generated trace is grounded in (maj. 52.9) the supervised evidence schema, and how the structured trace affects the final generated verdict across trial subsets. Table 4 evaluates the grounding quality of the generated SV-R trace. SpeakerLLM-VR produces valid three-block outputs in all trials (Format Valid = 100.0%), indicating reliable adherence to the prescribed evidence schema. At the clause level, individual profile-comparison statements reach 72.7% attribute-level grounding against the supervised evidence labels. At the summary level, profile-support grounding compares the generated profile evidence with the target three-way profile-support state (SUPPORTIVE, MIXED, CONFLICTING). SpeakerLLM-VR reaches 63.6% profilesupport grounding, exceeding the 52.9% majority baseline. The summary-level score is lower because it aggregates multiple clause-level comparisons into a single three-way state, making it a stricter metric. Together, full format validity and grounding at both clause and summary levels support evidence-grounded faithfulness to the supervised verification reasoning schema. Table 5 provides a subset diagnostic of the final generated verdict. SV-R maintains the overall generated-verdict accuracy while adding an evidence-organized decision trace, improving the overall score by +0.33 percentage points. The gain is larger on different-speaker trials (+0.96 points), and is most pronounced on supportive-profile different trials (+1.47 points), where the available profile 8
Table 5: Subset diagnostic for generated same/different verdicts on VoxCeleb1-O. SV uses the short verdict mode, whereas SV-R parses the verdict from the generated DECISION block. Aligned, mixed, and reversal splits are defined by the relation between profile-support level and the ground-truth label. Supp→diff and Confl→same denote the two reversal subsets. ∆ denotes SV-R minus standard SV accuracy. All values are accuracy (% ↑). GT-label split
Profile-support split
Hardest reversal
SpeakerLLM-VR
Diff.
Same
Aligned
Mixed
Reversal
Supp→diff
Confl→same
Overall
SV SV-R
94.25 95.20
99.32 99.03
97.31 97.87
95.22 95.32
97.15 96.81
78.53 80.00
99.11 98.58
96.79 97.12
∆
+0.96
−0.30
+0.56
+0.10
−0.34
+1.47
−0.53
+0.33
evidence is compatible but the speakers are different. This subset corresponds to profile-deceptive negative trials, where surface profile similarity can otherwise encourage a short verdict to predict “same”. Small decreases on same-speaker and conflicting-profile same-speaker subsets reflect nearceiling fluctuations, as these subsets already exceed 97–99% under the short verdict mode. Thus, SV-R is not intended as a uniformly stronger classifier than direct verdict generation. Rather, it maintains final verdict quality overall while exposing an evidence-organized trace, with the largest gains appearing where profile-similarity shortcuts are most harmful. Representative generated traces are provided in Appendix D.3. Training ablation. Table 6(a) shows that removing the speaker-tokenizer warm-up substantially degrades the Stage 1 variant across standard SV, speaker-profile QA, and environment QA, indicating that the speaker tokenizer benefits from an initial alignment phase before language-model adaptation. A per-attribute breakdown is provided in Appendix D.2.
Table 6: Training ablation. (a) Tokenizer warm-up for Stage 1. (b) Task trajectory for VR Tuning. Profile and Environ. denote mean accuracies. (a) Tokenizer warm-up Setting
SV
Profile
Environ.
w/o warm-up SpeakerLLM-Base
91.20 96.05
67.16 72.90
40.46 52.21
Table 6(b) shows that mixing Stage 1 and Stage 2 (b) Task trajectory tasks immediately after tokenizer warm-up achieves Setting SV SV-R marginally higher standard SV accuracy but sharply degrades SV-R accuracy. Thus, high standard w/o trajectory 97.09 91.71 same/different accuracy alone does not imply SpeakerLLM-VR 96.79 97.12 that the model has learned the SV-R interface; SpeakerLLM-VR achieves the best SV-R accuracy when further tuned after the speaker-understanding variant.
5
Conclusion
We presented SpeakerLLM, a speaker-specialized audio-LLM framework that unifies speakerprofile understanding, recording-condition understanding, utterance-pair speaker comparison, and evidence-organized verification reasoning in a natural-language interface. Our results show that multi-granularity speaker conditioning improves speaker understanding, and that verification reasoning targets with profile-support computation and reversal-aware decision composition produce valid, schema-grounded decision traces without sacrificing generated same/different verdict accuracy. SpeakerLLM provides a controlled starting point for speaker-specialized audio-LLMs that expose speaker evidence in language rather than only as backend scores.
6
Limitations and Future Work
This work studies controlled LLM-interface speaker evidence modeling. Our faithfulness analysis is grounded in the supervised reasoning schema rather than causal internal mechanisms; future work should add counterfactual, intervention-based, and human-grounded evaluations, and connect the language interface to calibrated score-based SV backends when thresholded operation is required. Scaling to larger in-the-wild corpora, broader languages and accents, and real noisy or far-field recordings is another natural extension. More broadly, voice-only agents require consent-aware interfaces for user enrollment, authorization, personalization, and speaker-aware user understanding. Because the framework uses biometric speaker identity and profile attributes, deployment should include consent-aware data use, privacy-preserving inference, and fairness evaluation across demographic and acoustic conditions. 9
References [1] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024. [2] Francis Rakotomalala, Hasindraibe Niriarijaona Randriatsarafara, Aimé Richard Hajalalaina, and Ndaohialy Manda Vy Ravonimanantsoa. Voice user interface: Literature review, challenges and future directions. System Theory, Control and Computing Journal, 1(2):65–89, 2021. [3] Xiaoan Liu, DaeHo Lee, Eric J Gonzalez, Mar Gonzalez-Franco, and Ryo Suzuki. Visionclaw: Always-on ai agents through smart glasses. arXiv preprint arXiv:2604.03486, 2026. [4] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757–15773, 2023. [5] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. [6] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=14rn7HpKVk. [7] Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, ChaoHan Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URL https://openreview.net/forum?id=FjByDpDVIO. [8] Najim Dehak, Patrick J. Kenny, Réda Dehak, Pierre Dumouchel, and Pierre Ouellet. Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing, 19(4):788–798, 2011. doi: 10.1109/TASL.2010.2064307. [9] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X-Vectors: Robust DNN embeddings for speaker recognition. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5329– 5333, 2018. doi: 10.1109/ICASSP.2018.8461375. [10] Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4879–4883, 2018. doi: 10.1109/ICASSP.2018.8462665. [11] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1–38, 2019. [12] Q Vera Liao, Daniel Gruen, and Sarah Miller. Questioning the ai: informing design practices for explainable ai user experiences. In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1–15, 2020. [13] Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. In Interspeech 2020, pages 3830–3834, 2020. doi: 10.21437/Interspeech.2020-2650. [14] Xiaoliang Wu, Chau Luu, Peter Bell, and Ajitha Rajan. Explainable attribute-based speaker verification. arXiv preprint arXiv:2405.19796, 2024. [15] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338–5348. PMLR, 2020. 10
[16] Jaejun Lee and Kyogu Lee. Vo-Ve: An Explainable Voice-Vector for Speaker Identity Evaluation. In Interspeech 2025, pages 3988–3992, 2025. doi: 10.21437/Interspeech.2025-591. [17] Yizhe Yang, Palakorn Achananuparp, He-Yan Huang, Jing Jiang, and Ee-Peng Lim. Speaker verification in agent-generated conversations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5655–5676, 2024. [18] Han Yin, Yafeng Chen, Chong Deng, Luyao Cheng, Hui Wang, Chao-Hong Tan, Qian Chen, Wen Wang, and Xiangang Li. SpeakerLM: End-to-end versatile speaker diarization and recognition with multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 34467–34475, 2026. [19] Massa Baali, Shuo Han, Syed Abdul Hannan, Purusottam Samal, Karanveer Singh, Soham Deshmukh, Rita Singh, and Bhiksha Raj. Colmbo: Speaker language model for descriptive profiling. arXiv preprint arXiv:2506.09375, 2025. [20] Phil Rose. Forensic speaker identification. cRc Press, 2002. [21] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?id=_VjQlMeSB_J. [22] Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023. [23] Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965, 2023. [24] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb: A Large-Scale Speaker Identification Dataset. In Interspeech 2017, pages 2616–2620, 2017. doi: 10.21437/Interspeech. 2017-950. [25] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. VoxCeleb2: Deep Speaker Recognition. In Interspeech 2018, pages 1086–1090, 2018. doi: 10.21437/Interspeech.2018-1929. [26] Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna. LibriTTS-R: A Restored MultiSpeaker Text-to-Speech Corpus. In Interspeech 2023, pages 5496–5500, 2023. doi: 10.21437/ Interspeech.2023-1584. [27] Khaled Hechmi, Trung Ngo Trong, Ville Hautamäki, and Tomi Kinnunen. Voxceleb enrichment for age and gender recognition. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 687–693. IEEE, 2021. [28] Masaya Kawamura, Ryuichi Yamamoto, Yuma Shirahata, Takuya Hasumi, and Kentaro Tachibana. LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning. In Interspeech 2024, pages 1850–1854, 2024. doi: 10.21437/Interspeech.2024-692. [29] David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise corpus. arXiv preprint arXiv:1510.08484, 2015. [30] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L. Seltzer, and Sanjeev Khudanpur. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5220–5224, 2017. doi: 10.1109/ICASSP.2017.7953152. [31] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023. 11
[32] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview. net/forum?id=nZeVKeeFYf9. [33] Ivan Yakovlev, Rostislav Makarov, Andrei Balykin, Pavel Malov, Anton Okhotnikov, and Nikita Torgashov. Reshape Dimensions Network for Speaker Recognition. In Interspeech 2024, pages 3235–3239, 2024. doi: 10.21437/Interspeech.2024-2116. [34] Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115. [35] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=Bkg6RiCqY7. [36] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=mZn2Xyh9Ec. [37] Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765, 2025. [38] Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez, Jesus Villalba-Lopez, and Najim Dehak. Speaker verification with speech-aware llms: Evaluation and augmentation. arXiv preprint arXiv:2603.10827, 2026. [39] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. TinyLLaMA: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024. [40] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [41] Manfred R. Schroeder. New method of measuring reverberation time. The Journal of the Acoustical Society of America, 37(3):409–412, 1965. doi: 10.1121/1.1909343.
12
Technical Appendices and Supplementary Material This supplementary material complements the main paper by providing the following sections. To support code reproducibility, we will release the source code along with a README file. A
Attribute and Metadata Details A.1 Detailed Attribute Taxonomy and Binning A.2 Environment Label Computation A.3 Acoustic Feature Extraction A.4 Metadata Coverage
14 14 14 14 15
B
Supervision Design and Target Construction B.1 Stage-1 Target Templates B.2 Stage-2 Target Templates B.3 Profile-Support Computation B.4 Decision Composition Rules B.5 Worked Examples
15 15 15 16 18 20
C
Details of Experimental Settings C.1 Dataset Statistics C.2 Model and Speaker-Tokenizer Hyperparameters C.3 Optimization Schedule C.4 Augmentation Protocol C.5 Evaluation Protocol C.6 Baseline Prompting
20 21 21 22 22 22 22
D
Detailed Results D.1 Speaker-Token Count Ablation D.2 Per-Attribute Analysis of Speaker-Tokenizer Warm-Up D.3 Representative SV-R Cases
23 23 24 24
13
Table 7: Detailed attribute taxonomy and binning rules used for supervision label construction. Pitch is discretized using gender-conditioned F0 percentile cutoffs, while timbral brightness uses gender-shared spectral-centroid percentile cutoffs. Noise and reverberation labels are generated by controlled simulation using SNR and pre-computed RT60, respectively. Attribute
Class
Boundary / rule
Speaker profile Gender
male / female
From public metadata (VoxCeleb Enrichment, LibriTTS-P).
Age
10 ordinal bins
1–7, 8–12, 13–17, 18–25, 26–35, 36–45, 46–55, 56–65, 66– 75, 76+. Derived from VoxCeleb Enrichment age estimates.
Region
8 classes
Nationality → regional/linguistic background mapping: North American, European, British/Irish, Latin/Hispanic, Oceanian, East/Southeast Asian, Middle Eastern/North African, African.
Pitch
5 gender-conditioned bins
Per-utterance mean F0 is ranked within each gender’s pooled distribution. Percentile cutoffs at 10%, 30%, 70%, 90% yield very low, low, normal, high, very high.
Timbral brightness
5 bins (gender-shared)
Per-utterance mean spectral centroid is ranked within the pooled distribution. Same percentile cutoffs (10/30/70/90%) yield muted, mellow, neutral, bright, brilliant.
Environment (online simulation) Noise
5 SNR-controlled classes
clean (≥ 20 dB), mild ([10, 20) dB), moderate ([5, 10) dB), severe ([0, 5) dB), extreme (< 0 dB).
Reverberation
5 RT60-controlled classes
minimal (≤ 0.3 s), slight ((0.3, 0.6] s), moderate ((0.6, 1.0] s), heavy ((1.0, 1.5] s), extreme (> 1.5 s).
A
Attribute and Metadata Details
This appendix specifies the attribute definitions used to construct SpeakerLLM supervision. It provides the binning boundaries, acoustic feature extraction procedure, and per-corpus metadata coverage underlying the attribute taxonomy summarized in the main paper. A.1
Detailed Attribute Taxonomy and Binning
Table 7 lists the class definitions and binning rules used to convert metadata, acoustic measurements, and simulated recording conditions into discrete supervision labels. A.2
Environment Label Computation
Noise labels are controlled by the signal-to-noise ratio (SNR) between the clean speech waveform s and the scaled non-speech noise waveform αn: P 2 s SNRdB = 10 log10 P t t 2 . (αn t) t The scale α is chosen to match the target SNR interval. For reverberation, we estimate broadband RT60 for each SLR28 room impulse response using Schroeder backward integration [41]. We fit a T30-style line to the −5 to −35 dB region of the energy decay curve and extrapolate it to 60 dB decay. The resulting RIR-level RT60 estimate is used for reverberation bucketing (Table 7); it is not intended as a full room-acoustic measurement protocol. A.3
Acoustic Feature Extraction
Pitch and timbral brightness are extracted from audio for both VoxCeleb [24] and LibriTTS-R [26] utterances. Pitch is derived from the per-utterance mean fundamental frequency (F0), estimated using librosa.pyin. Timbral brightness is derived from the per-utterance mean spectral centroid computed over STFT frames. Because F0 distributions differ substantially between male and 14
female speakers, pitch is discretized using gender-conditioned percentile cutoffs (Table 7), so that a given pitch label reflects relative position within the speaker’s gender group rather than an absolute frequency range. These acoustic descriptors are used only for training instances that require pitchor brightness-conditioned labels. Approximately 11,000 utterances (< 1% of the combined corpus) failed F0 extraction and are therefore excluded from pitch- and brightness-conditioned instances, while remaining available for tasks that do not require these descriptors. A.4
Metadata Coverage
Table 8 reports the fraction of utterances for which each speaker-profile attribute is available, broken down by source corpus. Because metadata availability is partial, attribute-specific training instances are generated only when the corresponding field exists. Missing attributes are not imputed. Table 8: Per-corpus metadata coverage used for profile-label construction. Pitch and timbral brightness are audio-derived descriptors; their coverage is reported after excluding utterances for which acoustic feature extraction fails. Attribute Gender Age Region Pitch Timbral brightness
B
VoxCeleb
LibriTTS-R
99.8% 86.6% 99.2% 99.6% 99.6%
99.7% 0% 0% 98.4% 98.4%
Supervision Design and Target Construction
This appendix documents how SpeakerLLM training targets are constructed. It covers three components: (i) the fixed target templates used in the Speaker Understanding stage and the VerificationReasoning Tuning stage (Tables 9–10), (ii) the penalty-based profile-support computation used by the verification reasoning target (Table 11), and (iii) the decision composition rules that map environment severity, profile-support level, and the ground-truth same/different label into the final DECISION block (Table 13). B.1
Stage-1 Target Templates
Stage 1 supports Speaker Understanding and uses the same scenario inventory across two target styles. During speaker-tokenizer warm-up, short-form targets are used to stabilize cue decoding and align audio-derived speaker representations with the language model input space. During sentence adaptation, the same scenarios are expressed as sentence-form targets. Table 9 lists the template for each scenario. Slots enclosed in braces (e.g., {gender}) are filled from the available per-utterance labels at training time. B.2
Stage-2 Target Templates
Stage 2 supports Verification-Reasoning Tuning and introduces two task families: attribute compatibility QA and verification reasoning targets. All Stage 2 targets are sentence-form from the outset. Table 10 lists the templates. The PROFILE _ COMPATIBILITY block of the verification reasoning target reuses the same atomic clause builders as the compatibility QA tasks (e.g., gender, age, region, and voice clauses), but omits the holistic summary sentence. The overall profile summary is instead composed inside the DECISION block using the profile-support level (see §B.4). Full speaker-profile and SV-R targets require the complete profile-attribute set for the involved utterance(s). Because age and region annotations are available only for VoxCeleb, these targets are generated on VoxCeleb utterances or pairs. The ENVIRONMENT _ STATUS block in SV-R is populated by applying the same online noise/reverberation simulation used for environment supervision to the corresponding VoxCeleb utterances. 15
Table 9: Stage 1 target templates for Speaker Understanding. Short form targets are used during speaker-tokenizer warm-up, and sentence form targets are used during sentence adaptation. The full speaker profile and joint acoustic profile targets are constructed by concatenating the available atomic sentences from the corresponding attribute rows. Group Scenario
Short form
Sentence form
Speaker profile (single-utterance) Gender
{gender}
Age
{age_phrase}
Region
{region}
Voice characteristic
{brightness} voice and {pitch} {gender}-range pitch
Full speaker profile
comma-separated atomic phrases
The speaker’s gender is inferred to be {gender}. The speaker’s age is {age_phrase}. The speaker’s regional background is {region}. The speaker has a {brightness} voice and {pitch} {gender}-range pitch. Concatenation of available atomic sentences: The speaker is {gender}. The speaker is likely {age_phrase}. The speaker has a {region} regional background. The speaker has a {brightness} voice and {pitch} {gender}-range pitch.
Environment (single-utterance) Noise class
{noise_class}
Reverberation class
{reverb_class}
Joint acoustic profile {noise} noise and {reverb} reverberation
The recording has {noise_class} noise. The recording has {reverb_class} reverberation. The recording has {noise_class} noise and {reverb_class} reverberation.
Utterance-pair tasks
B.3
standard SV
same / different
Noise comparison
speech1 / speech2 / similar
Reverb comparison
speech1 / speech2 / similar
These recordings are from the same speaker. These recordings are from different speakers. Speech 1 is noisier. / Speech 2 is noisier. Both recordings have similar noise levels. Speech 1 is more reverberant. / Speech 2 is more reverberant. Both recordings have similar reverberation levels.
Profile-Support Computation
The verification reasoning target uses an internal profile-support level with respect to a same-speaker interpretation: {SUPPORTIVE, MIXED, CONFLICTING}. This level summarizes whether the available speaker-profile attributes support, ambiguously mix with, or conflict with a same-speaker interpretation before the final verification verdict is composed. It is computed by summing attribute-level penalties over attributes that are available for both utterances (Table 11). The profile-support level is not itself the final verification label. Instead, it provides an intermediate profile-evidence state used by the decision composition rules in §B.4. This separation allows the target to include reversal cases, where surface profile evidence and the ground-truth verification label point in different directions. 16
Table 10: Stage 2 target templates for Verification-Reasoning Tuning. Compatibility QA targets compare one profile axis at a time, while the holistic variant concatenates the available profilecomparison clauses and adds an overall summary. The verification reasoning target follows a threeblock format: ENVIRONMENT _ STATUS, PROFILE _ COMPATIBILITY, and DECISION; the DECISION block is constructed according to the composition rules in Table 13. Family Subtype
Target template
Attribute compatibility QA (utterance-pair comparison) Gender Age
Region Voice
Holistic
Gender is similar. or Gender is different. The age ranges are similar. The age ranges are slightly different. The age ranges are very different. Linguistic background is similar. or Linguistic background is different. Pitch is {similar | somewhat different | very different}. Timbral brightness is {similar | somewhat different | very different}. Therefore, the vocal characteristics are {similar | somewhat different | very different}. Concatenation of all available atomic clauses above, followed by: Therefore, the overall speaker profile is {similar | somewhat different | very different}.
Verification reasoning target (three-block format) ENVIRONMENT _ STATUS
The first recording contains {noise1 } noise and {reverb1 } reverberation. The second recording contains {noise2 } noise and {reverb2 } reverberation.
PROFILE _ COMPATIBILITY
Same atomic clauses as compatibility QA, but without the overall summary sentence.
DECISION
Composed from environment severity × profile-support level × groundtruth same/different label. See Table 13 for the full composition rules.
Final target
ENVIRONMENT_STATUS: {environment_status text} PROFILE_COMPATIBILITY: {profile_compatibility text} DECISION: {decision text}
Gender receives the largest single penalty (+4), reflecting that a gender mismatch often co-occurs with divergence in downstream vocal cues. Region receives a deliberately smaller penalty (+1) to avoid over-penalizing bilingual, multilingual, or cross-regional speakers. Pitch and timbral brightness are treated as a grouped voice factor: the voice penalty is the maximum of the two sub-attribute penalties, with an additional +1 bonus when both sub-attributes are non-compatible. This captures the intuition that a joint mismatch in vocal range and timbral color is stronger evidence than either cue alone. For ordered attributes (age, pitch, timbral brightness), compatibility is determined by bin distance: identical bins are compatible (p = 0), adjacent bins are partial (p = 1), and bins separated by two or more steps are conflicting (p = 2). For pitch, the ordered scale follows gender-conditioned F0 percentile bins (Appendix A.1); for timbral brightness, the order is muted < mellow < neutral < bright < brilliant. Gender and region are binary: a mismatch incurs the penalty listed in Table 11; a match contributes zero. 17
Table 11: Profile-support scoring used by the verification reasoning target. Penalties are summed over attributes that are comparable for both utterances; missing attributes contribute no penalty. The grouped voice factor combines pitch and timbral brightness using a max-based penalty with an additional bonus for joint mismatch. Attribute
Condition
Penalty
Gender
different
+4
Age
adjacent bin gap ≥ 2 bins different
+1 +2 +1
Region
one sub-attr non-compatible max(ppitch , pbright ) both sub-attrs non-compatible max(ppitch , pbright ) + 1 where each sub-attr penalty p ∈ {0 (compatible), 1 (partial), 2 (conflicting)}
Voice (grouped)
total ≤ 1 total = 2–3, or no fields total ≥ 4
SUPPORTIVE MIXED CONFLICTING
B.4
Decision Composition Rules
The DECISION block is constructed from four phrases selected by three input variables. These rules are used to construct supervision targets; they are not provided to the model at inference time. 1. Environment clause — selected by pair severity (Table 12). 2. Profile summary — selected by profile-support level. 3. Connector — selected by the alignment between profile-support level and the ground-truth same/different label. 4. Verification verdict — selected by the same alignment. The full DECISION is their concatenation: DECISION = env clause + profile summary + connector + verification verdict
|
{z
}
pair severity
{z
|
profile support×GT label
}
This construction separates profile-level evidence from the final same/different verdict, allowing the target to express both aligned cases and reversal cases in a linguistically coherent form. Environment clause. The environment clause is selected from a three-way pair-severity level (Table 12). It describes how much the recording conditions may degrade speaker-relevant cues and is independent of the profile-support level and ground-truth verification label. Pair severity is derived deterministically from the per-utterance environment labels. Each noise class and reverberation class is assigned an ordinal degradation rank from 0 (clean/minimal) to 4 (extreme). The degradation rank of a single recording is the maximum of its noise and reverberation ranks. The pair-severity rank is then the maximum degradation rank over the two recordings. Pair severity is defined as Low if the pair rank is 0–1, Moderate if 2, and Extreme if 3–4. Profile summary, connector, and verification verdict. The remaining three phrases are jointly determined by profile-support level × ground-truth same/different label, yielding six cases (Table 13). The connector indicates whether the profile-level evidence and the final verification verdict point in the same direction (Likewise) or diverge (However). We mark reversal cases (⋆), where profile evidence and the ground-truth label disagree. These cases are included to prevent a shortcut in which surface profile similarity mechanically determines the verification verdict. Connector logic. Likewise is used when profile-level evidence and the ground-truth label are aligned (SUPPORTIVE+same or CONFLICTING+different), indicating that latent speaker-identity cues confirm the surface-level profile tendency. However is used in reversal cases and in the MIXED+different case, indicating that the final verdict departs from what the profile evidence alone would suggest. For 18
Table 12: Environment clause selection for the first sentence of the DECISION block. Pair severity determines how strongly recording-condition mismatch or degradation is described as affecting speaker-relevant cues. Pair severity
Environment clause
Low
Environmental mismatch or degradation is limited, so the speaker-relevant vocal cues remain clear. Environmental mismatch or degradation is present, so the speaker-relevant vocal cues are partially degraded. Strong environmental mismatch or severe degradation substantially weakens the speaker-relevant vocal cues.
Moderate Extreme
Table 13: Decision composition matrix. Each cell specifies the profile-summary clause, connector, and verification-verdict clause for one combination of profile-support level and ground-truth same/different label. ⋆ marks reversal cases, where profile-level evidence and the final verification verdict disagree. GT = same speaker
GT = different speakers
Supportive
Across the speaker profile, many attributes are similar. Likewise, the latent speaker-identity cues also show strong similarity. Taken together, the recordings are determined to be from the same speaker. [aligned]
⋆ Across the speaker profile, many attributes are similar. However, the latent speaker-identity cues show stronger separation. Taken together, the recordings are determined to be from different speakers. [reversal]
Mixed
Across the speaker profile, some attributes are similar, while others differ. The latent speakeridentity cues show stronger similarity. Taken together, the recordings are determined to be from the same speaker. [neutral]
⋆ Across the speaker profile, some attributes are similar, while others differ. However, the latent speaker-identity cues show stronger separation. Taken together, the recordings are determined to be from different speakers. [reversal]
Conflicting
⋆ Across the speaker profile, several attributes differ. However, the latent speaker-identity cues show stronger similarity. Taken together, the recordings are determined to be from the same speaker. [reversal]
Across the speaker profile, several attributes differ. Likewise, the latent speaker-identity cues also show clear differences. Taken together, the recordings are determined to be from different speakers. [aligned]
MIXED +same, no connector is inserted because the profile evidence is already ambiguous and does not require contrastive framing.
19
B.5
Worked Examples
Below are complete target examples produced by the verification reasoning target construction policy. They illustrate an aligned supportive case, a reversal case, and an aligned conflicting case. These examples are target templates used for supervision, not model-generated outputs. Example 1 — Aligned: supportive profile, same speaker ENVIRONMENT _ STATUS: The first recording contains no background noise and minimal reverberation. The second recording contains mild noise and slight reverberation. PROFILE _ COMPATIBILITY: Gender is similar. The age ranges are similar. Linguistic background is similar. Pitch is similar. Timbral brightness is similar. DECISION : Environmental mismatch or degradation is limited, so the speaker-relevant vocal cues remain clear. Across the speaker profile, many attributes are similar. Likewise, the latent speaker-identity cues also show strong similarity. Taken together, the recordings are determined to be from the same speaker. Example 2 — Reversal: supportive profile, different speakers (⋆) ENVIRONMENT _ STATUS: The first recording contains moderate noise and slight reverberation. The second recording contains mild noise and moderate reverberation. PROFILE _ COMPATIBILITY: Gender is similar. The age ranges are similar. Linguistic background is similar. Pitch is somewhat different. Timbral brightness is similar. DECISION : Environmental mismatch or degradation is present, so the speaker-relevant vocal cues are partially degraded. Across the speaker profile, many attributes are similar. However, the latent speakeridentity cues show stronger separation. Taken together, the recordings are determined to be from different speakers. Example 3 — Aligned: conflicting profile, different speakers ENVIRONMENT _ STATUS: The first recording contains severe noise and heavy reverberation. The second recording contains severe noise and heavy reverberation. PROFILE _ COMPATIBILITY: Gender is similar. The age ranges are very different. Linguistic background is similar. Pitch is very different. Timbral brightness is somewhat different. DECISION : Strong environmental mismatch or severe degradation substantially weakens the speaker-relevant vocal cues. Across the speaker profile, several attributes differ. Likewise, the latent speakeridentity cues also show clear differences. Taken together, the recordings are determined to be from different speakers.
C
Details of Experimental Settings
This appendix provides reproducibility details corresponding to Section 4.1. It includes dataset statistics, model and speaker-tokenizer hyperparameters, optimization schedules, augmentation protocol, evaluation protocol, and baseline prompting. Attribute definitions and binning boundaries are documented separately in Appendix A. 20
C.1
Dataset Statistics
Table 14 summarizes the training data used in the controlled base-scale setting adopted for all main comparisons and ablations. Table 15 lists the evaluation sets used for SV, speaker-profile QA, and environment QA. Table 14: Training data used in the controlled base-scale setting. Hours are measured from waveform headers. Speaker counts are corpus-level counts because VoxCeleb and LibriTTS-R use different speaker namespaces. Corpus / list
Utt.
Spk.
Hours
VoxCeleb1-dev LibriTTS-R clean-360h
148,642 116,462
1,211 904
340.4 190.4
Total
265,104
2,115
530.8
Table 15: Evaluation sets. VoxCeleb1-O provides the SV trials; speaker-profile QA is evaluated on the unique utterances appearing in those trials. LibriTTS-R test-clean is used for environment QA under controlled noise and reverberation labels. Set VoxCeleb1-O LibriTTS-R test-clean
C.2
Samples
Spk.
Hours
37,611 trials 4,837 utt.
40 39
10.8 8.5
Used for SV + profile QA environment QA
Model and Speaker-Tokenizer Hyperparameters
Tables 16 and 17 report the model configuration and trainable parameter counts used in the main experiments. Table 16: Model and speaker-tokenizer hyperparameters. The speaker encoder (ReDimNet-B3 [33]) is frozen, while the speaker-token adapter and LoRA parameters are trained according to the phase schedule in Table 18. Component
Setting
Speaker encoder Speaker embedding dim Frame-level feature dim Language model LM hidden dim Embedding-level tokens (MLP prefix) Sequence-level tokens (Q-Former) Total speaker tokens per utterance Q-Former hidden size Q-Former layers Q-Former attention heads Q-Former intermediate size LoRA target modules LoRA rank / alpha / dropout
ReDimNet-B3, frozen, pretrained on VoxCeleb2 192 1,152 Qwen2.5-1.5B-Instruct 1,536 16 32 48 512 4 8 2,048 q_proj, k_proj, v_proj, o_proj 16 / 32 / 0.05
Table 17: Trainable parameter counts. The speaker encoder and base language-model weights remain frozen; trainable parameters come from the speaker tokenizer and LoRA modules. Module
Parameters
Speaker tokenizer LoRA Speaker tokenizer + LoRA
21
21.2M 5.5M 26.7M
C.3
Optimization Schedule
Table 18 reports the per-phase training schedule. Table 19 lists optimization settings shared across all phases. Table 18: Training schedule across speaker-tokenizer warm-up, sentence adaptation, and VerificationReasoning Tuning. The first phase trains only the speaker tokenizer; the later phases train the speaker tokenizer together with LoRA parameters. Phase
Trainable
Targets
Steps Batch
Understanding tokenizer short-form 126,240 Sent. adapt. tok. + LoRA sentence-form 82,840 VR tuning tok. + LoRA verif. reasoning + compat. 220,920
Peak LR
Min LR Warmup
1.0×10−4
6,312 4,142 11,046
42 5.0×10−5 32 4.0×10−6 2.0×10−6 24 6.0×10−6 3.0×10−6
Table 19: Optimization settings shared across all training phases.
C.4
Setting
Value
Optimizer Weight decay Precision Attention Scheduler Hardware
AdamW 1.0×10−2 bf16 mixed precision FlashAttention-2 cosine annealing with warmup A6000 48 GB × 4
Augmentation Protocol
Noise and reverberation labels are generated online through acoustic augmentation. The severity class definitions for these labels are provided in Appendix A.1. For VoxCeleb utterances, noise and reverberation are each applied with probability 0.5, yielding approximately equal proportions of noise-only, reverberation-only, joint, and clean conditions. For LibriTTS-R utterances, noise-only, reverberation-only, and joint augmentation are each applied with probability 0.3, with the remaining 0.1 probability left as clean. When both factors are applied, reverberation is convolved first, followed by additive noise. C.5
Evaluation Protocol
Table 20 summarizes the evaluation protocol for each task group. Table 20: Evaluation protocol by task group. All tasks are evaluated as generated-answer accuracy after deterministic label parsing. SV denotes the standard same/different verdict mode, while SV-R uses the final verdict parsed from the generated DECISION block. Task
Output
Metric
SV SV-R Profile QA Environment QA
same/different verdict verdict in DECISION block closed attribute label closed condition label
accuracy accuracy accuracy accuracy
Parsing failures are counted as incorrect. At evaluation, when an utterance exceeds 15 seconds, we use a center crop of 15 seconds; shorter utterances are used in full. Metadata is used only for label construction and offline analysis, and is not provided to models at inference time. C.6
Baseline Prompting
Table 21 summarizes the prompting protocol used for each baseline model. 22
Table 21: Baseline prompting protocol. General audio-LLMs are evaluated with our closed-option prompts, while CoLMbo [19] is evaluated with its native prompt for supported profiling attributes. SA-TinyLLaMA [38] is listed for scope comparison because it reports score-based EER rather than prompted generated-answer accuracy. Baseline
Protocol
Notes
Qwen2.5-Omni-7B our closed-option prompts class options provided Qwen3.0-Omni-30B our closed-option prompts class options provided Audio Flamingo3 our closed-option prompts class options provided CoLMbo native prompt where supported gender, age, and region only SA-TinyLLaMA not evaluated under our prompted protocol reported score-based EER
For dataset-specific labels such as pitch, timbral brightness, noise, and reverberation, we include the class options and short class definitions in the prompt (see Appendix A.1 for the class taxonomy).
D
Detailed Results
This appendix expands the analyses summarized in the main paper. Section D.1 controls the number of speaker tokens to test whether the hierarchical tokenizer’s gain can be explained by token count alone. Section D.2 provides a per-attribute breakdown of the speaker-tokenizer warm-up ablation. Section D.3 provides representative generated SV-R traces. D.1
Speaker-Token Count Ablation
Table 3 in the main paper compares speaker-token adapter architectures under their default token configurations. Here, we additionally control the number of speaker tokens for MLP-only and Q-Former-only adapters to test whether the hierarchical tokenizer’s gain is attributable to token count alone. All variants are trained under Stage 1 speaker-tokenizer-only training with the same frozen ReDimNet-B3 [33] encoder and Qwen2.5-1.5B [34] backbone. Table 22: Speaker-token count ablation by adapter type. MLP-only and Q-Former-only adapters are evaluated with 32 and 48 speaker tokens. The hierarchical speaker tokenizer uses 16 embedding-level tokens and 32 sequence-level tokens, for a total of 48 speaker tokens. All values are accuracy (% ↑) under Stage 1 speaker-tokenizer-only training. Speaker Repr. Adapter MLP MLP
Environment
Embed. Frames Tokens SV Gender Age Region Pitch Bright. Noise Reverb ✓ ✓
Q-Former Q-Former Ours (MLP + Q-Former)
Speaker Profile
✓
32 86.5 48 93.7
99.1 98.9
30.2 40.5
78.1 80.7
57.4 52.3
41.3 37.6
32.6 30.3
32.5 32.2
✓ ✓
32 84.7 48 90.7
99.3 99.4
32.2 38.3
74.4 82.8
70.0 71.7
51.0 51.0
48.9 47.1
45.6 48.0
✓
16 + 32 95.6
99.7
39.5
79.8
72.3
53.1
47.7
50.0
Increasing the number of speaker tokens from 32 to 48 improves SV accuracy for both single-source adapters: MLP improves from 86.5 to 93.7, and Q-Former improves from 84.7 to 90.7. Thus, additional token capacity helps speaker-identity discrimination. However, the attribute-level patterns show that capacity alone does not remove the limitations of each representation source. For MLP, increasing token count improves SV and identity-correlated attributes such as age (30.2 → 40.5) and region (78.1 → 80.7), but pitch and timbral brightness degrade (57.4 → 52.3 and 41.3 → 37.6). For Q-Former, increasing token count improves SV and most profile attributes, but the 48-token Q-Former still remains below the hierarchical tokenizer in SV accuracy (90.7 vs. 95.6) and is weaker on global attributes such as age. These results support the representation-granularity hypothesis: the hierarchical tokenizer improves performance by combining embedding-level and sequence-level evidence, rather than by token count alone. 23
Table 24: Representative cases where SV fails but SV-R is correct. Metadata and profile-support states are for offline analysis only. Dec. neg. = profile-deceptive negative; Deg. pos. = degradationaware positive. Case
GT
Dec. neg.
diff. same
diff.
Deg. pos.
same diff.
same Gender, age, region compati- “Environmental degradation is present . . . vocal cues ble; pitch and brightness dif- partially degraded. The latent speaker-identity cues fer slightly; moderate noise in show strong similarity. . . . determined to be from the first recording. same speaker.”
D.2
SV
SV-R Offline evidence
SV-R decision excerpt
All profile attributes (gender, “Many attributes are similar. However, the latent age, region, pitch, brightness) speaker-identity cues show stronger separation. . . . deare compatible. termined to be from different speakers.”
Per-Attribute Analysis of Speaker-Tokenizer Warm-Up
Table 23 expands the tokenizer warm-up ablation summarized in Table 6(a) of the main paper. It compares SpeakerLLM-Base against a variant that skips the initial speaker-tokenizer-only warm-up and trains the speaker tokenizer together with LoRA-adapted language-model parameters from the start. Table 23: Per-attribute breakdown of the speaker-tokenizer warm-up ablation. SpeakerLLM-Base includes the initial speaker-tokenizer-only warm-up, whereas the ablated variant trains the speaker tokenizer and LoRA-adapted language model jointly from the start. All values are accuracy (% ↑). Speaker Profile Setting
Environment
SV
Gender
Age
Region Pitch Bright. Noise Reverb
w/o warm-up 91.20 SpeakerLLM-Base 96.05
98.98 99.89
34.34 39.83
78.40 83.14
65.60 72.44
43.96 54.19
38.05 52.70
42.86 51.72
Speaker-tokenizer warm-up improves both speaker verification and attribute understanding. SV accuracy improves from 91.20 to 96.05, while non-trivial speaker-profile attributes also improve: age (34.34 → 39.83), region (78.40 → 83.14), pitch (65.60 → 72.44), and timbral brightness (43.96 → 54.19). The largest gains appear for fine-grained acoustic or recording-condition attributes: noise (38.05 → 52.70) and reverberation (42.86 → 51.72). These patterns indicate that warm-up is most beneficial for cues that require stable alignment between audio-derived speaker representations and the language-model input space. Gender changes only slightly (98.98 → 99.89) because it is already near saturation. In contrast, pitch, timbral brightness, noise, and reverberation are more sensitive to whether the speaker-token interface is stabilized before LoRA adaptation. Thus, the warm-up phase functions as an alignment stage, rather than a minor optimization detail. D.3
Representative Speaker Verification Reasoning Cases
Table 24 provides representative generated SV-R traces for two difficult trial types. The metadata and profile-support states are not provided to the model at inference time; they are shown only as offline evidence for comparing the generated SV-R output with the target evidence schema. In the profile-deceptive negative case, the offline profile evidence is highly supportive: gender, age, region, pitch, and timbral brightness are all compatible, yet the ground-truth label is different. Standard SV predicts “same”, whereas SV-R first exposes the similar profile-level evidence and then predicts “different” based on stronger identity-level separation. In the degradation-aware positive case, the two utterances are from the same speaker, but one recording contains moderate noise and the voice characteristics partially differ. Standard SV predicts “different”, whereas SV-R describes the degradation and then preserves the same-speaker decision based on identity-level speaker evidence.
24