PairAlign: A Framework for Sequence Tokenization via SelfAlignment with Applications to Audio Tokenization Adhiraj Banerjee
Department of Electrical Engineering Indian Institute of Technology, Kanpur
Vipul Arora
arXiv:2605.06582v1 [cs.LG] 7 May 2026
Department of Electrical Engineering Indian Institute of Technology, Kanpur
Abstract Modern learning systems map raw sensory signals into continuous vectors, yet many operations—comparison, memory, retrieval, and reasoning—are naturally expressed over discrete symbolic structures. In language, this interface is given a priori through tokens; in perceptual domains such as audio, it must be learned. Recent audio tokenizers often rely on geometric vector quantization, semantic clustering, or codec-style reconstruction, assigning tokens locally at a frame or short-window level. Consequently, sequence-level properties such as cross-realization consistency, compactness, learned length control, termination, and edit-based similarity are rarely optimized directly. We introduce PairAlign1 , a framework for learning compact audio token sequences through sequence-level self-alignment. PairAlign treats tokenization as conditional sequence generation: an encoder maps speech to a continuous conditioning representation, and an autoregressive decoder generates the complete token sequence from BOS, learning token identities, ordering, length, and EOS placement. Given two content-preserving views of a segment, each view’s token sequence is trained to receive high likelihood under the other’s representation, while unrelated in-batch examples provide competing symbolic sequences. This gives a scalable surrogate for edit-distance preservation while discouraging many-to-one collapse. PairAlign combines a staged transition from VQ-style geometric tokenization to adaptive EMA-teacher sequence tokenization with cross-paired teacher forcing, prefix corruption, encoder-summary conditioning, structured self-attention dropout, hardest-K likelihood contrast, repetition-aware target generation, length-constrained decoding, and post-hoc timing recovery from cross-attention. On continuous 3-second speech segments, PairAlign learns compact, non-degenerate token sequences with broad vocabulary usage and strong cross-view consistency on LibriSpeech and TIMIT. On TIMIT retrieval, it preserves meaningful edit-distance search while reducing archive token count by approximately 55%. A continuous-sweep probe shows lower local token-overlap similarity than a dense geometric tokenizer, but stronger absolute length control and more bounded edit-operation trajectories under 100 ms shifts. These results reveal a compactness–locality trade-off: PairAlign gives up dense frame-level redundancy, but obtains a lower-rate symbolic interface for comparison, retrieval, and structural analysis. More broadly, PairAlign is an audio instantiation of sequence-symbolic predictive learning: like JEPA-style objectives, it predicts an abstract target associated with another view rather than reconstructing raw input, but the target is a learned variable-length symbolic sequence rather than a continuous latent. This points toward self-supervised symbolic-interface induction for continuous inputs requiring compact, stable, discriminative, and adaptively sized token sequences. 1 Preprint, Under Review
1
1
Introduction
Discrete tokens as an interface for perception and reasoning. Discrete token sequences provide a natural interface for comparison, retrieval, alignment, memory, editing, generation, and structured reasoning. In language, this interface is given a priori: models operate over text tokens, and sequence-level operations are directly defined in token space. In perceptual domains such as audio, however, the symbolic interface must be learned from continuous signals. This has made audio tokenization a central component of modern speech, music, general-audio, and audio–language systems (Mousavi et al., 2025). Discrete audio tokens now support neural audio codecs, speech and music generation, text-conditioned audio synthesis, multimodal modeling, and retrieval-oriented speech processing. Despite this progress, most audio tokenizers still induce symbols primarily through local assignment. Codecstyle systems such as SoundStream, EnCodec, and DAC learn compact acoustic codes through reconstructionoriented objectives and residual vector quantization (Zeghidour et al., 2021; Défossez et al., 2022; Kumar et al., 2023). Semantic tokenizers such as vq-wav2vec, HuBERT, and w2v-BERT derive discrete units from self-supervised speech representations, usually through clustering or vector quantization (Baevski et al., 2019; Hsu et al., 2021; Chung et al., 2021). Hierarchical and factorized tokenizers organize semantic, acoustic, speaker, prosodic, or reconstruction information across code streams (Zhang et al., 2023a; Ju et al., 2024). These approaches have been highly effective. However, in most of them, the global symbolic behavior of the token sequence is not the main object of learning. Sequence length, ordering, re-tokenization stability, editdistance geometry, and cross-realization consistency are typically inherited from encoder stride, quantizer geometry, reconstruction losses, clustering targets, or downstream post-processing. Why frame-local token assignment does not optimize symbolic sequence geometry. Geometric discretization is a powerful inductive bias. By assigning each encoder frame to a nearby centroid or quantizer entry, the tokenizer obtains a stable local rule for producing symbols. This makes training tractable, preserves timing information, and works well for compression, reconstruction, and frame-synchronous acoustic modeling. However, this stability comes from a restricted hypothesis space. The tokenizer is encouraged to make locally meaningful assignments, but the resulting string is not usually optimized as a sequence. This distinction matters when tokens are used as symbolic strings for retrieval, matching, indexing, alignment, or edit-based comparison. A useful symbolic speech representation should map different acoustic realizations of the same underlying content to compatible token sequences, while keeping unrelated content separated. The desired structure is therefore relational and sequence-level. For a related pair (x, x+ ) and an unrelated example x− , the tokenizations of x and x+ should be closer than either is to the tokenization of x− . Edit distance is a natural geometry for this setting because it accounts jointly for token identity, order, substitutions, insertions, deletions, and length. This edit-distance view motivated retrieval-oriented symbolic speech tokenization in wav2tok. wav2tok was designed to model the constraint that acoustically different realizations of similar spoken content should map to similar token sequences. Because edit distance is combinatorial and non-differentiable, wav2tok used a CTC-style sequence likelihood as a tractable surrogate for edit-distance preservation between paired views (Banerjee & Arora, 2022). BEST-STD is also retrieval-oriented and evaluates discovered speech tokens for spoken term discovery, but it does not use edit-distance preservation as the central token-induction principle (Singh et al., 2025a). PairAlign inherits the same relational goal as wav2tok—paired realizations should induce compatible symbolic strings while unrelated examples remain separated—but changes the probabilistic model used to impose it. Instead of using frame-synchronous CTC-style alignment over token posteriors, PairAlign uses cross-paired conditional sequence likelihood as a differentiable surrogate for editgeometry preservation. From sequence transduction to learned sequence tokenization. A related way to introduce sequence-level structure is through neural sequence transduction objectives such as CTC and RNN-T (Graves et al., 2006; Graves, 2012). These objectives are not geometric tokenizers: they do not require token identities to arise from nearest-centroid assignment. Rather, they define likelihoods over target sequences while marginalizing over unknown monotonic timing. CTC uses frame-indexed posteriors and a collapse map, mak2
ing its scoring structure frame-synchronous; RNN-T adds dependence on previously emitted labels through a prediction network, but still sums over monotonic input–output paths. wav2tok brought this transduction view into retrieval-oriented tokenization by using a CTC-style pairwise constraint to approximate editdistance preservation between paired speech-token sequences (Banerjee & Arora, 2022). PairAlign inherits this retrieval-oriented goal of making paired realizations agree as symbolic strings, but changes the role of transduction. Instead of scoring pairwise consistency through frame-synchronous posterior alignment, PairAlign uses conditional sequence generation to induce the token sequence itself. The output alphabet usage, token ordering, sequence length, and termination are part of the learned symbolic interface. Thus, the objective shifts from aligning frame-indexed token posteriors toward learning the symbolic string space in which future comparison and retrieval can occur. PairAlign: sequence tokenization through conditional generation. We introduce PairAlign, a framework for learning compact discrete audio token sequences through explicit sequence-level self-alignment. Given an input speech segment x, an encoder produces a continuous conditioning representation Z = Enc(x),
(1)
and an autoregressive decoder defines a conditional distribution over token sequences: p(T | Z) =
|T | Y
p(τl | τ<l , Z).
(2)
l=1
At inference time, decoding begins from BOS and terminates by emitting EOS. This formulation makes token identity, token order, output sequence length, and EOS placement explicit learned decisions rather than consequences of encoder frame rate, nearest-centroid assignment, or de-duplication. The central learning signal is pairwise self-alignment. Given two content-preserving realizations (x, x+ ), with encoder representations Z and Z + , the token sequence induced from one view should receive high conditional likelihood under the representation of the paired view: log p(T + | Z)
and
log p(T | Z + ).
(3)
Unrelated in-batch examples provide competing token sequences that should score lower under the same conditioning representation. Thus, PairAlign does not optimize edit distance directly. Instead, it uses pairwise conditional likelihood as a scalable surrogate for the same relational goal: related realizations should induce mutually predictable symbolic sequences, while unrelated inputs should remain separated. A staged path from geometric discretization to adaptive self-alignment. PairAlign is trained through a deliberately conservative three-stage pipeline. Stage I learns a contextual speech encoder and a nearest-centroid VQ tokenizer using self-supervised contrastive learning and a commitment objective. This stage is a valid geometric tokenizer in its own right and serves as a strong controlled baseline. Stage II freezes the Stage I encoder and vector quantizer, and trains an autoregressive decoder on deterministic token targets produced by the frozen geometric tokenizer. This introduces pairwise conditional likelihood in a stable regime: the target sequences are fixed while the decoder learns to predict one view’s tokenization from the paired view’s conditioning representation. Stage III replaces the deterministic teacher with an EMA teacher over the full encoder–decoder model. The EMA teacher generates adaptive token sequences in free-running mode, while the student is trained to align paired views and separate hard in-batch negatives. This allows the conditioning representation, the induced tokenization, the output length distribution, and the training signal to co-evolve. The progression is important: PairAlign does not discard geometric tokenization; it uses it as a stable starting point, then adds sequence-level self-alignment on top of it. Relation to JEPA-style predictive learning. This formulation also connects PairAlign to predictive representation-learning frameworks such as JEPA-style models. JEPA-style learning emphasizes prediction 3
in representation space rather than reconstruction of raw sensory input: a context representation is trained to predict an abstract target representation associated with another view or future region. PairAlign shares this predictive-abstraction principle, but changes the nature of the predicted object. The target is not a fixed-dimensional continuous latent representation. It is a learned, variable-length discrete token sequence induced from a paired content-preserving view. Thus, PairAlign can be interpreted as a sequence-symbolic analogue of JEPA-style learning, where self-supervision induces not only an abstract representation but also the symbolic sequence interface through which that representation is expressed. Alignment as token induction rather than post-training. The term alignment has a different role in PairAlign than in large language model post-training. In LLMs, alignment usually denotes behavioral adjustment over an already fixed symbolic interface. A base model is first trained over a fixed tokenizer, optionally adapted by supervised fine-tuning, and then aligned using human preferences, AI feedback, verifiable rewards, or offline preference objectives. Representative methods include RLHF, RLAIF, Constitutional AI, DPO, IPO, KTO, and reasoning-oriented RLVR methods (Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a;b; Rafailov et al., 2023; Azar et al., 2024; Ethayarajh et al., 2024; Shao et al., 2024; Guo et al., 2025). These methods reshape the distribution over responses, continuations, or reasoning traces, but the vocabulary and token boundaries are already given. PairAlign addresses a different problem. The symbolic interface is not fixed before alignment begins. Token identity, token ordering, sequence length, and termination are outputs of the tokenizer being learned. The alignment signal also does not come from an external preference source. It comes from the data geometry itself: content-preserving realizations of the same signal should induce compatible symbolic sequences. Thus, alignment in PairAlign is a representation-learning principle. It is used to induce, organize, and stabilize the token space itself, rather than to post-train a policy over an established tokenization. Conditional tokenization is not prompt continuation. PairAlign also differs from standard languagemodeling and text sequence-to-sequence objectives. GPT-style models learn left-to-right continuation (Radford et al., 2018; 2019); BERT predicts masked symbols using bidirectional context (Devlin et al., 2019); XLNet changes the autoregressive factorization through permutation-based prediction (Yang et al., 2019); UniLM uses attention masks to support bidirectional, unidirectional, and sequence-to-sequence language modeling in a shared Transformer (Dong et al., 2019); and denoising encoder–decoder models such as MASS and BART reconstruct clean text from corrupted symbolic input (Song et al., 2019; Lewis et al., 2020). PairAlign is related to these models only in the broad sense that it uses conditional autoregressive likelihoods. Its generation task is different. The encoder input is continuous speech, the output symbols are learned audio tokens rather than fixed text tokens, and the decoder is not producing a translation, summary, response, or reconstruction of corrupted text. At inference time, there is no textual prompt, no semantic prompt, no acoustic token prompt, and no target-side prefix to continue. The decoder begins from BOS and generates the complete symbolic tokenization autoregressively conditioned only on Z: Tb = arg max p(T | Z). T
(4)
The decoder must therefore use the acoustic condition to determine the initial symbolic trajectory, token identities, sequence length, and EOS placement. This makes input grounding central to the tokenizer itself. Decoder bypass as a central failure mode. Autoregressive tokenization provides the expressive machinery needed to model compact symbolic sequences, but it also introduces failure modes that do not arise in the same way for framewise geometric tokenizers. The decoder can collapse to short repetitive outputs, map many unrelated inputs to similar strings, produce unstable lengths, or learn an unconditional token prior that is weakly grounded in the acoustic representation. A particularly important failure mode is decoder bypass. During teacher forcing, the decoder receives the target-side prefix. It can therefore obtain low nexttoken loss by exploiting local token continuation patterns while underusing the conditioning representation Z. This is problematic because inference begins from BOS, where no clean target prefix is available. PairAlign treats these degeneracies as central design constraints. The method uses cross-paired likelihood so that the token sequence from one view must be predicted under the conditioning representation of the paired 4
view. Prefix corruption weakens the teacher-forced target-side shortcut. An encoder-summary bias injects input-dependent information at every decoding position. Structured self-attention dropout reduces reliance on the autoregressive self-attention pathway. Hardest-K in-batch likelihood contrast prevents unrelated teacher-generated sequences from becoming equally likely under the same condition. Entropy regularization, repetition-aware target generation, explicit length constraints, and differential encoder–decoder learning rates further stabilize joint self-alignment. The contribution is therefore not simply the use of an autoregressive decoder, but a training recipe that makes autoregressive tokenization practical while the symbolic interface itself is evolving.
Compactness, timing, and the sequence-level trade-off. A direct consequence of autoregressive tokenization is that output length is no longer tied to encoder frame rate. This enables substantially shorter symbolic sequences than frame-synchronous tokenizers, which is useful for storage, retrieval, edit-distance comparison, and downstream sequence processing. However, compact autoregressive tokens are not natively time-stamped. This creates a trade-off: PairAlign gains sequence compactness and explicit length control, but loses the immediate frame-to-token correspondence of geometric tokenizers. To bridge this gap, we introduce a post-hoc timing recovery procedure. After decoding, decoder crossattention is treated as a soft token-to-frame association. A lightweight monotonicity bias is applied, and a monotone Viterbi procedure extracts an approximate segmentation of encoder frames into decoded token positions. This procedure is used only at inference time. It does not modify training, does not impose hard monotonicity on the decoder, and is not required for ordinary token-space retrieval or comparison. It simply provides approximate timestamps when downstream applications require temporal grounding.
Evaluation of the learned symbolic interface. The experiments evaluate PairAlign as a learned symbolic interface rather than only as a source of token sequences. The Stage I geometric tokenizer is the main controlled baseline because it uses the same encoder front end and isolates the effect of autoregressive sequence generation, pairwise self-alignment, and EMA-teacher refinement. The first set of experiments measures cross-view consistency, collapse, token-inventory behavior, and editoperation structure. For each segment, an augmented content-preserving view is tokenized independently, and the two token sequences are compared using unigram Jaccard similarity, normalized edit similarity, exact-match rate, and edit-operation decomposition. PairAlign produces substantially shorter sequences while improving or preserving order-sensitive consistency. It also improves exact full-string agreement under perturbation, indicating that the learned sequence is more stable across realizations despite operating at a much lower token rate. The edit-operation decomposition gives a more diagnostic view than aggregate edit similarity alone. PairAlign requires far fewer absolute edit operations to transform an anchor tokenization into its positiveview tokenization. This reduction is visible across substitutions, insertions, and deletions, showing that compactness is not obtained by unstable token birth/death or large segmentation shifts. Among the edit operations, substitutions are the dominant mode of change. This matters because substitutions alter token identity while preserving sequence length. PairAlign therefore tends to preserve the broad positional structure of the sequence while updating individual token identities to reflect nuisance acoustic variation. If insertions and deletions dominated instead, the result would suggest unstable segmentation, with small perturbations repeatedly creating or removing token positions. The substitution-dominant edit profile therefore indicates that PairAlign’s compactness is accompanied by stable sequence allocation: perturbations mostly change token identities at existing positions, rather than repeatedly changing the sequence length through unstable token birth and death. This distinction is important because normalized edit similarity can obscure the difference between a long sequence with many local edits and a compact sequence with fewer absolute symbolic changes. PairAlign moves the tokenizer toward compact ordered consistency: the sequence is shorter, but its cross-view changes are more controlled in absolute edit geometry. Thus, the improvement is not merely a rate reduction; it is a change in the type of symbolic variability induced by the tokenizer. 5
Collapse and inventory diagnostics further rule out a degenerate explanation. The gains are not obtained by mapping many inputs to low-diversity strings or by shrinking the model to a small set of generic symbols. PairAlign maintains broad vocabulary usage and nontrivial position-wise diversity, while exhibiting a characteristic autoregressive pattern: stronger early input-conditioned commitment followed by broader token usage across the compact sequence. These diagnostics are essential because a generated tokenizer can otherwise appear consistent by collapsing to repeated strings, generic prefixes, or a small set of high-probability outputs. The retrieval experiment evaluates whether the compact symbolic space remains useful for edit-distance search. PairAlign preserves meaningful retrieval behavior while reducing the archive token count by approximately 55%. The geometric tokenizer remains stronger at the very top of the ranking, which is expected from its denser frame-derived structure and stronger local token persistence. However, PairAlign preserves full retrieval coverage and typical first relevant ranks at the top of the list while operating in a substantially smaller sequence space. This shows that PairAlign does not merely shorten token strings; it preserves enough ordered symbolic structure for comparison and retrieval under normalized edit distance. Compositionality probing as structural analysis. Beyond consistency and retrieval, this paper introduces a continuous-sweep probe for learned audio token spaces. The goal is not to prove formal compositionality, but to test whether the decoded token sequence evolves in a locally structured way when the acoustic context is changed continuously. This matters because a tokenizer can perform well on global retrieval metrics while still behaving like a holistic segment-level code. If a small shift in the input window causes arbitrary global re-tokenization, then the learned sequence is less useful as a reusable symbolic interface. The continuous-sweep probe shifts a fixed-duration speech window in small increments and compares the tokenizations of adjacent windows. For each adjacent pair, we compute normalized edit similarity, unigram Jaccard similarity, adjacent length change, and the substitution, insertion, and deletion counts in the optimal edit script. This produces a local trajectory of how the symbolic sequence changes under small context motion, separating token-overlap similarity from the absolute number and type of symbolic edits. The results reveal a compactness–locality trade-off. The geometric tokenizer obtains higher adjacent-window token-overlap similarity, as expected from a dense frame-derived symbolic trace. PairAlign has lower normalized overlap, but produces much shorter token sequences, smaller absolute length changes, and far fewer absolute edit operations. This means that the geometric tokenizer preserves a larger fraction of local token identities, while PairAlign changes through a smaller number of symbolic operations at a coarser sequence scale. The edit-operation profile is especially important for interpreting the sweep probe. PairAlign’s adjacentwindow changes are dominated by substitutions, with substantially smaller insertion and deletion counts than the geometric tokenizer. Substitution dominance indicates that neighboring windows often preserve the broad sequence allocation while some token identities are updated in response to the changed acoustic context. By contrast, a profile dominated by insertions and deletions would indicate unstable segmentation, where small context shifts repeatedly create or remove token positions. Thus, PairAlign’s local variation is better described as bounded context-sensitive symbolic relabelling than as uncontrolled token birth/death or wholesale sequence restructuring. This distinction is central to interpreting lower normalized overlap. Normalized overlap metrics ask what fraction of the sequence changes; absolute edit-operation counts ask how many symbolic operations are needed to move from one neighboring tokenization to the next. Because PairAlign emits far fewer tokens, a small number of changes can produce a larger normalized penalty. Thus, lower normalized edit similarity does not imply arbitrary tokenization. The edit-operation decomposition shows the opposite: PairAlign changes through a more bounded absolute edit trajectory at its own coarser symbolic scale. The substitutiondominant profile further suggests that the model often keeps the sequence scaffold stable while updating token identities, which is the expected behavior for a compact autoregressive tokenizer whose tokens carry broader contextual information than frame-local centroids. The probe should nevertheless be interpreted carefully. It does not show that token sequences transform under a small window shift as (a, b, c, d) → (b, c, d, e), nor does it establish phoneme-like units, strict token6
to-time localization, or clean shifted-subsequence behavior. Whole-sequence overlap can also be affected by shared low-entropy starting patterns, especially for autoregressive decoding from BOS. What the sweep experiment does show is more modest but important: PairAlign does not behave as an arbitrary holistic segment code. It retains nontrivial symbolic continuity, controls output length, uses a broad inventory, and changes through bounded substitution-dominant edit patterns under small context motion. This supports a weaker but meaningful form of local structure: adjacent windows remain related through controlled symbolic updates, even though the tokenizer does not preserve dense frame-level token reuse. Summary of empirical findings. Across the experiments, PairAlign occupies a different operating point from the geometric tokenizer. The geometric tokenizer preserves stronger fine-grained local overlap, which is useful for rank-1 retrieval and adjacent-window token reuse. PairAlign instead produces much shorter sequences, learns output length and termination, improves or preserves augmented-view consistency, maintains broad token usage without measured low-diversity collapse, and preserves meaningful edit-distance retrieval behavior with roughly half the archive token count. The edit-operation results sharpen this conclusion. PairAlign does not simply reduce sequence length by discarding structure. Under noisy content-preserving perturbations, it requires fewer substitutions, insertions, and deletions than the geometric systems. Under continuous sweep, it also changes through fewer absolute edit operations and smaller absolute length shifts, even though its normalized token-overlap similarity is lower. Across these analyses, substitutions are the dominant edit operation. This indicates that PairAlign tends to preserve a compact sequence scaffold while adapting token identities to nuisance variation or small context changes. The lower insertion and deletion burden indicates that the tokenizer is not primarily responding through unstable segmentation, excessive token birth/death, or large length fluctuations. Instead, it behaves like a coarser symbolic sequence model whose local changes are mostly controlled relabellings within a learned ordered interface. This trade-off is central to the paper. PairAlign is not intended to dominate dense frame-synchronous tokenization on every local-overlap or rank-1 retrieval metric. Instead, it provides a lower-rate sequencelevel symbolic interface whose structure is learned directly through self-alignment. The empirical evidence supports that interpretation: the learned sequences are stable under acoustic perturbation, useful for editdistance retrieval, locally structured under controlled context motion, and substantially more compact than frame-derived token strings, while avoiding the collapse modes that would make such compactness trivial. Summary of contributions.
The paper makes the following contributions.
First, it formulates audio tokenization as conditional sequence generation rather than only frame-level geometric assignment. This makes token identity, ordering, sequence length, and termination explicit objects of learning. Second, it introduces pairwise self-alignment as a scalable surrogate for edit-distance preservation. Related acoustic realizations are encouraged to induce mutually predictable token sequences, while unrelated in-batch examples are separated through likelihood contrast. Third, it connects audio tokenization to JEPA-style predictive learning by replacing continuous latent prediction with symbolic sequence prediction. PairAlign predicts an abstract target associated with another view, but the target is an induced token string with explicit order, length, termination, and edit-distance geometry. Fourth, it clarifies a different role for alignment than in LLM post-training. PairAlign does not align behavior over a fixed tokenizer using external preferences; it uses alignment to construct and stabilize the symbolic interface itself. Fifth, it provides a controlled training path from geometric tokenization to adaptive sequence-level symbolic learning. The method begins from a VQ-style tokenizer, trains a deterministic-teacher autoregressive decoder, and then refines the full encoder–decoder model using EMA-teacher self-alignment. Sixth, it identifies and addresses tokenization-specific degeneracy modes, including repetition, many-to-one collapse, unstable sequence length, and decoder bypass. The proposed stabilization recipe combines prefix 7
corruption, encoder-summary conditioning, structured self-attention dropout, hardest-K likelihood contrast, entropy regularization, repetition-aware teacher target generation, explicit length control, and differential learning rates. Seventh, it provides compact symbolic sequences together with optional post-hoc temporal grounding through decoder cross-attention and monotone decoding. Eighth, it broadens the evaluation of learned token spaces beyond consistency and retrieval by introducing continuous-sweep compositionality probing. This analysis tests whether token sequences evolve coherently under small controlled changes in acoustic context, and reveals the compactness–locality trade-off between PairAlign and frame-derived geometric tokenization. PairAlign as a general self-alignment framework. Although this paper studies continuous-speech audio tokenization, the underlying framework is broader. PairAlign applies whenever continuous or weakly structured inputs must be mapped into compact symbolic sequences whose structure should remain stable under nuisance variation while preserving discriminative content. In the present setting, the framework yields a compact audio token sequence that improves cross-realization consistency, supports edit-distance retrieval, learns adaptive output length, and exhibits locally structured behavior under continuous context motion. More generally, the results suggest that self-alignment can be used not merely to evaluate learned symbolic representations, but to help induce and organize them. Viewed through the lens of JEPA-style predictive learning, PairAlign replaces continuous latent prediction with symbolic sequence prediction, suggesting a route toward predictive systems that learn compact discrete event interfaces across views, time, tasks, or modalities.
2
Related Work
2.1
Audio Tokenization
Discrete audio tokenization has become a central abstraction for compression, speech modeling, music generation, general audio generation, and audio–language systems. This design space now spans neural audio codecs, semantic speech units, hierarchical semantic–acoustic tokenizers, disentangled speech tokenizers, codec tokenizers for language modeling, and recent low-frame-rate or scalar-quantized tokenizers, with systems differing in encoder–decoder design, quantization mechanism, reconstruction objective, semantic supervision, streamability, bitrate, and downstream use (Mousavi et al., 2025). This broader view is important because “audio tokenization” now refers not only to neural audio codecs, but also to semantic speech units, hierarchical semantic–acoustic tokenizers, disentangled speech tokenizers, codec tokenizers for language modeling, and recent low-frame-rate or scalar-quantized tokenizers. Quantization mechanisms. Most audio tokenizers first map the waveform to a sequence of continuous encoder representations and then discretize those representations. The simplest form is vector quantization (VQ), popularized in neural generative modeling by VQ-VAE (Van Den Oord et al., 2017), where each latent vector is replaced by the nearest entry in a learned codebook. Residual vector quantization (RVQ) extends this idea by applying multiple codebooks sequentially, with later codebooks quantizing the residual error left by earlier stages; this mechanism underlies many neural audio codecs, including SoundStream (Zeghidour et al., 2021), EnCodec (Défossez et al., 2022), and DAC (Kumar et al., 2023). Product quantization (PQ) (Jegou et al., 2010) and grouped vector quantization (GVQ) split the latent space into subspaces or channel groups and quantize them separately, improving codebook efficiency and scalability. Grouped residual vector quantization, used for example in HiFi-Codec (Yang et al., 2023), applies this factorization within an RVQ-style codec to reduce redundancy across quantizers. More recent alternatives such as finite scalar quantization (FSQ) (Mentzer et al., 2023) and lookup-free quantization (LFQ) (Yu et al., 2023) replace or simplify learned vector-codebook lookup by quantizing bounded scalar dimensions or binary-like latent factors. These mechanisms differ in codebook structure, bitrate control, stability, and token efficiency, but they usually share a frame-level assignment principle: each local encoder frame is mapped to one or more discrete indices. 8
Codec-style acoustic tokenizers. A major family of audio tokenizers is derived from neural audio codecs. SoundStream (Zeghidour et al., 2021) introduced an end-to-end neural audio codec based on RVQ, enabling variable-rate audio compression through multiple quantizer stages. EnCodec (Défossez et al., 2022) further popularized RVQ-based neural audio coding with high-quality reconstruction and streamable operation. Descript Audio Codec (DAC) (Kumar et al., 2023) improved RVQGAN-style codec training through stronger adversarial and reconstruction losses, periodic inductive biases, and improved codebook usage. Other codec tokenizers, including HiFi-Codec (Yang et al., 2023), AudioDec (Wu et al., 2023), FunCodec (Du et al., 2024), StableCodec (Zhang et al., 2025), SNAC (Siuzdak et al., 2024), BigCodec (Xin et al., 2024), and Mimi (Défossez et al., 2024), explore different trade-offs among bitrate, latency, reconstruction quality, codebook structure, and compatibility with language modeling. Although these systems differ substantially in architecture and training objective, their discrete units are still typically obtained by quantizing framesynchronous acoustic latents. Semantic speech tokenizers. A second family focuses less on waveform reconstruction and more on linguistic or semantic abstraction. vq-wav2vec (Baevski et al., 2019) learns discrete speech representations through vector quantization inside a self-supervised framework. HuBERT (Hsu et al., 2021) obtains discrete speech targets by clustering self-supervised speech features and then predicts masked cluster assignments. w2v-BERT (Chung et al., 2021) combines contrastive and masked-prediction objectives and has been used as a source of semantic tokens in downstream audio-language systems. These tokenizers are influential because their units are more invariant to speaker, channel, and low-level acoustic variation than purely reconstructive codec tokens. However, the units are still usually induced from frame-level representations through clustering, VQ, or related discrete assignment procedures; sequence-level properties such as compactness, re-encoding stability, and edit-distance consistency are not usually optimized directly. Hierarchical semantic–acoustic tokenizers. Several tokenizers attempt to bridge semantic speech units and reconstruction-oriented acoustic codes. SpeechTokenizer (Zhang et al., 2023a) uses an encoder–decoder architecture with RVQ, where the first quantizer layer is guided by semantic teacher representations and later layers preserve acoustic and paralinguistic details. This creates a hierarchy in which early codes are more content-oriented while later codes carry speaker, prosody, and reconstruction information. WavTokenizer (Ji et al., 2024) targets audio language modeling by reducing the codec token rate while retaining reconstruction quality and semantic utility. SemantiCodec (Liu et al., 2024) similarly reflects the broader effort to construct tokenizers that are useful not only for waveform reconstruction, but also for language-model-based audio generation and understanding. These methods move beyond a purely compression-centric view of codec design, but their discrete interfaces are still generally produced through local quantization of encoder features. Disentangled and factorized tokenizers. Another line of work explicitly factorizes the audio representation into separate attributes such as linguistic content, speaker identity, prosody, timbre, or acoustic detail. FACodec (Ju et al., 2024), for example, separates speech factors to support controllable speech generation and voice conversion. SpeechTokenizer (Zhang et al., 2023a) also reflects this trend through its hierarchical semantic–acoustic RVQ design. Such factorized tokenizers are useful because different code streams can be assigned different modeling roles: one stream may represent linguistic content, while others encode speaker, style, or acoustic realization. However, factorization primarily changes what information different codes carry. It does not by itself make the global token sequence an explicit object of alignment. Tokenizers used by audio language models. Discrete tokenizers have become the interface through which audio is made compatible with language-modeling machinery. AudioLM (Borsos et al., 2023) combines semantic tokens from w2v-BERT (Chung et al., 2021) with acoustic tokens from SoundStream (Zeghidour et al., 2021). MusicLM (Agostinelli et al., 2023) uses semantic conditioning together with SoundStream (Zeghidour et al., 2021) acoustic tokens for text-to-music generation. AudioGen (Kreuk et al., 2022) and MusicGen (Copet et al., 2023) model discrete EnCodec (Défossez et al., 2022) codes for text-conditioned audio and music generation. VALL-E (Wang et al., 2023a) formulates text-to-speech as conditional language modeling over neural codec codes, using EnCodec-style acoustic tokens. AudioPaLM (Rubenstein et al., 2023), Spear-TTS (Kharitonov et al., 2023), UniAudio (Yang et al., 2024), VioLA (Wang et al., 2023b), and LauraGPT (Du et al., 2023) further illustrate how semantic and acoustic token streams can serve as 9
a bridge between audio and autoregressive sequence modeling. In these systems, the tokenizer is usually trained before the language model and then treated as a fixed symbolic interface. Alternative and low-rate tokenizers. Recent work has also reconsidered the quantization bottleneck itself. Rather than simply adding more RVQ stages, several designs aim to reduce token rate, improve codebook utilization, or make token streams easier for language models to consume. FSQ-based tokenizers (Mentzer et al., 2023) replace learned vector-codebook lookup with scalar quantization over bounded latent dimensions, reducing some training instabilities associated with codebook learning. LFQ-style designs (Yu et al., 2023) similarly simplify discrete latent assignment by avoiding conventional learned lookup tables. WavTokenizer (Ji et al., 2024) reduces the number of acoustic tokens needed per second for audio language modeling, while BigCodec (Xin et al., 2024) and related low-frame-rate codec tokenizers seek more compact symbolic streams. These developments show that token efficiency and quantizer structure are now central design questions, especially when audio tokens are used by large sequence models. Structural limitation of frame-level token induction. Despite this diversity, most existing audio tokenizers share a common structural property: token identity is induced primarily through frame-level or short-window-level assignment. In reconstruction codecs such as SoundStream (Zeghidour et al., 2021), EnCodec (Défossez et al., 2022), DAC (Kumar et al., 2023), HiFi-Codec (Yang et al., 2023), and WavTokenizer (Ji et al., 2024), the assignment is driven by acoustic latent geometry, bitrate control, and reconstruction fidelity. In semantic tokenizers such as vq-wav2vec (Baevski et al., 2019), HuBERT (Hsu et al., 2021), and w2v-BERT-derived token streams (Chung et al., 2021), the assignment is driven by self-supervised feature geometry or clustered pseudo-labels. In hierarchical and factorized systems such as SpeechTokenizer (Zhang et al., 2023a) and FACodec (Ju et al., 2024), the code streams are shaped to carry different kinds of information. However, in all of these cases, the global symbolic behavior of the token sequence is usually not the primary training target. A particularly important consequence is that output length is usually not a learned symbolic decision. In frame-synchronous or codec-style tokenizers, the number of emitted tokens is largely determined by input duration, encoder stride, quantizer rate, or the number of code streams. Even when consecutivetoken deduplication or low-frame-rate codec design reduces the token count, length is typically controlled by architectural rate choices or post-processing rather than by a decoder deciding how many symbols are needed to represent the input. Thus, compactness is often imposed through frame rate, bitrate, or codebook design, rather than learned as part of the symbolic sequence itself. This distinction matters when tokens are used not only for reconstruction or generation, but also as symbolic strings for retrieval, matching, indexing, or alignment. A tokenizer may reconstruct audio well while still producing long, redundant, or unstable token sequences. Similarly, two acoustically perturbed versions of the same content may map to token strings whose edit distance is larger than desirable, even if their waveforms or continuous latent representations remain close. Sequence compactness, ordering, re-encoding stability, cross-view consistency, and edit-based similarity are therefore often inherited indirectly from encoder stride, quantizer geometry, codebook usage, semantic supervision, or downstream post-processing, rather than being learned directly in token space. Relation to PairAlign. PairAlign instead makes length an explicit part of the learned symbolic interface. Because the decoder generates from BOS and terminates with EOS, the model learns not only token identity and order, but also how many symbols to emit for a given input segment. This distinguishes PairAlign from fixed-rate or frame-synchronous tokenizers, where length is mainly inherited from the input timeline or quantizer design. This makes PairAlign complementary to existing tokenizers. Codec tokenizers such as SoundStream (Zeghidour et al., 2021), EnCodec (Défossez et al., 2022), DAC (Kumar et al., 2023), and WavTokenizer (Ji et al., 2024) optimize a discrete interface primarily for reconstruction, compression, and efficient downstream modeling. Semantic tokenizers such as vq-wav2vec (Baevski et al., 2019), HuBERT (Hsu et al., 2021), and w2v-BERT-based token streams (Chung et al., 2021) improve invariance and linguistic abstraction. Hierarchical and disentangled tokenizers such as SpeechTokenizer (Zhang et al., 2023a) and FACodec (Ju et al., 10
2024) improve the organization of information across code streams. PairAlign instead focuses on the symbolic behavior of the resulting sequence: whether token length, token ordering, and cross-realization consistency can be shaped directly by conditional sequence likelihood. Summary. PairAlign should therefore be understood as moving beyond purely frame-synchronous token induction. Stage I preserves the stability of geometric tokenization, while later stages add an explicit sequence-level learning principle on top of it. The resulting tokenizer is intended not only to assign local acoustic or semantic codes, but also to produce compact and comparable token strings whose sequence structure is aligned across related audio realizations. 2.2
Alignment in Large Language Models
Alignment as post-training over a fixed symbolic interface. In large language models (LLMs), alignment typically denotes a post-training stage that reshapes the behavior of a pretrained or instructiontuned policy while leaving the symbolic interface fixed. A standard pipeline first trains a base language model with next-token prediction, optionally adapts it with supervised fine-tuning (SFT), and then applies an additional alignment procedure to increase the likelihood of preferred responses while constraining the policy from drifting too far from a reference model. Representative examples include reinforcement learning from human feedback (RLHF), as used in InstructGPT (Ouyang et al., 2022) and earlier summarization work (Stiennon et al., 2020); Anthropic HH-RLHF (Bai et al., 2022a); Constitutional AI / RLAIF (Bai et al., 2022b); Direct Preference Optimization (DPO) (Rafailov et al., 2023); Kahneman–Tversky Optimization (KTO) (Ethayarajh et al., 2024); Identity Preference Optimization (IPO) (Azar et al., 2024); and reasoningoriented RLVR methods such as GRPO / DeepSeekMath (Shao et al., 2024) and DeepSeek-R1 (Guo et al., 2025). In these settings, the model already consumes and emits tokens from a fixed vocabulary. Alignment changes the distribution over responses, but it does not learn the tokenizer itself. Preference-based and RL-based alignment. A central theme in LLM alignment is the use of comparative or evaluative supervision to modify generation behavior. RLHF methods such as InstructGPT (Ouyang et al., 2022), learning to summarize from human feedback (Stiennon et al., 2020), and Anthropic HHRLHF (Bai et al., 2022a) train a reward or preference model from pairwise human judgments and then optimize the policy with a KL-regularized RL objective, commonly using PPO (Schulman et al., 2017). RLAIF and Constitutional AI replace part of this human-feedback pipeline with model-generated feedback, written principles, or AI-judge evaluations, as in Constitutional AI (Bai et al., 2022b), RLAIF (Lee et al., 2023), and rule-based reward modeling for safety alignment (Mu et al., 2024). These methods differ in the source of feedback, but they share the same structure: a pretrained policy is adjusted using an external preference or reward signal while remaining anchored to an existing reference model. Offline preference optimization and listwise alignment. A second line of work removes the explicit online RL loop and directly optimizes the policy on preference data. DPO (Rafailov et al., 2023) rewrites KL-regularized reward optimization as a classification-style objective over preferred and rejected responses. SLiC-HF (Zhao et al., 2023) uses sequence-likelihood calibration with a ranking loss; IPO (Azar et al., 2024) provides an alternative preference-learning formulation that avoids some deterministic-preference pathologies; and KTO (Ethayarajh et al., 2024) extends alignment to unpaired desirable and undesirable examples. Other reference-free or simplified objectives include SimPO (Meng et al., 2024) and ORPO (Hong et al., 2024). Listwise methods such as RRHF (Yuan et al., 2023), PRO (Song et al., 2024), and LiPO (Liu et al., 2025a) further generalize preference learning from pairwise comparisons to ranked candidate sets. Although these methods differ in objective design, they still optimize behavior over a fixed text-token sequence space. Reasoning-oriented post-training with verifiable rewards. Recent reasoning-oriented alignment methods extend the same post-training view to settings where rewards can be verified automatically. GRPO / DeepSeekMath (Shao et al., 2024) replaces a learned critic with group-relative reward normalization, making RLVR more scalable for mathematical reasoning. DAPO (Yu et al., 2025), Dr. GRPO (Liu et al., 2025b), and related variants modify importance weighting, advantage normalization, length normalization, and regularization to stabilize long-chain-of-thought training. DeepSeek-R1 (Guo et al., 2025) further shows that 11
large-scale RL with verifiable rewards can elicit reasoning behaviors such as self-verification and reflection. Even in these cases, however, the symbolic interface remains fixed: the model is not learning what its tokens are, but how to generate better sequences over an existing vocabulary. Shared structural assumptions. Despite substantial algorithmic variation, LLM alignment methods share two structural assumptions. First, the symbolic interface is already established: the model receives and emits sequences over a fixed tokenizer, and alignment changes the distribution over those sequences rather than the tokenizer itself. Second, the aligned policy is usually constrained relative to a reference policy, either explicitly through KL regularization, as in PPO-based RLHF (Ouyang et al., 2022; Schulman et al., 2017), or implicitly through likelihood-ratio objectives, as in DPO (Rafailov et al., 2023), IPO (Azar et al., 2024), and related preference-optimization methods. Thus, LLM alignment is primarily a problem of preference shaping over an already meaningful symbolic sequence space. Relation to PairAlign. PairAlign is conceptually related to this literature because it also uses comparative structure to shape model behavior. However, the object being aligned is fundamentally different. In RLHF, RLAIF, DPO, KTO, IPO, GRPO, and related LLM post-training methods, the vocabulary and token boundaries are fixed before alignment begins. Alignment then adjusts which responses, continuations, or reasoning traces are preferred under that existing interface. In PairAlign, by contrast, the symbolic interface is itself learned: token identity, sequence length, and token ordering are evolving outputs of the tokenizer. The goal is therefore not to align a pretrained generator over a stable vocabulary, but to induce a tokenization mechanism whose outputs remain consistent, discriminative, compact, and non-degenerate across multiple realizations of the same underlying audio content. Conditional tokenization is not prompt continuation. This distinction is especially important because the autoregressive decoder in PairAlign is not used as an open-ended continuation model. In standard LLM generation, the decoder first processes an input prompt, instruction, or dialogue context, and the target response is generated as a continuation conditioned on that prompt. Even in many conditional generation settings, a textual prompt, semantic prompt, acoustic prompt, or partial target-side prefix provides an explicit symbolic context that the decoder can process before producing the output. PairAlign does not have such a decoder-side prompt. At inference time, tokenization begins from BOS and generates a complete symbolic sequence conditioned on the encoder representation Z: Tb = arg max p(T | Z). T
(5)
There is no externally supplied text prompt, no partial audio-token sequence to continue, and no targetside symbolic context whose continuation is the desired output. The only source of input specificity is the conditioning representation exposed through cross-attention and the encoder-summary pathway. The decoder must therefore use Z to establish the initial symbolic trajectory, choose token identities, determine sequence length, and emit EOS. Thus, although the decoder is causal, the task is conditional tokenization from BOS, not prompt continuation. This matters for how alignment should be interpreted. In standard LLM post-training, the model is usually shaped to produce better responses or continuations under a fixed prompt distribution. In PairAlign, the decoder is part of the tokenizer itself. Its free-running output defines the symbolic representation assigned to an input signal. Consequently, alignment must ensure that the generated sequence is grounded in the conditioning representation, rather than merely plausible under the decoder’s learned token-sequence prior. Alignment as representation learning rather than post-hoc policy shaping. This distinction changes the methodological role of alignment. In standard LLM post-training, alignment is typically applied after pretraining and SFT, using human preferences, AI feedback, verifiable rewards, or offline preference pairs. In PairAlign, alignment is integrated into representation learning itself. Stage I first establishes a geometric tokenizer; Stage II introduces a causal autoregressive decoder trained from deterministic token targets; and Stage III jointly refines the encoder and decoder using an EMA teacher and pairwise selfalignment. Thus, alignment in PairAlign is not a post-hoc preference adjustment over a fixed symbolic system. It is part of the mechanism by which the symbolic system is induced, organized, and stabilized. 12
Tokenization-specific failure modes. The relevant failure modes are therefore different from those in conventional LLM alignment. LLM alignment can suffer from reward hacking, verbosity bias, over-refusal, preference overfitting, and loss of response diversity, because the optimizer reshapes a policy over existing text tokens. PairAlign instead faces tokenization-specific degeneracies: repetition, many-to-one collapse, unstable token ordering, length instability, and decoder bypass. Decoder bypass arises because teacher-forced training gives the decoder access to the target-side prefix. A model can therefore obtain high teacher-forced likelihood by learning local token continuation statistics while making weak use of the acoustic condition Z. This is problematic because inference begins from BOS, where no informative target prefix or prompt is available. If the conditioning pathway has not been learned strongly, the decoder may emit generic high-probability prefixes and then continue them autoregressively, producing input-insensitive tokenizations. Thus, a model can appear successful under teacher-forced likelihood while failing as a conditional tokenizer. PairAlign addresses this failure mode through mechanisms that make conditional grounding part of the training design. Cross-paired likelihood requires the token sequence from one view to be predictable from the conditioning representation of the paired view. Prefix corruption reduces the reliability of the teacherforced target prefix. The encoder-summary bias injects an input-dependent signal into every decoding position. Structured self-attention dropout weakens the decoder’s token-side continuation shortcut. Finally, hardest-K in-batch likelihood contrast prevents unrelated examples from becoming equally likely under the same conditioning representation. Together, these mechanisms are tailored to learned tokenization rather than post-hoc policy alignment. Summary. PairAlign should therefore not be viewed as an instance of standard LLM alignment. The connection is conceptual rather than procedural. RLHF, RLAIF, DPO, KTO, IPO, GRPO, and related methods align behavior over an already established textual interface. PairAlign uses alignment to construct and stabilize the interface itself. Moreover, its autoregressive decoder does not continue an existing token sequence or decode from a symbolic prompt at inference time; it generates a complete symbolic tokenization from BOS conditioned on Z. This places PairAlign closer to representation learning and tokenizer induction than to conventional post-training alignment, even though it borrows the broader idea that comparative structure can provide a powerful organizing signal. 2.3
Language-Modeling and Denoising Sequence-to-Sequence Objectives
Sequence modeling objectives. PairAlign uses an encoder–decoder architecture and an autoregressive decoder, which places it broadly near sequence-to-sequence modeling. However, its role is different from the usual text sequence-to-sequence setting. In text sequence-to-sequence models, the encoder typically reads a symbolic source sequence and the decoder generates a target text sequence, as in translation, summarization, dialogue, or denoising reconstruction. PairAlign instead uses the encoder to produce a continuous acoustic condition Z, and the decoder emits a compact symbolic tokenization of the same underlying input. Thus, the decoder is not generating a linguistic response, summary, translation, or continuation; it is part of the tokenizer itself. Several language-modeling and pre-training objectives are relevant for positioning this design. GPT-style language models train a left-to-right decoder to predict the next token from the previous tokens alone (Radford et al., 2018; 2019). BERT trains a bidirectional encoder with masked language modeling, where selected input tokens are replaced by mask symbols and predicted from context (Devlin et al., 2019). XLNet generalizes autoregressive modeling through permutation-based factorization, allowing prediction under sampled token orders rather than only strict left-to-right order (Yang et al., 2019). UniLM uses different self-attention masks to support bidirectional, unidirectional, and sequence-to-sequence language modeling within a shared Transformer (Dong et al., 2019). MASS masks a contiguous source span and trains a sequence-to-sequence model to generate the missing span (Song et al., 2019). BART further generalizes denoising sequenceto-sequence pre-training by corrupting the encoder input through transformations such as token masking, deletion, text infilling, sentence permutation, and document rotation, and training an autoregressive decoder to reconstruct the original text (Lewis et al., 2020). 13
Relation to PairAlign. PairAlign is closest to these methods only in the broad sense that it uses conditional autoregressive likelihoods over sequences. It differs in the object being learned. A left-to-right language model learns token continuation without an external conditioning representation. This is not sufficient for PairAlign: the decoder must generate an input-specific tokenization from Z, rather than a plausible token string under its own prior. Permuted language modeling is also not a natural fit, because PairAlign’s decoded order is part of the learned symbolic interface and must match inference, where generation proceeds causally from BOS. Masked language modeling and multitask masked language modeling operate over already meaningful text tokens, whereas PairAlign is trying to induce the symbolic interface itself. Masked sequence-to-sequence methods such as MASS and BART corrupt the source-side text and reconstruct missing or original content; PairAlign does not use prefix corruption in this source-denoising sense. The distinction is particularly important for BART. In BART, the encoder receives a corrupted textual input and the decoder is trained to reconstruct the corresponding clean text (Lewis et al., 2020). PairAlign uses a different form of corruption. The acoustic condition supplied to the encoder and used by decoder cross-attention is a content-preserving view of the same speech segment: the underlying lexical content and temporal order are preserved, while nuisance factors such as gain, additive noise, reverberation, and channel filtering may be changed. Corruption is applied only to the decoder-side teacher-forcing prefix, and the clean token sequence remains the prediction target. Thus, prefix corruption in PairAlign is not a BART-style audio denoising objective. It is a regularization element inside the cross-paired teacher-forcing loss, designed to reduce reliance on the target-side prefix and encourage the decoder to use the acoustic condition. Why prefix corruption is used. The cross-paired teacher-forcing objective asks the decoder to assign high likelihood to the token sequence from one view while conditioning on the encoder representation of the paired view. This is the sequence-level alignment signal: paired realizations should remain mutually predictable in token space. However, ordinary teacher forcing gives the decoder access to the clean targetside prefix. This can make the alignment task too easy in an unintended way, because some next-token decisions can be explained by local token continuation patterns rather than by the paired acoustic condition. Prefix corruption makes this cross-paired prediction problem harder. Selected tokens in the shifted teacherforcing input are replaced by a mask symbol, but the model is still trained to predict the clean target sequence. The objective therefore remains causal next-token prediction under cross-conditioning, but the target-side prefix is made less reliable. This encourages the likelihood used in the alignment loss to depend more on the conditioning representation Z than it would under fully clean teacher forcing. This should not be interpreted as a complete solution to decoder bypass. Prefix corruption only weakens one shortcut: reliance on an overly informative teacher-forced prefix. By itself, it does not guarantee that the decoder uses the acoustic condition correctly. For this reason, PairAlign combines prefix corruption with other grounding and separation mechanisms, including cross-attention to Z, the encoder-summary bias, structured self-attention dropout, EMA-teacher self-alignment, and in-batch likelihood contrast. Prefix corruption is therefore one component of the alignment objective, not an independent guarantee of input grounding. Why arbitrary denoising schemes are not adopted. The broader noising space used in text denoising models is not directly transferable to PairAlign. Text infilling, deletion, sentence permutation, and document rotation are meaningful in BART because the corrupted source is still a symbolic text object with a welldefined reconstruction target (Lewis et al., 2020). In PairAlign, however, both sides of the encoder–decoder interface impose stronger ordering constraints. On the encoder side, the temporal order of the acoustic context is part of the content to be tokenized; deleting, rotating, or permuting acoustic frames would alter the conditioning signal and turn cross-realization token alignment into missing-content reconstruction or artificial order recovery. On the decoder side, the generated token order is itself part of the learned symbolic representation; arbitrary reordering or deletion of decoder-side tokens would disrupt the causal generation process and obscure the sequence structure that the tokenizer is meant to learn. For this reason, PairAlign does not adopt general-purpose source-denoising schemes from text pre-training. The encoder condition is kept temporally intact, apart from mild content-preserving waveform augmentations used to construct paired acoustic views. The only decoder-side corruption is prefix corruption during teacher 14
forcing, where selected prefix tokens are replaced by a mask symbol while the clean target sequence and its left-to-right order are preserved. Thus, corruption in PairAlign is used to weaken the teacher-forced prefix shortcut without changing the ordered acoustic condition or the ordered token sequence being learned. Summary. PairAlign can be implemented as an encoder–decoder sequence model, but it is not a text sequence-to-sequence model transplanted to audio. The decoder is not trained to continue a textual prompt, reconstruct a corrupted source sequence, or generate an externally specified target. Instead, it starts from BOS and produces the symbolic tokenization assigned to the acoustic input. Prefix corruption is introduced only as a regularizer for the cross-paired teacher-forcing objective. The acoustic condition provided through cross-attention remains a content-preserving view of the same speech segment, so the underlying content and temporal order are preserved. The clean token sequence remains the prediction target. Thus, prefix corruption does not define a BART-style denoising objective. Its role is to make the alignment loss less solvable from a clean decoder-side prefix alone, thereby encouraging the decoder to use the acoustic condition when inducing aligned token sequences. 2.4
Audio Generation
Discrete audio tokens as a generative interface. Recent progress in audio generation has been strongly shaped by the use of discrete audio tokens as the modeling interface. A growing body of audio–language and codec-language-model work formulates audio generation as sequence modeling, where a neural generator predicts audio tokens under textual, acoustic, or multimodal conditioning (Su et al., 2025; Wu et al., 2024). This abstraction has unified a wide range of tasks, including speech synthesis, speech continuation, text-tomusic generation, text-to-audio generation, audio editing, and language-guided audio manipulation. Codec-language models for speech, music, and general audio. A representative line of work first converts waveforms into discrete token sequences using a neural audio codec or related tokenizer, and then trains a generative model over those sequences. AudioLM (Borsos et al., 2023) combines SoundStream (Zeghidour et al., 2021) acoustic tokens with semantic tokens from w2v-BERT (Chung et al., 2021) to model speech and music continuation as language modeling over audio tokens. MusicLM (Agostinelli et al., 2023) similarly builds on SoundStream (Zeghidour et al., 2021), while using MuLan (Huang et al., 2022) and w2vBERT (Chung et al., 2021) representations to support text-conditioned music generation. AudioGen (Kreuk et al., 2022) uses a SoundStream-style residual-vector- quantized audio tokenizer (Zeghidour et al., 2021) for text-guided general audio generation. VALL-E (Wang et al., 2023a) uses EnCodec (Défossez et al., 2022) tokens for zero-shot text-to-speech synthesis, while VALL-E X (Zhang et al., 2023b) and SpeechX (Wang et al., 2024) further extend codec-language modeling to cross-lingual speech generation and speech transformation. MusicGen (Copet et al., 2023) also uses EnCodec (Défossez et al., 2022) tokens, showing that a comparatively simple autoregressive language model over codec streams can support controllable music generation. More unified or multitask systems, including UniAudio (Yang et al., 2024), VioLA (Wang et al., 2023b), AudioPaLM (Rubenstein et al., 2023), and LauraGPT (Du et al., 2023), further illustrate the role of codec tokens as a shared interface for combining speech, audio, and language generation tasks. Generation objectives in codec-language models. Although these systems are often described under the common language of autoregressive audio modeling, the underlying generation tasks are not all the same. Some settings are genuine continuation problems: the model receives a partial semantic or acoustic token prefix and predicts the remaining audio tokens. Other settings are prompt-conditioned generation problems: the decoder first processes a text prompt, speaker prompt, semantic prompt, acoustic prompt, or multimodal instruction, and then generates the target audio-token sequence under that prompt context. In both cases, the decoder is given an explicit symbolic or learned prompt context before target generation begins. PairAlign differs from both cases. Its decoder is autoregressive, but the task is not to continue an externally provided audio-token sequence, nor to generate an audio sample from a textual or semantic prompt. At inference time, the model starts from BOS and generates the entire compact tokenization conditioned only on Z = Enc(x). The generated sequence is therefore not a continuation of a token prefix or a response to a decoder-side prompt; it is the symbolic representation assigned to the input signal. This makes input 15
grounding a central requirement: the decoder must use the conditioning representation to determine which sequence to generate. Why codec tokens became effective for generation. The codec-based formulation is powerful because it decomposes audio generation into two coupled but conceptually distinct problems. The tokenizer or codec maps long waveforms into compact symbolic sequences, and the generative model learns a distribution over those symbols. This interface is attractive for large-scale modeling because it shortens audio sequences, allows audio and text tokens to be interleaved or jointly modeled, and supports high-fidelity waveform reconstruction through the codec decoder. As a result, neural codec language modeling has become a dominant paradigm for modern speech, music, and general-audio generation. Tokenizer-induced bottlenecks. The success of codec-language models also makes the tokenizer a critical bottleneck. Many widely used codecs, including SoundStream (Zeghidour et al., 2021) and EnCodec (Défossez et al., 2022), produce tokens through frame-level or frame-synchronous quantization of continuous encoder representations, often using residual vector quantization. This design is well suited to reconstruction quality, bitrate control, and streamable compression. However, it also means that the symbolic stream inherited by the downstream generator is primarily shaped by local reconstruction and quantization objectives. Sequence-level properties of the token stream—such as cross-realization consistency, edit-distance stability, controllable symbolic compactness, termination behavior, and robust local ordering—are usually not imposed directly as tokenizer training objectives. Instead, these properties are expected to emerge indirectly from the learned latent geometry, the codec architecture, and the downstream language model. Length control is especially important in this setting. A downstream generator inherits the token rate and sequence length distribution of the tokenizer. If the tokenizer emits long redundant streams, generation becomes more expensive and long-range modeling becomes harder. If it emits overly compressed or unstable streams, generation may lose controllability or content fidelity. Thus, the tokenizer’s length behavior is not merely an efficiency detail; it directly shapes the sequence modeling problem faced by downstream audio generators. Mismatch between tokenizer training and sequence modeling. This creates a structural mismatch between how audio tokens are used and how many tokenizers are trained. In downstream generation, the model operates over token sequences, so token ordering, token length, repetition patterns, and local continuation structure directly affect generation behavior. Yet the tokenizer is often optimized primarily as a frame-level compression and reconstruction mechanism. Thus, while codec tokens provide an effective interface for generation, the sequence-level organization of that interface is usually not learned as a primary object of optimization. The mismatch is particularly important when generation starts from BOS without a decoder-side prompt. At the first few decoding steps, the model has little token-side context, so the conditioning representation must determine the initial symbolic trajectory. If the token space is poorly organized, or if training overrewards prediction from teacher-forced prefixes, the generated sequence can become plausible as a token string but insufficiently specific to the input. For a downstream generator, this may reduce controllability or faithfulness. For PairAlign, the consequence is more direct: the generated sequence is the tokenization itself, so weak grounding immediately degrades the symbolic interface. Relation to PairAlign. PairAlign is motivated by this mismatch between the sequence-level use of tokens in modern audio generation and the predominantly geometric or frame-synchronous way in which many audio tokenizers are induced. Unlike AudioLM (Borsos et al., 2023), MusicLM (Agostinelli et al., 2023), AudioGen (Kreuk et al., 2022), VALL-E (Wang et al., 2023a), and MusicGen (Copet et al., 2023), our goal is not to propose a new downstream audio generator over an existing codec token stream. Instead, PairAlign focuses on the token space itself: it asks whether the symbolic sequence interface can be learned with explicit sequence-level self-alignment, rather than inherited only from frame-level geometric discretization. Concretely, PairAlign begins from a VQ-style geometric tokenizer, but then introduces autoregressive token generation and pairwise self-alignment so that related audio realizations are encouraged to induce mutually predictable token sequences. This shifts the learning signal from local token assignment alone toward sequence-level consistency, compactness, discriminability, and controlled termination. The decoder starts 16
from BOS at inference time and generates the complete token sequence conditioned on the encoder representation, without first processing a text prompt, semantic prompt, acoustic prompt, or target-side token prefix. Accordingly, the training objective is designed not merely to learn plausible token continuations, but to learn an input-conditioned tokenization rule. PairAlign also makes length behavior learnable. Because the decoder emits EOS, symbolic rate and sequence length are outputs of the tokenization model rather than fixed consequences of encoder stride, quantizer rate, or codec frame rate. This opens the possibility of adaptive-rate tokenizers whose sequence length is shaped by self-alignment and input complexity rather than by a predetermined frame rate alone. Complementarity to codec-language models. PairAlign is therefore complementary to codeclanguage-model systems. Those systems demonstrate the value of modeling over discrete audio tokens for speech, music, and general-audio generation. PairAlign studies how the token sequences themselves can be learned, organized, and stabilized when the decoder must generate them from BOS under an acoustic condition rather than from a symbolic prompt or partial target prefix. The broader aim is to provide a more sequence-aware symbolic interface that could support retrieval and comparison, as studied in this paper, and may also benefit future audio generation, editing, and language-guided control. 2.5
Audio Retrieval
Retrieval as robust matching under acoustic variability. Audio retrieval aims to identify semantically or lexically relevant audio content from large archives despite variability in speaker identity, channel, speaking rate, background acoustics, and recording conditions. In speech, a canonical formulation is queryby-example spoken term detection (QbE-STD), where a spoken query is used to retrieve utterances containing the same lexical item without requiring a text query. Early spoken-term detection systems were often ASRcentric: retrieval was performed over phone lattices, grapheme lattices, posteriorgrams, or related symbolic representations produced by automatic speech recognizers (Mamou et al., 2007; Miller et al., 2007; Wang et al., 2008; Abad et al., 2016). Such systems can be effective when accurate ASR is available, but their performance can degrade under acoustic mismatch, short queries, and out-of- vocabulary or unseen words (Saraclar & Sproat, 2004; Can & Saraclar, 2011). Direct sequence matching and its scalability bottleneck. A complementary line of work avoids committing to a recognizer output and treats retrieval as direct acoustic sequence matching. Dynamic time warping (DTW) and related template-matching approaches compare variable-length feature sequences under monotone alignment constraints, making them well suited to spoken queries whose duration and speaking rate may differ from matching archive segments (Ram et al., 2018; 2020; Tsai, 2021). This alignmentaware formulation is attractive because it preserves local temporal structure and does not require explicit word transcription. However, exhaustive DTW-style search is computationally expensive at archive scale, especially when retrieval must be performed repeatedly for many queries. Embedding, fingerprinting, and hashing approaches. To improve scalability, later work shifted toward embedding-based retrieval, in which discriminative acoustic representations are learned so that nearestneighbor search can replace per-query dynamic programming. This includes acoustic word embeddings, weakly supervised segment-level representations, and retrieval pipelines trained to bring matched speech segments closer in embedding space (Chung et al., 2016; He et al., 2016; Kamper et al., 2016; Chen et al., 2018; Hu et al., 2021; Banerjee & Arora, 2023). These methods substantially reduce retrieval cost, but many assume reliable word segmentation, clean candidate segments, or well-defined acoustic units, which are difficult to obtain in continuous speech. Related fingerprinting and hashing methods further improve efficiency by constructing compact indexable signatures, but can remain sensitive to speaker, channel, and acoustic-domain variation (Singh et al., 2022; 2023; 2024). Together, these lines of work reveal a central tension in audio retrieval: representations must preserve alignment-relevant structure, but must also be compact, stable, and efficiently searchable. Discrete tokens as an indexable retrieval substrate. Discrete audio tokens are especially attractive for retrieval because they turn continuous speech into compact symbolic sequences that can be stored, com17
pared, and indexed efficiently. In this setting, tokenization is not merely a preprocessing step for compression or generation; it directly shapes the geometry on which retrieval operates. A retrieval-oriented tokenizer should map acoustically different realizations of related speech to nearby token sequences, while keeping unrelated speech separated. wav2tok (Banerjee & Arora, 2022) is the closest prior work to this paper in spirit because it makes sequencelevel alignment part of token learning. Instead of evaluating edit- or alignment-aware similarity only after tokenization, wav2tok introduces a CTC-style objective, together with clustering and contrastive learning, to encourage paired utterances to produce compatible token sequences under a monotonic sequence criterion. Thus, wav2tok can be viewed as an alignment-aware retrieval tokenizer: the symbolic sequence is trained so that related speech remains comparable under a sequence-level matching geometry. Retrieval-oriented geometric tokenizers. A separate line of retrieval-oriented tokenizers learns discrete units through geometric self-supervision rather than explicit sequence-level token alignment. BESTSTD (Singh et al., 2025a) combines contrastive learning with vector quantization to obtain scalable selfsupervised speech tokens for retrieval. Cross-utterance variability is handled before token learning through implicit frame-level alignment: DTW over continuous features selects anchor–positive frame pairs for the contrastive objective, while a commitment loss trains the quantizer. The resulting tokens are useful for retrieval, but their sequence structure is still primarily inherited from frame-level geometric assignment rather than learned through a sequence-alignment objective. Subsequent work strengthens this geometric retrieval-tokenization paradigm. LAST-STD (Singh et al., 2025c) targets multilingual retrieval and mitigates codebook index collapse through an optimal-transport regularizer (Cuturi, 2013), while BEST-STD 2.0 (Singh et al., 2025b) improves noise-robust tokenization using an optimal-transport-based balanced clustering objective. These methods show that retrieval-useful discrete speech units can be learned scalably without reconstruction supervision, but they do not make the generated token sequence itself the primary object of alignment. PairAlign follows the alignment-aware direction opened by wav2tok, but changes the probabilistic mechanism. Rather than using a CTC-style frame-synchronous sequence criterion over geometric or clustered token targets, PairAlign treats tokenization as conditional autoregressive sequence generation. The model starts from BOS, emits a complete token string, and terminates with EOS; token identity, ordering, length, and termination are therefore learned properties of the symbolic interface. Pairwise self-alignment then trains related acoustic views to induce mutually predictable token sequences, while unrelated examples provide competing symbolic strings. Thus, PairAlign extends alignment-aware retrieval tokenization from monotonic sequence scoring toward learned sequence-token generation. Relation to PairAlign. PairAlign is closely related to retrieval-oriented tokenization in its overall goal: both seek symbolic representations that remain stable under nuisance variation while being compact, discriminative, and useful for matching. The key difference is where sequence structure is imposed. In most retrieval-oriented geometric tokenizers, token identity is defined through frame-synchronous discretization of encoder features, and sequence-level consistency is encouraged indirectly through frame-level positives, auxiliary losses, or downstream matching procedures. This design has an important practical advantage: because tokens are produced frame by frame, they are naturally time-aligned and therefore well suited to localization-sensitive retrieval. PairAlign shifts the emphasis from frame-level token assignment to sequence-level token learning. Starting from a VQ-style tokenizer, it introduces a causal autoregressive decoder and trains it through pairwise selfalignment so that related realizations are encouraged to induce mutually predictable token sequences. Thus, sequence consistency, compactness, and token ordering are modeled more explicitly in the symbolic space itself, rather than being inherited only indirectly from frame-synchronous discretization. Trade-off: sequence compactness versus native timing. This shift introduces a clear trade-off. Because autoregressive decoding produces compact token sequences rather than frame-synchronous labels, PairAlign representations are not natively time-aligned in the same way as geometric tokenizers. For applications requiring precise temporal localization, this is a genuine limitation. In the present work, we partially 18
address this limitation through a post-hoc timing recovery procedure based on decoder cross-attention and lightweight monotone decoding, which provides approximate temporal grounding without changing the training objective. Complementarity to retrieval-oriented tokenization. A second difference is the operating regime. Retrieval-oriented tokenizers are often designed around relatively short windows or localized matching units, whereas PairAlign is trained on longer continuous segments so that autoregressive sequence modeling has enough context to learn nontrivial token order and compact symbolic structure. Accordingly, PairAlign should not be viewed as a direct replacement for retrieval-oriented geometric tokenization. Rather, it is a complementary direction: retrieval-oriented tokenizers prioritize frame-synchronous, localization-friendly symbolic representations, while PairAlign prioritizes explicit sequence-level symbolic structure and retains a practical bridge to time-aware applications through approximate post-hoc grounding. 2.6
Neural Sequence Transduction, CTC, RNN-T, wav2tok, and PairAlign
Neural sequence transduction. sequence
Neural sequence transduction studies how a model maps an input X = [x1 , . . . , xT ]
to an output sequence Y = [y1 , . . . , yU ], often with U ̸= T and with unknown correspondence between input and output positions. This problem appears in speech recognition, handwriting recognition, machine translation, transliteration, speech synthesis, and related sequence-to-sequence tasks. Different neural transduction models mainly differ in how they represent, constrain, or marginalize the input–output alignment. CTC and RNN-T are alignment-marginalized transduction objectives (Graves et al., 2006; Graves, 2012). Attention-based encoder–decoder models instead learn a soft alignment through decoder attention over encoder states, as in neural machine translation and Listen, Attend and Spell (Bahdanau et al., 2014; Luong et al., 2015; Chan et al., 2015). Monotonic attention, monotonic chunkwise attention, and hard monotonic attention further constrain this alignment structure for streaming, speech-like, or approximately monotone transduction settings (Raffel et al., 2017; Chiu & Raffel, 2017b; Wu & Cotterell, 2019). These methods establish the general neural-transduction pattern: a continuous or symbolic input sequence is encoded, an output sequence is generated, and an alignment mechanism connects output positions to input positions. CTC as frame-synchronous alignment marginalization. CTC is not a geometric tokenizer. It does not require token identities to arise from nearest-centroid assignment or vector quantization. Rather, it defines a frame-synchronous posterior model over an output alphabet augmented with a blank symbol. Let Z = [z1 , . . . , zT ] denote encoder states and let Y = [y1 , . . . , yU ] be an externally specified target sequence. For a frame-level alignment path π = (π1 , . . . , πT ), with πt ∈ A ∪ {∅}, CTC factorizes the path probability over encoder time: T Y pCTC (π | Z) = p(πt | Z). (6) t=1
The target probability is obtained by summing over all paths that collapse to Y: pCTC (Y | Z) =
X
pCTC (π | Z),
(7)
π∈B−1 (Y)
where B removes blanks and repeated labels. Thus, CTC is frame-synchronous in its scoring structure, even when the encoder itself is contextual or bidirectional. It does not model output labels through an autoregressive factorization p(yu | y<u , Z). The symbolic target sequence is assumed to be known; the objective learns to align that target to the input under monotonic collapse. 19
RNN-T as autoregressive monotonic transduction. RNN-T relaxes the output-independence structure of CTC by introducing a prediction network conditioned on previously emitted non-blank labels. At a lattice state (t, u), the model scores either a blank transition, which advances in input time, or a label transition, which emits the next output symbol: p(k | t, u) = p(k | ht , gu ),
gu = PredNet(y<u ).
The probability of Y is still obtained by marginalizing over monotonic input–output paths: X Y pRNNT (Y | X) = p(πt,u | ht , y<u ).
(8)
(9)
π∈A(Y) (t,u)∈π
RNN-T therefore contains autoregressive label-history dependence, but it is not a standard encoder–decoder factorization over a single output trajectory. Its likelihood remains an alignment-marginalized transduction likelihood. As with CTC, the output alphabet and target sequence semantics are specified by the task before training. Attention-based and monotonic transduction. Attention-based encoder–decoder models provide a different solution to the same transduction problem. Instead of summing over discrete monotonic paths, they generate each output symbol autoregressively while attending to encoder states: p(Y | X) =
U Y
p(yu | y<u , cu ),
(10)
u=1
where cu is an attention-derived context vector over the encoder sequence. This formulation underlies attention-based neural machine translation (Bahdanau et al., 2014; Luong et al., 2015) and speech recognition models such as Listen, Attend and Spell (Chan et al., 2015). Because unconstrained attention need not be monotonic, later work introduced online monotonic attention and monotonic chunkwise attention to better match streaming speech and other monotone transduction settings (Raffel et al., 2017; Chiu & Raffel, 2017b). Related hard-attention transduction models make the alignment variable explicit: for example, (Wu & Cotterell, 2019) enforce strict monotonicity in a hard-attention sequence-to-sequence model and compute the exact marginal likelihood over monotonic alignments with dynamic programming, showing that monotonicity can be a useful inductive bias when alignments are learned jointly with the transduction model. These monotonic attention models are especially compatible with PairAlign. They preserve the encoder– decoder view while replacing unconstrained cross-attention with an alignment mechanism that is monotonic by construction. The present paper uses a Whisper-style Transformer decoder (Radford et al., 2023), whose cross-attention is not constrained to be strictly monotonic. This choice keeps the architecture simple and makes the proposed self-alignment objective easy to instantiate. At the same time, speech-to-symbol mappings are approximately monotone, and prior work has observed that cross-attention in speech encoder– decoder models can contain useful soft or “fuzzy” alignment structure even without a hard monotonic constraint (Neekhara et al., 2024). We use this observation only for post-hoc temporal grounding: after decoding, we treat decoder cross-attention as a soft token-to-frame association, apply a weak monotonic prior, and recover approximate timestamps through monotone Viterbi post-processing. This design separates the core tokenization objective from the timing-recovery procedure. PairAlign itself does not require a monotonic decoder, and the reported instantiation uses unconstrained Transformer crossattention. However, extending PairAlign to a strictly monotonic version is conceptually direct: one can replace the standard cross-attention module with a monotonic, chunkwise-monotonic, hard-monotonic, or other alignment-aware decoder while keeping the same cross-paired self-alignment objective. Such a variant would make token-to-frame correspondence part of the model architecture rather than recovering it after decoding, and is a natural extension for segmentation, streaming, or localization-sensitive retrieval. Distinction from PairAlign. PairAlign belongs to the broad encoder–decoder transduction family, but it uses transduction for a different object. In conventional neural sequence transduction, the symbolic target sequence Y is given by supervision or pseudo-supervision. The model learns to map X to Y, while the alignment mechanism handles the unknown correspondence between input and output positions. 20
PairAlign instead uses sequence transduction to induce the symbolic interface itself. Given an acoustic condition Z = Enc(x), it defines a direct conditional distribution over a learned token sequence T = [τ1 , . . . , τL ]: L Y pθ (T | Z) = pθ (τl | τ<l , Z). (11) l=1
There is no training-time marginalization over frame-level alignment paths, no frame-synchronous posterior sequence whose collapse defines the output, and no externally specified transcript-like target. The absence of training-time alignment marginalization does not preclude using a monotonic decoder architecture; it only means that the present instantiation does not marginalize over a CTC- or RNN-T-style alignment lattice. At inference time, decoding begins from BOS, and the generated sequence itself is the tokenization assigned to the input. This also changes the status of output length. In CTC and RNN-T, the model learns an alignment distribution over a target sequence whose length and symbols are externally specified by the training example. In PairAlign, there is no fixed transcript-like target length. The decoder’s EOS decision determines the length of the learned tokenization itself. Thus, output length is not merely a byproduct of alignment marginalization or frame collapse; it is part of the induced symbolic representation. This changes the role of the objective. CTC, RNN-T, and conventional attention-based transducers optimize likelihood of a given target sequence. PairAlign does not have such a fixed target sequence. For paired + + content-preserving views (xi , x+ i ), with encoder states (Zi , Zi ) and induced token sequences (Ti , Ti ), the positive alignment signal is cross-paired conditional likelihood: log pθ (Ti+ | Zi ) + log pθ (Ti | Zi+ ).
(12)
This likelihood is not used to recognize a fixed label string. It is used as a differentiable surrogate for the desired token-space relation: two acoustic realizations of the same content should induce mutually predictable symbolic sequences. Positive predictability alone is not sufficient, because a collapsed tokenizer could make many inputs predictable by assigning generic sequences. PairAlign therefore combines Eq. 12 with in-batch likelihood contrast. For a conditioning representation Zi , the paired teacher sequence should score higher than mismatched teacher sequences from other examples: s̄(Tbi+ | Zi ) > s̄(Tbj+ | Zi ),
j ̸= i,
(13)
where s̄ denotes length-normalized conditional log-likelihood. Thus, PairAlign is not only a conditional sequence-generation objective; it is a discriminative self-alignment objective over learned symbolic sequences. Relation to wav2tok. wav2tok provides the closest prior connection between retrieval-oriented speech tokenization and transduction-style sequence likelihood (Banerjee & Arora, 2022). Like PairAlign, wav2tok treats speech tokens as symbolic sequences whose edit-distance behavior matters for retrieval. Its objective can be understood as adding a CTC-style pairwise sequence constraint to a frame-level token posterior model. Given paired views, wav2tok encourages the token sequence from one view to be likely under the framewise posterior sequence of the other view: − log pCTC (Ti+ | Pi ) − log pCTC (Ti | Pi+ ),
(14)
where Pi and Pi+ are frame-indexed token-posterior sequences. This is already more sequence-aware than purely local geometric assignment: the loss requires a paired token sequence to be recoverable under a monotonic alignment model, rather than only requiring aligned frames to be close in embedding space. PairAlign retains this central idea—paired views should agree as symbolic sequences—but changes both the probability model and the learned interface. The optional Stage I+ objective in this paper follows the wav2tok direction by using a no-blank CTC-style term to strengthen the VQ tokenizer while token identity remains tied to frame-level posteriors. PairAlign proper then moves beyond this regime. The sequence probability is no longer pCTC (T | P ), obtained by monotonic marginalization over frame-indexed posteriors; it is the autoregressive conditional likelihood in Eq. 11, obtained from a decoder that generates the compact tokenization from BOS. 21
This shift has two consequences. First, output length and EOS placement become explicit decoder decisions rather than consequences of framewise posterior collapse. Second, token dependencies are modeled directly through pθ (τl | τ<l , Z), instead of being mediated only by monotonic alignment paths. PairAlign can therefore be viewed as a sequence-generative extension of the wav2tok principle: wav2tok uses a CTC-style objective to make a frame-level tokenizer more sequence-consistent, whereas PairAlign uses cross-paired autoregressive likelihood and in-batch contrast to induce a compact symbolic sequence interface more directly. Summary of the distinction. The relevant distinction is not geometric versus non-geometric alone. CTC and RNN-T are not geometric tokenizers; they are alignment-marginalized neural sequence transduction objectives. CTC uses frame-synchronous posteriors and monotonic collapse. RNN-T adds autoregressive label-history dependence but still marginalizes over monotonic input–output paths. Attention-based encoder–decoders generate outputs autoregressively while attending to encoder states, and monotonic attention variants impose stronger alignment structure for streaming or speech-like settings. All of these methods map inputs to externally specified target sequences. PairAlign repurposes neural sequence transduction for tokenizer induction. It maps an acoustic input to a learned symbolic sequence whose identity, ordering, length, and termination are part of the representationlearning problem. It inherits from wav2tok the retrieval-oriented goal of making paired speech realizations agree as symbolic strings, but replaces frame-posterior alignment marginalization with direct conditional autoregressive generation of the token sequence itself. The objective therefore shifts from explaining a given target sequence under unknown alignment to inducing the symbolic target sequence through cross-view predictability, length-normalized likelihood, and discriminative separation from unrelated examples. 2.7
Self-Supervised Learning Paradigms and PairAlign
Self-supervised learning paradigms. Self-supervised learning defines training signals from the structure of the data itself rather than from external labels. Several broad paradigms are relevant to PairAlign. A first family learns by recovering hidden, corrupted, or compressed content. Classical denoising autoencoders reconstruct clean inputs from corrupted observations (Vincent et al., 2008). Masked language modeling predicts missing text tokens from bidirectional context (Devlin et al., 2019), while masked image modeling and masked acoustic modeling apply related ideas to images and speech (Xie et al., 2022; Liu et al., 2020). In speech SSL, wav2vec 2.0 combines masked latent prediction with contrastive learning over quantized targets, while HuBERT and w2v-BERT use masked prediction over clustered or learned hidden targets (Baevski et al., 2020; Hsu et al., 2021; Chung et al., 2021). These methods learn representations by requiring the model to infer unavailable parts of the signal from surrounding context. A second family is based on multiview correspondence. Two or more views of the same underlying example are constructed, and the model is trained so that the corresponding representations agree. In speech, such views may be obtained by changing nuisance factors such as gain, noise, reverberation, filtering, or channel coloration while preserving the underlying spoken content. Contrastive methods such as DrLIM, CPC, MoCo, and SimCLR use positive and negative examples to organize representation space (Hadsell et al., 2006; Oord et al., 2018; He et al., 2020; Chen et al., 2020). Non-contrastive and redundancy-reduction methods such as BYOL, SimSiam, Barlow Twins, and VICReg avoid explicit negatives through architectural asymmetry, stop-gradient operations, momentum teachers, variance constraints, covariance regularization, or redundancy reduction (Grill et al., 2020; Chen & He, 2021; Zbontar et al., 2021; Bardes et al., 2021). From a multiview perspective, these methods rely on the assumption that different views preserve task-relevant shared information while discarding view-specific nuisance variation (Tian et al., 2020; Tsai et al., 2020; Federici et al., 2020; Shwartz Ziv & LeCun, 2024). A third family uses prediction, clustering, or teacher-generated targets. CPC predicts future latent representations using a contrastive objective (Oord et al., 2018). APC predicts future acoustic frames or features from past context using an autoregressive model (Chung et al., 2019). VQ-APC introduces a vector-quantized bottleneck into this predictive framework, coupling future prediction with discrete latent organization (Chung et al., 2020). Clustering-based and online-assignment methods such as DeepCluster, SwAV, and HuBERT construct pseudo-targets from the data and train models to predict or stabilize those assignments (Caron 22
et al., 2018; 2020; Hsu et al., 2021). Teacher-student and EMA-teacher methods provide another route to self-supervised targets: a teacher network, often updated as an exponential moving average of the student, supplies stable targets for the student to match (Grill et al., 2020; Caron et al., 2021; Baevski et al., 2022). Connection to predictive world models and JEPA-style learning. PairAlign is also related to predictive representation-learning frameworks such as JEPA-style models, which emphasize prediction in representation space rather than reconstruction of raw sensory input (LeCun et al., 2022; Assran et al., 2023; Bardes et al., 2024; Fei et al., 2023). In such frameworks, a context representation is trained to predict an abstract target representation of another view, masked region, or future state, rather than the raw input itself. This principle is closely aligned with the motivation behind PairAlign: the model should predict an abstract object associated with a paired view, rather than reconstructing the waveform, spectrogram, or raw acoustic observation. PairAlign can therefore be viewed as a sequence-symbolic analogue of JEPA-style predictive learning. The key difference is the form of the predicted target. Standard latent-prediction objectives typically predict fixed-dimensional continuous representations, local latent targets, or contextual embedding statistics, as in JEPA-style models and related masked or teacher-student representation prediction methods (Assran et al., 2023; Bardes et al., 2024; Baevski et al., 2022; Fei et al., 2023). PairAlign instead predicts a learned, variablelength discrete token sequence produced from another content-preserving view of the same underlying audio segment. Thus, the target has explicit order, token identity, an induced vocabulary, an EOS decision, and an edit-distance geometry. The model must learn not only what information should be predictable across views, but also how that information should be serialized into a compact symbolic form. This sequence-symbolic formulation gives PairAlign affordances that ordinary continuous latent-prediction objectives do not directly expose: indexability, length adaptivity, symbolic comparison, and compatibility with sequence-level retrieval. In this sense, PairAlign provides an audio instantiation of symbolic JEPAstyle learning, where self-supervision induces not only an abstract representation space but also the discrete sequence interface through which the representation is expressed. Predictive speech SSL versus PairAlign. Predictive speech SSL methods are important precedents because they show that prediction over speech can induce useful representations, and that discrete bottlenecks can be integrated into predictive learning. However, they optimize a different object from PairAlign. APCstyle models predict future acoustic features or local latent targets. VQ-APC adds vector quantization, but the learning problem remains tied to future-frame or local-latent prediction. Masked acoustic models reconstruct hidden acoustic observations from context. These objectives may yield continuous features, quantized latents, or clustered units, but they do not train a decoder to generate the complete symbolic tokenization assigned to an input segment. PairAlign changes the prediction target. The target is not a future acoustic frame, a masked feature vector, or a local quantized latent. The target is a complete ordered token string representing the input segment, and the length of this string is also learned. This differs from many SSL targets whose temporal resolution is inherited from the encoder, mask schedule, or clustering frame rate. PairAlign therefore treats symbolic rate and termination as part of the self-supervised learning problem. Thus, prediction in PairAlign is sequence-level and symbolic: the model assigns conditional likelihood to a token sequence with explicit order, length, and termination. In this sense, PairAlign is closer to conditional sequence modeling than to ordinary future-frame prediction, although the purpose of the decoder is not audio generation or text generation, but tokenization. Relation to EMA-teacher and self-distillation SSL. EMA-teacher SSL methods are also relevant to PairAlign. BYOL uses a momentum encoder and stop-gradient prediction to learn visual representations without explicit negatives (Grill et al., 2020). DINO uses a momentum teacher together with centering and sharpening to learn self-distilled visual representations (Caron et al., 2021). data2vec generalizes this teacher-student principle across speech, vision, and language by training a student to predict contextual latent targets produced by an EMA teacher (Baevski et al., 2022). These methods demonstrate that slowly 23
moving teachers can stabilize self-supervised learning when the targets are generated by the model itself rather than supplied externally. PairAlign uses a related stabilization principle in Stage III, but the teacher target has a different form. In BYOL, DINO, and data2vec, the teacher provides continuous representation targets, probability targets, or contextual latent targets. The student is trained to match those targets under another view or a masked version of the input. In PairAlign, the EMA teacher generates an entire symbolic token sequence in freerunning autoregressive mode. The student is then trained to assign high conditional likelihood to the sequence generated from the paired view. Thus, the EMA teacher in PairAlign stabilizes an evolving symbolic interface, not only an embedding target or contextual latent target. This distinction is central because PairAlign’s target space itself changes during training. Stage I defines token identity by nearest-centroid vector quantization. Stage II freezes this geometric tokenizer and uses it as a deterministic teacher, giving stationary sequence targets while the autoregressive decoder learns cross-paired conditional prediction. Stage III replaces the fixed teacher with an EMA teacher over the full encoder–decoder model, allowing the encoder representation, decoder, and generated token sequences to coevolve. The slow teacher update prevents the symbolic target distribution from drifting too abruptly, while the student objective imposes cross-paired alignment and in-batch likelihood separation. PairAlign therefore uses EMA self-distillation not only for representation matching, but for adaptive self-alignment of generated token sequences. Relation to retrieval-oriented tokenization. wav2tok is closer to PairAlign than most embeddinglevel SSL methods because it treats speech tokens as symbolic sequences whose retrieval behavior matters (Banerjee & Arora, 2022). It combines clustering, contrastive learning, and a CTC-style likelihood constraint so that paired speech realizations induce token sequences that remain compatible under a monotonic sequence criterion. This already moves beyond ordinary frame-level representation learning by introducing an ordersensitive sequence constraint. PairAlign inherits this retrieval-oriented motivation, but changes the probability model used to impose sequence structure. wav2tok keeps token induction tied to frame-level posteriors and uses CTC-style alignment marginalization to compare paired token sequences. PairAlign instead uses an autoregressive decoder to generate the compact token sequence directly from the acoustic condition. Consequently, token order, token dependencies, output length, and EOS placement become explicit decoder decisions rather than consequences of framewise posterior collapse or nearest-centroid assignment. Thus, PairAlign can be viewed as a sequence-generative extension of the retrieval-oriented tokenization principle: paired realizations should agree as symbolic strings, but agreement is optimized through cross-paired conditional likelihood rather than through frame-synchronous alignment marginalization. PairAlign as cross-paired generative self-alignment. PairAlign is a multiview SSL method in which view agreement is defined over learned symbolic sequences rather than only over continuous embeddings. For a content-preserving pair (x, x+ ), the encoder produces conditioning representations for both views. The token sequence associated with one view is then trained to be likely under the conditioning representation of the other view. Thus, the anchor representation scores the positive-view token sequence, and the positive representation scores the anchor-view token sequence. This cross-paired direction is central: agreement is measured by conditional sequence likelihood, not merely by vector-space proximity. PairAlign is generative in a specific and limited sense. The autoregressive decoder generates the tokenization assigned to an acoustic input from BOS under the encoder condition. The generated object is not waveform audio, a spectrogram, a text response, or a continuation from an external prompt. It is the compact symbolic sequence that defines the representation of the input itself. The conditional likelihood of this sequence is then used as a self-supervised alignment score between paired views. This distinguishes PairAlign from APC and VQ-APC. APC-style methods use prediction to learn speech representations. PairAlign uses conditional generation to induce and align the symbolic sequence interface itself. The decoder is therefore not only an auxiliary predictor placed on top of a fixed representation; it is part of the tokenizer. 24
Why corruption is restricted. PairAlign uses corruption only in a restricted role. The acoustic views supplied to the encoder remain content-preserving and temporally ordered. This is necessary because the temporal order of the acoustic context is part of the content being tokenized. Strongly deleting, rotating, or permuting encoder-side frames would change the problem from cross-realization token alignment into missing-content reconstruction or artificial order recovery. The decoder side also has an ordering constraint. The generated token order is itself part of the learned symbolic representation. Arbitrary deletion or reordering of decoder-side target tokens would therefore conflict with the causal generation process whose output defines the tokenization. For this reason, PairAlign does not adopt broad text-denoising schedules as encoder-side or decoder-side corruptions. The only decoderside corruption used in the sequence-prediction path is prefix corruption during teacher forcing: selected tokens in the shifted prefix are replaced by a mask symbol, while the clean ordered target sequence remains the prediction target. Its purpose is not source denoising, but to weaken reliance on the target-side prefix and make the cross-paired likelihood depend more strongly on the acoustic condition.
Symbolic alignment rather than embedding invariance alone. Standard multiview SSL usually aims to learn embeddings that are invariant or equivariant to chosen augmentations. PairAlign shares the multiview motivation, but expresses agreement in token space. A successful tokenizer should produce compact sequences whose token identities, ordering, lengths, and edit-based similarities remain stable across content-preserving realizations, while unrelated inputs remain distinguishable. This makes the failure modes different from ordinary embedding-level SSL. Collapse can occur not only as representation collapse, but also as many-to-one tokenization, repetitive decoding, unstable sequence length, poor EOS behavior, or decoder bypass. Decoder bypass is especially relevant because teacher forcing gives the decoder access to the target-side prefix during training. The decoder may then obtain high likelihood by exploiting local token continuation patterns while underusing the acoustic condition. PairAlign addresses this through cross-paired prediction, prefix corruption, encoder-summary conditioning, structured self-attention dropout, EMA-teacher self-alignment, entropy regularization, and hardest-K likelihood contrast against mismatched in-batch sequences. These mechanisms are specific to the goal of learning an input-grounded symbolic tokenizer, rather than only an invariant continuous representation.
Summary. PairAlign is best understood as a self-supervised framework for generative sequence tokenization. It borrows the multiview principle that content-preserving views should agree, the predictive spirit of APC-style speech SSL, the pseudo-target logic of clustering and VQ methods, the retrieval-oriented sequence view of wav2tok, the stabilization principle of EMA-teacher methods such as BYOL, DINO, and data2vec, and the representation-space prediction principle associated with JEPA-style learning. However, it changes the object being learned. The target is not an invariant embedding, a future acoustic feature, a masked frame, a local cluster assignment, a contextual latent representation, or a fixed-dimensional predicted embedding. The target is an aligned, compact, and comparable symbolic sequence. Thus, PairAlign occupies a distinct position in the SSL landscape. Like multiview SSL, it learns from paired views without external labels. Like predictive SSL, it uses conditional likelihood. Like pseudo-target methods, it relies on teacher-generated targets. Like EMA-teacher SSL, it uses a slowly moving teacher to stabilize learning. Like JEPA-style learning, it predicts an abstract target associated with another view rather than reconstructing raw sensory input. Unlike these methods, however, the conditional generator is the tokenizer itself, and the EMA teacher supplies evolving symbolic strings rather than only continuous or distributional representation targets. This is the sense in which PairAlign uses self-supervision not only to learn a representation, but to induce the symbolic interface through which the representation is expressed. 25
3
Methodology
3.1
Problem Formulation: Sequence Tokenization as Conditional Language Modeling
Let D = {xi }N i=1 denote a dataset of variable-length sequences. Each sequence x is first mapped to a sequence of continuous latent representations by an encoder: Z ∈ Rd×T ,
Z = Enc(x),
(15)
where d is the latent dimension and T is the downsampled temporal length. We define sequence tokenization as the problem of mapping the continuous encoder sequence Z to a compact sequence of discrete symbols T = [τ1 , . . . , τL ], τl ∈ A, (16) where A is a finite alphabet with |A| entries and typically L ≪ T . The goal is not merely to discretize the signal, but to produce a symbolic sequence that is compact, ordered, and discriminative while preserving the similarity structure of the underlying input. We formulate tokenization as conditional language modeling. Given encoder representations Z, an autoregressive decoder DecAR (·) defines a left-to-right distribution over token sequences: p(T | Z; θAR ) =
L Y
p(τl | τ<l , Z; θAR ).
(17)
l=1
This formulation allows the tokenizer to model dependencies among output tokens while still grounding every next-token decision in the continuous encoder representation through cross-attention. Edit-distance preservation as the target property. The intended token space should preserve relational similarity. For a similar pair (x, x+ ) and an unrelated example x− ∈ D, let their tokenizations be T , T + , and T − , respectively. Ideally, the similar pair should be closer in token space than either sequence is to the unrelated example: ED(T , T + ) < min ED(T , T − ), ED(T + , T − ) , (18) where ED(·, ·) denotes edit distance. This criterion captures the desired behavior of a symbolic tokenizer: content-preserving variations should induce compatible token sequences, while unrelated inputs should remain separated. Why edit distance is the right evaluation geometry. Edit distance is a natural similarity measure for token sequences because it accounts jointly for token identity, order, length, and local structural alignment. Unlike set-based measures such as Jaccard similarity, edit distance does not discard temporal order or repeated structure. It penalizes substitutions, insertions, and deletions, making it sensitive to the types of changes that matter in sequential representations. This is important for audio, where phonetic and temporal structure are carried not only by which symbols appear, but also by their order, duration-dependent repetition pattern, and local alignment. Edit distance also exposes degenerate tokenization behavior more directly than unordered overlap measures. A collapsed tokenizer may reuse a small set of generic symbols across many inputs; such outputs can appear spuriously similar under set overlap, but still fail under an order- and length-sensitive metric. Thus, Eq. 18 expresses the structural property we want the learned symbolic space to satisfy. Why direct edit-distance optimization is not practical. Although edit distance defines the desired geometry, it is not a convenient training loss for a neural tokenizer. The standard edit-distance computation is a dynamic program over discrete sequences whose operations are substitutions, insertions, and deletions. These operations depend on hard token identities and discrete alignment choices. As a result, the editdistance value is piecewise constant with respect to the model logits over large regions and changes only 26
when an argmax token or an alignment decision changes. This gives no useful gradient almost everywhere, and unstable high-variance gradient estimates if one treats token generation as a sampling problem. Directly optimizing Eq. 18 would also require comparing variable-length generated sequences inside a triplet or ranking objective. This introduces several coupled difficulties: discrete decoding must occur inside the training loop, edit-distance alignments must be recomputed for many candidate negatives, and the resulting objective provides only a coarse sequence-level signal rather than token-level credit assignment. Thus, the issue is not that edit distance is conceptually wrong; it is that its raw form is combinatorial, non-differentiable with respect to the neural parameters, and poorly conditioned as a direct optimization target. Pairwise conditional likelihood as a scalable surrogate. Instead of optimizing edit distance directly, we design a differentiable surrogate that targets the same ranking structure. For a similar pair (xi , x+ i ), let Zi+ = Enc(x+ i ),
Zi = Enc(xi ),
(19)
and let Ti and Ti+ denote their token sequences. Rather than minimizing ED(Ti , Ti+ ) directly, we make the token sequence from one view predictable from the conditioning representation of the other: log p(Ti+ | Zi )
and
log p(Ti | Zi+ ).
(20)
Because these are autoregressive sequence likelihoods, they provide dense token-level supervision while still depending on token identity, order, and sequence termination. If the tokenization of one view receives high likelihood under the paired view’s conditioning representation, the two views are encouraged to share compatible symbolic structure. This surrogate is not mathematically equivalent to edit-distance minimization. It is instead an optimizationfriendly proxy for the same relational goal: paired views should induce mutually predictable token sequences, whereas unrelated examples should not. The advantage is that likelihoods are differentiable with respect to decoder logits and encoder representations, provide local credit assignment at each token position, and can be combined with in-batch contrastive separation. From pairwise predictability to discriminative token space. Positive-pair predictability alone is insufficient. A degenerate model could assign generic or repetitive sequences to many inputs and still make paired examples mutually predictable. To avoid this, we construct minibatches of disjoint positive pairs, B (xi , x+ i ) i=1 ,
(21)
and use token sequences from other batch elements as negatives. The desired surrogate ranking is that the paired sequence should receive higher conditional likelihood than mismatched token sequences from unrelated examples. Later stages implement this principle explicitly through in-batch hard-negative likelihood separation and entropy-based anti-collapse regularization. Thus, PairAlign does not claim to optimize Eq. 18 exactly. Instead, it converts the edit-distance preservation requirement into a scalable training principle: similar views should be mutually predictable as token sequences, while unrelated views should remain discriminable under conditional sequence likelihood. Notion of similarity in audio sequences. In this manuscript, we focus on audio. Two sequences x and x+ are considered similar when they represent different acoustic realizations of the same underlying content. Operationally, we construct such pairs using content-preserving augmentations: x+ ≈ Aug(x),
(22)
where Aug(·) may include controlled gain changes, mild additive noise, filtering, reverberation, rate variation, or other perturbations that preserve the lexical identity of the signal. In the word-level setting, this corresponds to lexical equivalence under augmentation-induced acoustic variability; in the continuous-speech setting, it corresponds to paired excerpts that preserve the same underlying spoken content while changing nuisance factors. 27
Overview of the training framework. To realize this formulation, PairAlign uses a three-stage training pipeline that moves from stable geometric discretization to adaptive sequence-level self-alignment. In Stage I, we train a continuous encoder and couple it with a nearest-centroid vector quantizer. This yields a strong VQ-style geometric tokenizer that provides both an initial symbolic interface and a controlled baseline. In Stage II, we freeze the Stage I encoder and vector quantizer and train an autoregressive decoder on deterministic token targets from this frozen teacher. This stage introduces the pairwise conditional likelihood surrogate in a stable setting: the decoder learns to assign high likelihood to the token sequence of one view when conditioned on the paired view’s encoder representation. Because the targets are fixed, Stage II establishes a stable conditional sequence model before self-generated supervision is introduced. In Stage III, we replace the deterministic teacher with an EMA teacher over the full encoder–decoder model and jointly refine both modules. The EMA teacher generates adaptive token targets, the student is trained with bypass-resistant masked teacher forcing, and in-batch likelihood contrast prevents many-to-one collapse. Together, these stages provide a practical optimization path for learning compact, robust, and edit-distance-aware symbolic tokenizations without directly backpropagating through edit distance. 3.2
Stage I: Learning a Base Encoder
In the first stage, our objective is to learn an encoder that produces rich and consistent latent representations suitable for downstream tokenization. We adopt a unidirectional Mamba encoder, based on the selective state-space model of Dao & Gu (2024), to efficiently capture temporal dependencies in sequential data. Mamba Encoder. Structured State-Space Models (S4) (Gu et al., 2021) represent sequential data through the evolution of a hidden state governed by linear time-invariant dynamics. Given an input x(t) ∈ RD , the hidden state h(t) ∈ RN evolves according to: h′ (t) = Ah(t) + Bx(t),
y(t) = Ch(t),
(23)
where A ∈ RN ×N , B ∈ RN ×D , and C ∈ RD×N . These continuous dynamics are discretized via a timescale parameter ∆ to obtain recurrent updates of the form ht = Ãht−1 + B̃xt ,
yt = Cht .
(24)
The Mamba model (Dao & Gu, 2024) generalizes this formulation by introducing time-varying parameters (At , Bt , Ct , ∆t ) that adapt dynamically with the input xt , yielding a selective state-space model capable of modeling longer contexts with improved efficiency. Self-Supervised Training. The encoder is trained with a self-supervised frame-level contrastive objective, as introduced in BEST-STD (Singh et al., 2025a), to produce discriminative frame-level embeddings. Pairs of utterances of the same sequence are aligned via dynamic time warping (DTW), and anchor–positive pairs are constructed at the frame level. A contrastive loss encourages embeddings of aligned frames to be similar, while unrelated frames act as negatives. Formally, for a batch example (Zi , Zi+ ) with DTW alignment, the contrastive loss is T
(i)
Lcontrast =
exp(zi,t · z̃i,t∗ /τ ) 1X − log , PN T t=1 exp(zi,t · z̃i,t∗ /τ ) + n=1 exp(zi,t · zi,n /τ )
(25)
where zi,t and z̃i,t∗ are aligned embeddings, τ is a temperature hyperparameter, and {zi,n } are negatives from other pairs in the batch. 28
To prepare the representations for discretization, we introduce a commitment loss that pulls embeddings toward their assigned VQ centroids: (i)
Lcommit =
T̃ T 1X 1X q 2 q 2 ∥z̃i,t − z̃i,t ∥2 , ∥zi,t − zi,t ∥2 + T t=1 T̃ t=1
(26)
q is the quantized embedding (nearest centroid). The total objective is a weighted combination: where zi,t B
L=
1 X (i) (i) Lcontrast + λ Lcommit , B i=1
(27)
where λ balances the two losses. From Encoder to Tokens. Once the encoder produces contextual embeddings Zi = [zi,t ]Tt=1 ∈ Rd×T , we apply a vector quantizer (VQ) h(·) with codebook C = {ca ∈ Rd | a ∈ A}, where |A| = K. For each frame q t, the VQ assigns a token index τt ∈ A and its quantized embedding zi,t via τi,t = h(zi,t ) = arg min ∥zi,t − ca ∥22 , a∈A
q zi,t = cτi,t .
(28)
This yields the (length-Ti ) raw token sequence Tiraw = [τi,1 , . . . , τi,T ]. Optionally, we apply a run-length deduplication operator ϕ(·) to reduce consecutive repetitions and obtain the compact token sequence Ti = i ϕ(Tiraw ) = [τi,l ]L l=1 with Li ≤ Ti during inference. The centroids C are updated via exponential moving average (EMA) to ensure stable codebook usage and prevent collapse. The resulting discrete token sequences Ti form the supervision targets for Stage II, where an autoregressive decoder is trained to model p(Ti | Zi ; θAR ). Relation to VQ-VAE-style tokenization and baseline relevance. Although PairAlign ultimately departs from purely geometric discretization, the first stage of our framework remains closely connected to the VQ-VAE family of tokenizers (Van Den Oord et al., 2017; Baevski et al., 2019; Hsu et al., 2021; Chung et al., 2021). Like VQ-VAE-style approaches, Stage I first maps the input sequence to a continuous latent representation and then applies nearest-neighbor vector quantization against a learned codebook, together with a commitment objective that encourages encoder outputs to remain close to their assigned centroids. In this sense, the encoder–quantizer interface used in PairAlign is structurally aligned with the dominant paradigm in prior work: discrete symbols are initially induced by geometric partitioning of a continuous latent space. This makes Stage I a meaningful and controlled bridge to the existing literature on neural codecs, self-supervised quantizers, and retrieval-oriented VQ tokenizers, all of which rely on the same basic principle that token identity is determined locally by latent proximity. At the same time, our use of a strong contextual encoder before quantization ensures that the resulting VQ assignments constitute a competitive and representative baseline for the later autoregressive stages. The Mamba encoder produces temporally contextualized, discriminative frame-level embeddings, while the VQ layer converts them into stable frame-synchronous token sequences exactly in the spirit of prior geometric tokenization methods. Consequently, if one were to stop after Stage I, the resulting system would already define a valid VQ-based tokenizer comparable in form to a large class of related methods. This is important methodologically: the gains of PairAlign in Stages II and III should be interpreted not as replacing a weak front end, but as building on top of a strong VQ-style discretization baseline. Our framework therefore remains directly comparable to related work based on VQ or VQ-VAE-like tokenization, while making explicit the central limitation of that paradigm: geometric quantization can provide stable local symbol assignments, but it does not by itself impose the sequence-level structure, compactness, or edit-distance consistency that PairAlign targets through autoregressive modeling and self-alignment. Optional Sequence-Level Strengthening of the VQ Tokenizer. While Stage I already yields a strong geometric tokenizer, the sequence-level structure it induces remains largely implicit: the contrastive objective 29
acts at the level of aligned frame embeddings, while the VQ layer assigns token identities independently at each time step. As a result, preservation of token order and cross-view sequence consistency is not enforced directly, but must instead emerge indirectly from the learned latent geometry. A natural extension is therefore to enrich Stage I with an additional explicit pairwise sequence-level constraint, as introduced in wav2tok (Banerjee & Arora, 2022), before transitioning to the autoregressive stages of PairAlign. The role of this intermediate strengthening step is to inject more direct order-sensitive supervision into the VQ tokenizer itself, while still remaining entirely within the frame-synchronous geometric discretization regime. Operationally, we first optimize the encoder and codebook using the Stage I self-supervised objective, obtaining a stable base geometric tokenizer. Only after this base model has been learned do we introduce the additional soft sequence-level constraint. This delayed introduction is important: the sequence-level term relies on well-formed token posteriors, and is therefore substantially better conditioned once the encoder has already learned a discriminative latent space. Let two paired views (xi , x+ i ) produce encoder representations Zi = Enc(xi ), and corresponding deduplicated VQ token sequences Ti = ϕ h(Zi ) ,
Zi+ = Enc(x+ i ),
(29)
Ti+ = ϕ h(Zi+ ) ,
(30)
where h(·) denotes nearest-centroid quantization and ϕ(·) denotes run-length deduplication. To define a soft sequence-level consistency objective, we first convert each frame representation into posterior probabilities over the token alphabet A. For the anchor view, let pi,t (a) ≜ p(τ = a | zi,t ) = P
⊤ exp(zi,t ca )
⊤ ′ a′ ∈A exp(zi,t ca )
,
a ∈ A,
(31)
and similarly for the positive view, + p+ i,t (a) ≜ p(τ = a | zi,t ).
(32)
Thus, for each paired example, the anchor and positive representations induce framewise posterior sequences Ti+ , a∈A Pi+ = p+ . i,t (a) t=1
Ti , a∈A Pi = pi,t (a) t=1 ,
(33)
We then encourage the deduplicated token sequence from one view to remain likely under the frame-level posterior sequence of the paired view using a no-blank CTC-style objective (Banerjee & Arora, 2022). In the forward direction, this gives + Li,→ (34) CTC = − log pCTC Ti | Pi , and in the reverse direction, + Li,← CTC = − log pCTC Ti | Pi .
(35)
Here, pCTC (T | P ) denotes the no-blank CTC marginal probability of token sequence T under the framewise posterior sequence P , where alignment is constrained to be monotonic. The pairwise sequence-consistency loss is then defined symmetrically as L(i) seq =
1 i,→ LCTC + Li,← CTC . 2
(36)
This term strengthens order information in the VQ tokenizer. Unlike the Stage I contrastive objective, which only enforces local alignment in representation space, the CTC loss requires the token sequence extracted from one view to be recoverable from the frame-level posteriors of the paired view under a monotonic alignment model. Thus, the tokenizer is encouraged not only to assign stable local codes, but also to preserve a soft notion of sequential consistency across paired realizations. 30
After the base Stage I training has converged, the strengthened objective for this auxiliary phase becomes B
LStageI+ =
1 X (i) (i) Lcontrast + λcommit Lcommit + λCTC L(i) seq . B i=1
(37)
Because the CTC term typically operates on a different numerical scale from the frame-level losses, we use an adaptive weighting strategy to stabilize this second-phase optimization: λCTC = γ
Lcontrast , Lseq + ε
(38)
where γ > 0 is a constant, ε > 0 is a numerical stabilizer, and Lcontrast and Lseq denote the corresponding minibatch averages. This keeps the effective contribution of the sequence-level term proportional to the contrastive objective during training, preventing the CTC term from dominating optimization when the token posterior model is still imperfect. We emphasize that this auxiliary strengthening phase remains conceptually distinct from PairAlign proper. Here, sequence information is used only to strengthen the VQ tokenizer through a soft pairwise constraint, while token identity is still defined by frame-synchronous nearest-centroid quantization. By contrast, the later PairAlign stages move beyond geometric discretization by learning compact token sequences directly through autoregressive conditional generation and self-alignment. 3.3
Stage II: A Scalable Surrogate for Edit-Distance Pairwise Token Consistency
We now introduce the training objective that serves as our scalable surrogate for the edit-distance constraint in Eq. 18, and instantiate it in two progressively more expressive stages. The central idea is to avoid direct optimization of edit distance over discrete sequences, which is combinatorial and difficult to scale, and instead optimize a pairwise likelihood objective that encourages token sequences from similar realizations to remain mutually predictable under cross-conditioning while being discriminable from in-batch alternatives. Stage II is also the point at which we introduce the autoregressive decoder, which forms the sequencemodeling component of PairAlign. Given encoder representations Z, the decoder DecAR (·) defines a causal left-to-right conditional language model over token sequences, so that each token is predicted from the preceding generated tokens together with the conditioning encoder latents through cross-attention. This decoder provides the mechanism through which PairAlign moves beyond the frame-synchronous geometric discretization of Stage I and begins to model token sequences directly. Let (xi , x+ i ) denote a pair of acoustically distinct but lexically equivalent sequences, with encoder representations Zi = Enc(xi ), Zi+ = Enc(x+ (39) i ). For a candidate token sequence T ′ , we define its conditional log-likelihood under the autoregressive decoder DecAR (·) as |T ′ | X ′ ′ ′ s(T | Zi ; θAR ) ≜ log πθAR (T | Zi ) = log πθAR τl′ | τ<l , Zi , (40) l=1
and its length-normalized version as |T ′ |
1 X ′ s̄(T | Zi ; θAR ) ≜ log πθAR τl′ | τ<l , Zi . ′ |T | ′
(41)
l=1
The normalized score s̄(·) removes the bias toward short sequences and is used throughout the sequel. Pairwise surrogate objective. For a similar pair (xi , x+ i ), the desired behavior is that the token sequence associated with x+ should score highly under Z , and symmetrically the token sequence associated with xi i i 31
should score highly under Zi+ . At the same time, unrelated token sequences within the minibatch should receive lower scores. Rather than constructing explicit triplets and optimizing a margin-based loss, we cast this as a contrast over sequence likelihoods: for each anchor conditioning representation, the positive paired tokenization is treated as the correct target among a set of in-batch competing token sequences. This yields a scalable surrogate for Eq. 18. If the model consistently assigns higher conditional likelihood to the paired tokenization than to competing in-batch tokenizations, then similar sequences are encouraged to occupy nearby regions in token space while unrelated sequences are repelled. The surrogate therefore preserves the relational content of the edit-distance constraint without requiring direct discrete edit-distance optimization. Why begin with a deterministic teacher? A key challenge is that the same model that evaluates token sequences could also be asked to generate them, making the objective prone to instability and tokenization collapse. We therefore begin with the simplest instantiation of the surrogate: a deterministic teacher derived from the Stage I encoder and vector quantizer. This choice provides stationary token targets while the decoder learns the pairwise conditional language model. Only after this base conditional model is established do we replace the deterministic teacher by an EMA teacher derived from the full model and allow both encoder and decoder to evolve jointly. 3.3.1
Deterministic Teacher from Frozen Encoder and Nearest-Centroid VQ
In Stage II, we keep the Stage I encoder and vector quantizer fixed and train only the autoregressive decoder. Let the frozen encoder outputs be Zi+ = Encfrz (x+ i ),
Zi = Encfrz (xi ),
(42)
and let the frozen nearest-centroid vector quantizer produce the corresponding raw token sequences Ti+
Tiraw = [τi,1 , . . . , τi,Ti ],
raw
+ + = [τi,1 , . . . , τi,T + ].
(43)
i
Applying the run-length deduplication operator ϕ(·) yields compact targets Ti = ϕ(Tiraw ),
Ti+ = ϕ(Ti+
raw
).
(44)
Motivation for the frozen VQ teacher. This design serves two purposes. First, the nearest-centroid VQ provides a deterministic tokenization rule: token identity is a fixed function of the frozen latent representation and the codebook. Second, freezing the encoder prevents the supervision targets from drifting while the decoder is being trained. Together, these choices yield a stable teacher for the pairwise surrogate, reducing optimization noise and making Stage II a controlled bridge from geometric discretization to autoregressive sequence modeling. Methodologically, this is also important for comparison with VQ- and VQ-VAE-style baselines: if training were stopped after this stage, the model would still remain anchored to a standard geometric tokenizer, differing only in the learned conditional decoder on top of it. Cross-paired teacher forcing. We implement DecAR (·) as a Whisper-style Transformer decoder (Radford et al., 2023). At each decoding step, the decoder applies causal self-attention over the previously generated token prefix and cross-attention to the encoder latents Z, thereby defining the conditional left-to-right sequence distribution scored by s̄(·). Given the deduplicated targets from the frozen teacher, we train the decoder by maximizing the lengthnormalized conditional score of each view’s token sequence under the paired view’s conditioning representation: s̄(Ti+ | Zi ; θAR ) and s̄(Ti | Zi+ ; θAR ). (45) The first score evaluates how likely the positive-view token sequence is when the decoder is conditioned on the anchor representation, while the second score applies the same criterion in the reverse direction. This cross-paired teacher-forcing construction instantiates the pairwise surrogate introduced above using fixed deterministic targets. Because the encoder and vector quantizer are frozen in Stage II, the target token sequences remain stationary while the decoder learns a stable pairwise conditional language model. 32
3.3.2
Preventing Decoder Bypass in Cross-Paired Teacher Forcing via Prefix Corruption, Encoder-Summary Bias, and Self-Attention Dropout
Stage II trains the autoregressive decoder using fixed token targets obtained from the frozen Stage I geometric tokenizer. This fixed teacher provides stationary supervision while the decoder learns a pairwise conditional sequence model. However, the Stage II objective is not ordinary language-model training. It uses crosspaired teacher forcing: the token sequence produced from one view is evaluated under the conditioning representation of the paired view. Therefore, the objective is useful only insofar as high likelihood reflects successful use of the conditioning representation Z, rather than merely successful continuation of the provided target-side prefix. Decoder bypass under teacher-forced likelihood. A central optimization risk in Stage II is that the decoder can achieve low teacher-forced loss without learning a genuinely input-grounded conditional tokenizer. During teacher forcing, the decoder is given the target-side prefix at every prediction step. This prefix contains substantial left-context information about the target sequence. As a result, the decoder can predict many next tokens by exploiting its autoregressive pathway—that is, by modeling local token continuation patterns from the provided prefix—while placing little effective reliance on the cross-attended acoustic representation Z. In the limiting case, the decoder may largely bypass Z: the objective can be optimized well because the target prefix itself supplies a strong shortcut for next-token prediction. This distinction is critical because the decoder in PairAlign is not merely a language model over plausible token strings. It is the mechanism that defines the tokenizer. Given an acoustic representation Z, it must generate the symbolic sequence specific to that input. At inference time, the decoder is no longer supplied with a clean ground-truth target prefix. It must begin from BOS, condition on Z, and then use its own previously generated tokens as the prefix for subsequent steps. Therefore, if the decoder has learned during training to solve next-token prediction primarily from teacher-forced prefixes, rather than from the acoustic condition, its free-running behavior can become input-insensitive. The early decoding steps are especially important. At these positions, the token-side prefix contains little or no information, so the model should rely strongly on Z to establish the input-specific symbolic trajectory. If, instead, the decoder ignores Z or uses it only weakly, it can commit to a generic high-probability prefix. Once such a prefix has been generated, later autoregressive steps may simply continue that prefix rather than recover the correct input-dependent token sequence from the acoustic representation. This produces a direct train–test mismatch: the model appears strong under teacher-forced likelihood, but free decoding becomes weakly conditioned on the input. In the extreme case, distinct acoustic inputs are mapped to the same or highly similar decoded sequences, yielding sequence-level collapse. We refer to this failure mode as decoder bypass. The term denotes a specific mismatch: the decoder appears successful under teacher-forced likelihood, but its success is supported primarily by the target-side autoregressive pathway rather than by the conditioning pathway through Z. As a result, the decoder may fail to learn an input-dependent tokenization rule: during free decoding, it can ignore the acoustic condition and instead generate a generic or trivial token sequence. Regularizing the teacher-forced objective. We use three complementary mechanisms to reduce decoder bypass: (i) prefix corruption, (ii) encoder-summary bias, and (iii) structured self-attention dropout. Prefix corruption weakens the reliability of the teacher-forced left context by randomly replacing selected prefix tokens with a mask symbol while keeping the prediction target unchanged. This makes teacher forcing a harder conditional prediction problem: the decoder must predict the clean next token from a partially degraded token history together with Z, rather than relying entirely on a clean local continuation history. Encoder-summary bias provides a direct conditioning signal to every teacher-forced decoding position. Instead of relying only on cross-attention layers to discover the acoustic condition, we compute a compact summary of the encoder representation and add it to the decoder token stream at every position. This gives each decoding step immediate access to a global summary of the conditioning signal before self-attention and cross-attention are applied. The decoder input therefore combines the previous token embedding, the positional embedding, and the encoder-summary embedding, so each prediction is conditioned on both the 33
acoustic content and the current position in the output sequence. This creates a permanent input-dependent pathway to the decoder and reduces the chance that the model solves teacher-forced prediction only from the autoregressive prefix. Structured self-attention dropout is the most direct intervention against the autoregressive shortcut. During training, we stochastically drop the decoder self-attention residual branch, thereby weakening the tokenside continuation pathway itself. When this branch is suppressed, the decoder must route more predictive information through the conditioning pathway, including cross-attention to Z and the injected encodersummary bias. Together, these mechanisms make the cross-paired teacher-forcing objective better aligned with the intended inference-time behavior. The model is still trained as a causal autoregressive decoder and still learns useful token-sequence dependencies. However, achieving low loss becomes more difficult through autoregressive prefix continuation alone. The decoder is therefore encouraged to behave as an input-conditioned tokenizer rather than as a generic token-sequence continuation model. Progressive relaxation of decoder-side regularization. The strength of the decoder-side regularization is scheduled during training rather than kept fixed throughout. In the early phase, prefix corruption and self-attention branch dropout are useful because they make the cross-paired teacher-forcing task harder to solve through the target-side prefix alone. However, if kept too strong until the end of training, these mechanisms can also keep the decoder in an artificially restricted operating regime. We therefore use a linear schedule for both the prefix-corruption rate and the self-attention dropout probability: s end s end pmask − pstart psa (s) = pstart + psa − pstart , (46) pmask (s) = pstart mask , sa sa mask + S S where s ∈ [0, S] is the current scheduled training step, S is the total number of scheduled steps, and the end start and end values specify the initial and final strengths of each regularizer. In our setting, pstart mask > pmask end start and psa > psa , so both regularizers are progressively relaxed during training. This schedule has two purposes. First, it creates a curriculum over the difficulty of the teacher-forced prediction task. Early in training, the decoder is forced to rely more strongly on the acoustic condition because the target-side prefix and self-attention pathway are partially unreliable. As training progresses, the corruption and dropout rates are gradually reduced, making the objective closer to ordinary conditional autoregressive likelihood. Second, it progressively restores the decoder toward its fuller-capacity inference configuration. By the final scheduled phase, the model is optimized under a less restricted decoder pathway, reducing train–test mismatch while preserving the conditioning behavior learned during the harder early phase. Prefix corruption as harder teacher forcing.
Let
Y = [y1 , . . . , yM ]
(47)
denote the clean target token sequence, and let U = [u0 , u1 , . . . , uM −1 ] = [BOS, y1 , . . . , yM −1 ]
(48)
denote the shifted teacher-forcing input prefix. We sample a mask set M ⊆ {1, . . . , M }
(49)
over target positions. Each valid non-special prefix position is included in M with probability pmask (s), following the linear schedule in Eq. 46. The corrupted teacher-forcing prefix Ue is obtained by replacing the prefix token associated with each selected prediction location by a mask symbol: ( MASK, m ∈ M and um−1 ∈ / {BOS, EOS, PAD}, u em−1 = (50) um−1 , otherwise. The decoder is then evaluated on the clean target sequence using this corrupted prefix: pθAR (ym | u e<m , Z). 34
(51)
Next-token prediction with corrupted teacher forcing. Prefix corruption preserves the standard causal prediction problem: the decoder still predicts the clean next token at each step under the same autoregressive factorization. What changes is the reliability of the teacher-forced input prefix. By replacing selected prefix tokens with MASK, the model can no longer solve every prediction from a fully clean targetside history. Instead, it must combine the remaining prefix context with the conditioning representation Z and the encoder-summary bias c(Z). Thus, prefix corruption does not suppress autoregressive modeling; it regularizes it. The decoder still learns token-sequence dependencies, but high likelihood is made harder to achieve through local prefix continuation alone. This encourages the teacher-forced objective to reflect input-conditioned prediction rather than decoder bypass. Encoder-summary bias. Prefix corruption reduces the reliability of the target-side prefix, but it does not by itself guarantee that the decoder will use the acoustic condition. To make the conditioning signal available at every prediction step, we introduce an encoder-summary bias. Given encoder states Z = [z1 , . . . , zT ],
zt ∈ Rd ,
(52)
we first compute a pooled encoder summary T
z̄ =
1X zt , T t=1
(53)
or, when padding is present, the corresponding mask-normalized average over valid encoder positions. The summary is normalized and projected into the decoder state space: c(Z) = Wc LayerNorm(z̄) + bc ,
c(Z) ∈ Rddec .
(54)
Let em = E(e um−1 )
(55)
denote the token embedding at teacher-forcing input position m. In the decoder used in this work, positional information is represented by an additive learned positional embedding pm ∈ Rddec .
(56)
We therefore form the decoder input by adding the token embedding, positional embedding, and projected encoder summary: eem = em + pm + c(Z). (57) Thus, each prediction step receives three complementary signals: the previous token identity through em , the decoding position through pm , and the global acoustic condition through c(Z). The explicit addition of pm follows the positional-embedding convention of the Whisper decoder used in our implementation. It tells the decoder which prediction step is being processed, which is needed for modeling token order, sequence length, and EOS placement. For decoder variants that encode position internally, for example through rotary or relative position-aware attention, a separate additive positional embedding is not required. In those cases, c(Z) can still be added to the decoder token stream, while position information is supplied by the attention mechanism. The LayerNorm in Eq. 54 stabilizes this conditioning pathway before the summary is added to the decoder token stream. Because c(Z) is injected at every decoder position, normalization keeps the pooled summary scale comparable across examples, prevents the summary bias from dominating token and positional information, and reduces sensitivity to encoder-distribution drift during joint refinement. This provides a permanent pathway from the encoder representation to every decoder input position. The positional dependence is handled by the decoder’s positional encoding: in our implementation this is the 35
additive term pm , while in other Transformer variants it may be supplied inside the attention mechanism itself. The same encoder summary is injected at each step, but its effect is interpreted in the context of the current token identity and the current decoder position. Early decoding steps, which have little target-side context, therefore still receive direct acoustic information together with their position in the output sequence. Later steps receive the same input-dependent summary while also having access to a richer autoregressive prefix. This mechanism is different from a standard textual prefix. The encoder summary is not an additional token prepended to the sequence, nor is it a prompt that the decoder can simply continue. Instead, it is an input-dependent bias added directly to the decoder token stream at every prediction position. Thus, every next-token decision receives acoustic conditioning before the Transformer blocks are applied, reducing the chance that the decoder relies only on the teacher-forced prefix. The summary pathway is initialized conservatively so that the decoder begins close to the standard teacherforced model. Equivalently, the projection Wc can be zero-initialized or initialized with a small scale. This makes the encoder-summary path available from the start while avoiding a large disruptive shift at the beginning of Stage II training. Masked and unmasked score components. Following the distinction between where corruption is applied and where the prediction objective is evaluated, we define separate length-normalized scores over masked and unmasked prediction positions. Let M = {1, . . . , M } \ M
(58)
after excluding padding positions. With the gated encoder-summary bias included in the decoder input, the masked-position score is s̄m (Y | Z; θAR ) =
1 X log pθAR (ym | u e<m , Z, c(Z)), |M|
(59)
1 X log pθAR (ym | u e<m , Z, c(Z)). |M|
(60)
m∈M
and the unmasked-position score is s̄u (Y | Z; θAR ) =
m∈M
Here Z denotes the full encoder sequence used by cross-attention, while c(Z) denotes the pooled encodersummary bias injected into the decoder token stream. These two conditioning paths are complementary: cross-attention provides time-resolved access to the encoder states, whereas the encoder-summary bias provides a global input-dependent signal at every decoding position. We combine the two components into a masked teacher-forcing score: s̄MTF (Y | Z; θAR ) = α s̄m (Y | Z; θAR ) + (1 − α) s̄u (Y | Z; θAR ),
α ∈ [0, 1].
(61)
The corresponding loss is LMTF (Y | Z; θAR ) = −s̄MTF (Y | Z; θAR ).
(62)
Equivalently, the same score can be written as a weighted length-normalized log-likelihood over valid prediction positions: s̄MTF (Y | Z; θAR ) = P
1
X
m∈V wm m∈V
wm log pθAR (ym | u e<m , Z, c(Z)),
(63)
where V denotes the set of valid non-padding prediction positions. The weights wm specify the loss-placement regime: setting wm = 0 for m ∈ / M gives masked-only prediction, using positive weights for all m ∈ V gives an all-position loss, and assigning larger weights to masked positions gives an intermediate objective. 36
Interpretation of the masking weight. When α = 1, the score is computed only over masked positions. In this case, the decoder is rewarded for predicting clean target tokens at locations whose teacher-forcing context has been corrupted. When α = 0, the score is computed only over unmasked positions, making the objective closer to ordinary teacher forcing, except that the prefix used by the decoder may still contain masks at other locations. Intermediate values combine the two regimes. Masked positions increase the difficulty of teacher-forced prediction, while unmasked positions preserve the stabilizing signal of standard autoregressive likelihood training. Unless otherwise stated, we use the balanced setting α = 0.5, giving equal weight to masked and unmasked prediction positions throughout the experiments. Self-attention dropout for decoder-bypass control. Prefix corruption and encoder-summary bias reduce the ease with which the model can solve the teacher-forced objective from the target-side prefix alone, but they do not remove the autoregressive pathway. To address decoder bypass more directly, we apply structured dropout to the self-attention residual branch of the decoder. Let h(r) denote the hidden state entering decoder block r, and let SA(r) (·) and CA(r) (·, ·) denote the self-attention and cross-attention sublayers. We modify the block update as h(r+1) = h(r) + g (r) ⊙ SA(r) (h(r) ) + CA(r) (h(r) , Z) + · · · ,
(64)
g (r) ∼ Bernoulli(1 − psa )
(65)
where is a sample-wise gate and psa is the self-attention branch dropout probability. When the self-attention branch is dropped, the decoder cannot rely as strongly on token-side recurrence at that layer and must route more information through the conditioning pathway. This makes structured self-attention dropout the primary decoder-side mechanism for weakening the bypass route, while prefix corruption and encoder-summary bias make the input-conditioned route easier and more reliable to use. Regularized Stage II pairwise objective. Using the deterministic token targets from the frozen Stage I encoder and vector quantizer, we train the autoregressive decoder under cross-paired masked teacher forcing with gated encoder-summary bias and self-attention branch dropout. For an anchor–positive pair (xi , x+ i ), the forward direction evaluates the positive-view token sequence under the anchor representation, while the reverse direction evaluates the anchor-view token sequence under the positive representation. The Stage II objective is B i 1 Xh s̄MTF (Ti+ | Zi ; θAR ) + s̄MTF (Ti | Zi+ ; θAR ) . (66) LStageII = − B i=1 The first term encourages the frozen teacher token sequence of the positive view to receive high regularized likelihood under the anchor conditioning representation. The second term applies the same criterion in the reverse direction. Both terms are evaluated using the masked teacher-forcing score in Eq. 61, with corrupted teacher-forcing prefixes, gated encoder-summary bias, and self-attention branch dropout active during training. Minimizing Eq. 66 learns a pairwise conditional language model over fixed geometric-token targets. The crosspaired structure encourages mutually predictable token sequences across content-preserving views. Prefix corruption makes the teacher-forced likelihood less dependent on a fully clean target-side history. Gated encoder-summary bias injects a compact input-dependent condition into every decoding position, giving each next-token prediction direct access to the encoder-level summary. Structured self-attention dropout directly weakens the autoregressive bypass pathway and encourages greater dependence on the acoustic conditioning representation. Because the encoder and vector quantizer remain frozen in Stage II, the supervision is fixed and deterministic. This provides a stable bridge between the Stage I geometric tokenizer and the adaptive EMA-teacher self-alignment introduced in Stage III. 3.4
Stage III: Full-Model EMA Self-Alignment for Adaptive Tokenization
Stage II establishes a stable pairwise conditional decoder by using fixed token targets from the frozen Stage I geometric tokenizer. This is an intentionally conservative bridge: the target sequences are stationary, the 37
encoder is fixed, and only the autoregressive decoder is trained to model cross-paired token likelihoods. However, this also limits the expressiveness of the learned symbolic interface. The decoder can only learn to predict token sequences defined by the initial geometric tokenizer; it cannot improve the encoder representation or adapt the tokenization rule itself. Stage III removes this limitation by replacing the deterministic Stage I teacher with an exponential moving average (EMA) teacher over the full encoder–decoder model. Both the encoder and autoregressive decoder are now updated jointly, allowing the continuous conditioning space and the induced symbolic sequences to co-evolve. The EMA teacher supplies slowly varying sequence-level targets, while the student is optimized using the same bypass-resistant masked teacher-forcing score s̄MTF introduced in Stage II. Thus, Stage III extends the pairwise surrogate from fixed-token decoder training to adaptive full-model self-alignment. A central design principle in this stage is to separate three roles that would otherwise be entangled in selftraining. First, the EMA teacher generates target sequences in free-running autoregressive mode using its full conditional capacity. Second, positive cross-paired student alignment is evaluated with s̄MTF , so that the student remains regularized against decoder bypass. Third, in-batch negative comparison uses teacher-forced likelihoods whose prefixes are kept uncorrupted, so that hard negatives reflect symbolic confusability rather than prefix-corruption artifacts. This separation allows the symbolic interface to evolve while preserving stability, input dependence, and discriminability. 3.4.1
Full-Model EMA Teacher and Joint Encoder–Decoder Optimization
EMA teacher over the complete tokenizer.
Let the student parameters be
θ = (θenc , θAR ),
(67)
where θenc and θAR denote the encoder and autoregressive decoder parameters, respectively. We maintain teacher parameters θ̃ as an exponential moving average of the full student model: θ̃ ← α θ̃ + (1 − α) θ,
α ∈ (0, 1).
(68)
g g AR (·), The teacher therefore contains both a teacher encoder Enc(·) and a teacher autoregressive decoder Dec while the student contains the corresponding trainable modules Enc(·) and DecAR (·). For a paired example (xi , x+ i ), the EMA teacher produces conditioning latents g i ), Zei = Enc(x
g + ), Zei+ = Enc(x i
(69)
Zi+ = Enc(x+ i ; θenc ).
(70)
and the student produces Zi = Enc(xi ; θenc ),
Because θ̃ evolves more smoothly than θ, the teacher provides adaptive targets without exposing training to the full instability of direct self-generation by the rapidly changing student. Full-capacity EMA target generation. erates a reference token sequence
e the EMA teacher genGiven a teacher conditioning latent Z, Tb = (τ̂1 , . . . , τ̂L̂ )
(71)
using the top-p sampling procedure described in Section 3.4.3. Teacher generation is free-running autoregressive decoding rather than teacher-forced likelihood evaluation, so prefix corruption is not applicable in this setting. We therefore keep the EMA decoder at full capacity: self-attention branch dropout is disabled, and all conditional pathways remain active. In particular, the teacher decoder uses causal self-attention over its e and the gated encoder-summary previously generated tokens, cross-attention to the teacher encoder states Z, e bias induced by c(Z). Concretely, decoding begins from BOS, and each next token is sampled from a top-p truncated distribution. We apply the compound repetition penalty and differential stochastic schedule from Section 3.4.3: early decoding steps use higher temperature, larger top-p, and a weaker repetition penalty to encourage nondegenerate early token usage, while later steps are decoded more sharply. A sample-specific length cap 38
derived from the conditioning length prevents unbounded generation, and EOS is forced once this cap is reached. For each pair (xi , x+ i ), the EMA teacher generates two adaptive reference sequences, e+ ; θ̃). Tbi+ = Sample(Z i
Tbi = Sample(Zei ; θ̃),
(72)
These sequences replace the fixed Stage I VQ targets as the self-alignment supervision. The important distinction is that target generation is performed by the slowly moving EMA teacher, while likelihood evaluation and parameter updates are performed by the student. This avoids the most unstable form of self-training, in which a rapidly changing student directly generates and fits its own targets. Bypass-resistant student likelihood. Although the teacher generates targets with the full-capacity decoder, the student is not trained on positive pairs using an ordinary teacher-forced objective alone. The decoder-bypass risk from Stage II remains: if the student is evaluated only under uncorrupted teacher-forced prefixes, it may explain teacher-generated sequences through autoregressive continuation while underusing the conditioning representation Z. Therefore, positive student alignment in Stage III uses the Stage II masked teacher-forcing score s̄MTF (· | Z; θ). This means that, on the student side, teacher-forcing prefixes are corrupted, self-attention branch dropout is active during training, and the gated encoder-summary bias remains part of the decoder input. The student still has access to both conditioning pathways: cross-attention to the full encoder sequence Z, and the global encoder-summary bias c(Z). Consequently, Stage III uses full-capacity EMA targets, but fits them using a bypass-resistant student objective. Teacher-forced likelihoods for symbolic confusability. For negative comparison, we use the lengthnormalized likelihood s̄(· | Z; θ) from Eq. 41, evaluated with teacher-forcing prefixes left uncorrupted. The conditional architecture remains active: the decoder cross-attends to Z, and the gated encoder-summary bias c(Z) is still injected. Self-attention branch dropout may be retained during this likelihood evaluation to keep the comparison aligned with the regularized student pathway, but the token prefixes themselves are not corrupted. This distinction is important because the in-batch likelihood matrix is used to measure symbolic confusability. It should answer whether a conditioning representation prefers its paired teacher-generated token sequence over other teacher-generated token sequences in the minibatch. That comparison should not depend on a particular random prefix-corruption pattern. Thus, s̄MTF is used for positive bypass-resistant alignment, whereas s̄ with uncorrupted teacher-forcing prefixes is used for hard-negative selection and contrastive comparison. Differential learning rates for stable joint refinement. In Stage III, the encoder and autoregressive decoder are optimized jointly, but they play different roles. The decoder directly assigns sequence likelihoods, adapts to the evolving EMA-generated symbolic targets, and separates positive pairs from hard in-batch negatives. It therefore needs to respond relatively quickly to the sequence-level supervision. The encoder, by contrast, defines the continuous conditioning space on which both teacher generation and student likelihood evaluation depend. If the encoder changes too rapidly, the conditioning distribution, the teacher-generated targets, and the student likelihood landscape can all drift simultaneously. This creates a tightly coupled optimization problem in which the target space and scoring function co-evolve too abruptly. To reduce this coupling, we use differential learning rates: ηenc = 0.1 ηdec ,
(73)
where ηdec is the decoder learning rate. The decoder is thus allowed to adapt more rapidly to the evolving symbolic supervision, while the encoder is updated more conservatively so that the conditioning space changes smoothly. This preserves useful structure inherited from Stages I–II while allowing the full model to refine its tokenization behavior. 39
Cross-paired positive alignment scores. Stage III positive scores as
Using the student representations Zi and Zi+ , we define the
Pii ≜ s̄MTF (Tbi+ | Zi ; θ),
Pii+ ≜ s̄MTF (Tbi | Zi+ ; θ).
(74)
The first score evaluates the EMA-teacher tokenization of the positive view under the anchor student representation, and the second score applies the same criterion in the reverse direction. Unlike Stage II, these targets are no longer fixed outputs of a frozen VQ tokenizer. They are generated by an EMA teacher of the full model, so the training signal can evolve as the encoder and decoder improve. At the same time, because the positive scores are evaluated with s̄MTF , the student remains trained under the same bypass-resistant likelihood that was introduced in Stage II. 3.4.2
Preventing Tokenization Collapse with In-Batch Likelihood Contrast
Maximizing only the cross-paired positive scores is not sufficient. A degenerate tokenizer can assign high likelihood to paired examples while also assigning high likelihood to unrelated examples, producing manyto-one symbolic collapse. Stage III therefore adds an in-batch discriminative pressure: for each conditioning representation, the paired teacher-generated sequence should score higher than the most confusable teachergenerated sequences from other examples in the minibatch. In-batch likelihood matrix for symbolic confusability. We construct the negative-comparison matrices using teacher-forced likelihoods whose prefixes are kept uncorrupted. The purpose of this matrix is not to test robustness to prefix corruption, but to measure whether a conditioning representation prefers its matched symbolic sequence over other candidate symbolic sequences. The decoder remains conditional throughout this evaluation: cross-attention to Z and the encoder-summary bias c(Z) are still active. For the forward direction Zi → Tbj+ , we define → Mij ≜ s̄(Tbj+ | Zi ; θ),
i, j ∈ {1, . . . , B},
(75)
where the diagonal entry corresponds to the paired teacher-generated target and the off-diagonal entries correspond to mismatched teacher-generated targets from other batch items. Similarly, for the reverse direction Zi+ → Tbj , we define ← Mij ≜ s̄(Tbj | Zi+ ; θ). (76) Each row fixes a conditioning representation, and each column provides a candidate symbolic sequence. The desired behavior is that the diagonal entry dominates the off-diagonal entries. This separates the role of positive alignment from the role of negative comparison: s̄MTF trains paired prediction under the bypassresistant objective, while the likelihood matrices above identify which unpaired teacher-generated sequences are most confusable under the conditional decoder. → Hardest-K confusable negatives. For each anchor i, we rank the off-diagonal forward scores {Mij : j ̸= i} and retain the indices of the K largest entries: → → HK (i) ≜ TopK {Mij : j ̸= i}, K . (77)
These are the teacher-generated token sequences that currently appear most compatible with the wrong conditioning representation Zi . The reverse-direction hard-negative set is defined analogously: ← ← HK (i) ≜ TopK {Mij : j ̸= i}, K . (78) Focusing on the hardest K alternatives has two effects. It reduces the cost of the contrastive objective relative to using all off-diagonal entries, and it concentrates the learning signal on the mistakes that matter most for collapse prevention: cases where unrelated symbolic sequences are already assigned high conditional likelihood. 40
Row-wise contrast over sequence likelihoods. Using the diagonal entry and its hardest off-diagonal competitors, we optimize a row-wise log-softmax over length-normalized sequence likelihoods. For the forward direction, B exp Mii→ /τℓ 1 X → P , log (79) LNCE = − → /τ B i=1 exp Mii→ /τℓ + j∈H→ (i) exp Mij ℓ K
and symmetrically for the reverse direction, L← NCE = −
B exp Mii← /τℓ 1 X , P log ← /τ B i=1 exp Mii← /τℓ + j∈H← (i) exp Mij ℓ
(80)
K
where τℓ > 0 is a temperature over likelihood scores. This objective instantiates the pairwise surrogate at the sequence level. For each conditioning representation, the matched EMA-teacher sequence is encouraged to outrank the most confusable mismatched sequences in the batch. The contrastive term therefore complements the positive s̄MTF objective: positive alignment makes paired views mutually predictable, while the in-batch contrast prevents unrelated examples from becoming equally likely under the conditional sequence model. Entropy regularization. To further discourage collapse and maintain a sufficiently rich predictive distribution, we retain an entropy regularizer on the student logits. Let ℓSb,t,v denote the student predictive logit for batch index b, time step t, and vocabulary item v, and let pSb,t,v = softmax(ℓSb,t,: )v .
(81)
We use the negative entropy term B
Lentropy =
T
V
1 XXX S pb,t,v log pSb,t,v . BT t=1 v=1
(82)
b=1
Since this quantity is the negative of the predictive entropy, minimizing it encourages higher-entropy token distributions. This counteracts pathological over-confidence and helps prevent the model from concentrating probability mass on a small set of generic tokens during self-alignment. Final Stage III objective. The final Stage III objective combines three complementary pressures: (i) cross-paired positive self-alignment using s̄MTF ; (ii) hardest-K in-batch contrast over teacher-forced likelihoods with uncorrupted prefixes; and (iii) entropy regularization on the student predictive distribution. The positive self-alignment loss is B
Lpos ≜ −
1 X Pii + Pii+ , B i=1
(83)
where Pii and Pii+ are the cross-paired positive scores defined using s̄MTF . This term makes the student assign high likelihood to the EMA-teacher sequence from the paired view in both directions, while retaining the Stage II bypass-control mechanisms on the student side. The full objective is ← LStageIII = Lpos + λNCE L→ NCE + LNCE + λentropy Lentropy ,
(84)
where λNCE and λentropy control the relative strengths of the in-batch contrastive term and the entropy term. The terms play distinct roles. The positive term transfers adaptive EMA-teacher tokenizations across paired views. The NCE term prevents many-to-one collapse by requiring the paired sequence to outrank the hardest mismatched teacher sequences under teacher-forced likelihoods whose prefixes are kept uncorrupted. The entropy term discourages premature concentration on a small set of generic tokens. Together, these components allow the symbolic interface to evolve under the EMA teacher while preserving input grounding, discriminability, and sequence-level diversity. 41
Summary of the progression from Stage II to Stage III. The training procedure increases expressiveness in a controlled manner. Stage II uses a frozen encoder and deterministic nearest-centroid VQ teacher to provide stationary token targets while training the decoder with bypass-resistant cross-paired teacher forcing. Stage III then replaces the fixed teacher with an EMA teacher over the full encoder–decoder model and jointly updates both modules. The key distinction in Stage III is the separation of target generation, positive alignment, and negative comparison. Teacher targets are generated by the EMA model in free-running autoregressive mode, where prefix corruption is not applicable. The EMA decoder is kept at full capacity: self-attention branch dropout is disabled, and the full conditional architecture remains active. Positive student alignment is evaluated using s̄MTF , so the student learns to predict paired teacher sequences under corrupted prefixes, encodersummary bias, and self-attention branch dropout. Negative comparison is performed with teacher-forced likelihoods whose prefixes are kept uncorrupted, so hard negatives reflect symbolic confusability rather than prefix-corruption effects. This progression preserves the stability of the deterministic Stage II bridge while allowing Stage III to refine the tokenization rule itself. The result is a self-alignment objective in which the EMA teacher supplies adaptive symbolic targets, the student is regularized against decoder bypass, and the in-batch contrast prevents collapse into input-insensitive or many-to-one tokenizations. 3.4.3
Teacher Target Generation via Top-p Sampling with Compound Repetition Penalty and Differential Stochasticity
During Stage III training, the EMA teacher generates target token sequences by free-running autoregressive decoding via stochastic top-p sampling (Holtzman et al., 2019). This procedure is used only for teachertarget generation during training. Its purpose is to expose controlled symbolic variability while suppressing degenerate loops, excessive token reuse, and uncontrolled over-generation. Final decoding after training is instead performed with beam search, as described in Section 3.4.4. Teacher generation is not teacher-forced likelihood evaluation: there is no ground-truth prefix supplied to the decoder and therefore no prefix-corruption operation to apply. The EMA teacher is kept at full capacity during generation: self-attention branch dropout is disabled, and all conditioning pathways remain active, including cross-attention to the teacher encoder states and the gated encoder-summary bias. (b)
Let L(l) ∈ RB×V denote the teacher logits at decoding step l, for batch size B and vocabulary size V . Let Tb<l denote the already generated prefix for batch item b. For each batch element, we compute token-frequency counts X freq(b) ⊮[τ = v], v ∈ {1, . . . , V }, (85) v = (b) b<l τ ∈T optionally over a fixed-length recent prefix window. Special symbols that should not be penalized, such as BOS, EOS, and PAD, are assigned zero frequency. Compound repetition penalty. To discourage repetitive decoding, we apply a sign-aware repetition penalty to the logits before top-p truncation. Given a repetition factor γl > 1 at step l, we define
(l)
ℓ̃b,v =
(b) ℓ(l) γ −freqv , b,v l
ℓb,v ≥ 0,
(l) freq(b) ℓb,v γl v ,
ℓb,v < 0.
(l)
(l)
(86)
Thus, repeated tokens with positive logits are damped, while repeated tokens with negative logits are pushed further down. The first occurrence of a token is unpenalized, and repeated occurrences are increasingly discouraged. This suppresses short loops while preserving stochasticity among the remaining candidate tokens. 42
Top-p filtering (Holtzman et al., 2019) and temperature scaling. penalty, we sort the adjusted logits in descending order:
After applying the repetition
(l) S(b) , I(b) = sort ℓ̃b,: .
(87)
The sorted logits are converted to probabilities, and the smallest prefix of tokens whose cumulative probability mass exceeds the top-p threshold is retained. All remaining logits are masked to −∞, while the top-1 token is always kept as a safety condition. Temperature scaling is then applied to the filtered logits: (b)
Sf
(b)
← Sf /τl ,
(b)
qf
(b)
= softmax Sf
,
(88)
and the next token is sampled from the resulting truncated distribution. Differential stochasticity for early decoding. The earliest autoregressive decisions are made with little or no token-side context, yet they strongly influence the rest of the decoded sequence. If the first few steps collapse to a narrow set of reusable prefixes, later decoding often remains trapped in the corresponding continuation patterns. We therefore use a step-dependent sampling schedule that allocates more stochasticity to the beginning of generation and sharper decoding to later positions. For the first Kearly decoding steps, we use higher temperature, larger top-p, and a weaker repetition penalty. For later steps, we switch to a sharper and more conservative regime: τearly > τlate ,
pearly > plate ,
γearly < γlate .
( (τearly , pearly , γearly ),
l ≤ Kearly ,
(τlate , plate , γlate ),
l > Kearly .
(89)
Equivalently, (τl , pl , γl ) =
(90)
This schedule encourages diverse high-level symbolic commitments at the start of decoding while keeping later continuations more stable. Length-constrained EOS handling. Target generation uses a sample-specific length cap Lcap = min Lmax − 1, max Lmin , ⌊ρ Tcond ⌋ .
(91)
Once l ≥ Lcap , EOS is forced by masking all other vocabulary items and assigning all probability mass to EOS. This ensures that every active sequence terminates by the sample-specific cap. Training-time teacher targets. For each teacher conditioning latent, the EMA model generates a target sequence by repeated application of repetition-penalized top-p sampling under the step-dependent schedule: Tbb = TopP-Sample Zeb ; {pl }, {τl }, {γl }, ρ , b = 1, . . . , B. (92) These sequences provide the adaptive EMA-teacher targets used by the Stage III self-alignment objective. 3.4.4
Inference-Time Beam Search with Compound Repetition Penalty
After training, final token sequences are decoded using beam search. This is intentionally different from the stochastic top-p sampling used by the EMA teacher during Stage III training. Sampling provides controlled target variability during self-alignment; beam search provides a more stable and reproducible tokenization at inference time. Inference decoding is free-running autoregressive generation. As in EMA-teacher generation, there is no teacher-forced prefix and therefore no prefix corruption to apply. The trained decoder is evaluated at full capacity: self-attention dropout is disabled, and the full conditional architecture remains active. 43
Algorithm 1 EMA-Teacher Target Generation via Top-p Sampling with Compound Repetition Penalty (b) Require: Teacher logits L ∈ RB×V at decoding step l, generated prefixes {Tb<l }B b=1 , step-dependent threshold pl , temperature τl , repetition factor γl (b) Ensure: Next sampled tokens (τ̂l )B b=1 (b) (b) 1: Compute frequency counts freq v from Tb<l 2: Apply repetition penalty: ( −freq(b) v
L[b, v] γl
L̃[b, v] ←
,
freq(b) L[b, v] γl v ,
L[b, v] ≥ 0,
L[b, v] < 0
3: Sort logits in descending order:
(Ssort , I) ← sort(L̃, descending along V ) 4: Convert sorted logits to probabilities and compute cumulative mass 5: Retain the smallest prefix whose cumulative probability exceeds pl 6: Ensure the top-1 token is always retained 7: Mask all remaining logits to −∞ 8: Apply temperature scaling:
Sf ← Sf /τl 9: Form the truncated sampling distribution:
qf ← softmax(Sf ) 10: Sample token indices k ∼ Multinomial(qf ) 11: Map sampled indices back to the original vocabulary: (b)
τ̂l
← I[b, k[b]]
(b) B )b=1
12: return (τ̂l
Beam-search state. Let Z ∈ RT ×D denote the conditioning latent for a single example, and let Lmax be the global decoding limit. At decoding step l, beam search maintains K partial hypotheses, o n (k) (k) k = 1, . . . , K , (93) Bl = T1:l , Sl (k)
(k)
where T1:l is the k-th partial sequence and Sl is its cumulative score. The beams are initialized from BOS, with the first beam assigned score 0 and all remaining beams initialized to −∞. At each step, every active beam is expanded by one token. Let (k)
(k)
ℓl,v = logitv DecAR (T<l , Z)
(94)
denote the decoder logit for vocabulary item v when extending beam k. Compound repetition penalty. To discourage repetitive inference-time sequences, we apply the same sign-aware repetition-control principle used during teacher generation. For beam k, let X (k) (k) cl,v = ⊮[τm = v] (95) m<l
be the number of occurrences of token v in the current beam prefix. The repetition-adjusted logits are (k) −cl,v (k) ℓ(k) γrep , ℓl,v ≥ 0, l,v (k) ℓ̃l,v = (96) (k) (k) ℓ(k) γ cl,v , ℓ < 0. rep l,v l,v 44
Special symbols such as BOS, EOS, and PAD are excluded from the penalty. This reduces pathological token reuse without requiring post-hoc sequence editing. EOS constraints and beam expansion. Inference uses the same sample-specific length cap: Lcap = min Lmax − 1, max Lmin , ⌊ρ Tcond ⌋ .
(97)
Before the cap is reached, active beams expand normally. Once l ≥ Lcap , EOS is forced by masking all other vocabulary items. Finished beams are thereafter constrained to emit PAD only. After applying the repetition penalty and EOS constraints, we compute log-probabilities (k)
(k)
log pl,v = log softmax ℓ̃l,:
v
.
(98)
Candidate continuation scores are (k,v) (k) (k) Sel+1 = Sl + log pl,v .
(99)
The top K candidates across all beam–token pairs (k, v) are retained to form Bl+1 . Optionally, beam scores may be length-normalized to reduce the bias toward short hypotheses: (k)
(k)
Sl
←
Sl , lαlen
(100)
where αlen = 0 recovers standard unnormalized beam search. Final sequence selection. Beam expansion proceeds until all beams terminate or the global limit Lmax is reached. The final decoded token sequence is the highest-scoring beam: Tb = arg
max T (k) ∈Bfinal
(k)
Sfinal .
(101)
After EOS appears, all subsequent positions are padded with PAD for batching consistency. Thus, inferencetime decoding is written as Tbb = BeamSearch(Zb ; K, ρ, γrep , αlen ) ,
b = 1, . . . , B.
(102)
Why beam search at inference time? Beam search provides a stable final symbolic representation after training has completed. Compared with stochastic sampling, it reduces output variance across repeated decodes, while the compound repetition penalty suppresses pathological loops without ad hoc post-processing. 3.5
Training with Continuous Audio Streams
Continuous-speech training as the common data regime. In practical deployment, an audio tokenizer must operate on continuous, unsegmented speech rather than on isolated lexical units. Accordingly, PairAlign is trained on randomly sampled 3-second speech excerpts from LibriSpeech (Panayotov et al., 2015). This choice is shared across the full framework: the model first learns a continuous representation, then learns to predict compact symbolic sequences, and finally refines the tokenization rule through EMAteacher self-alignment. What changes across stages is therefore not the underlying data regime, but the form of supervision applied to paired views sampled from that regime. This design is important for the objective in Section 3.1. The method is not trained with explicit wordlevel edit-distance supervision. Instead, it learns from content-preserving paired views and uses conditional sequence likelihood as a scalable surrogate for edit-distance preservation. Training directly on continuous speech therefore matches the intended use case: the tokenizer must assign stable, discriminative, and compact token sequences to ordinary speech windows without relying on pre-segmented lexical units. 45
Algorithm 2 Inference-Time Beam Search with Compound Repetition Penalty Require: Conditioning latent Z, BOS/EOS/PAD indices, beam size K, global max length Lmax , length-cap ratio ρ, minimum cap Lmin , repetition factor γrep , optional length-normalization exponent αlen Ensure: Final decoded sequence Tb 1: Compute sample-specific cap Lcap 2: Initialize beam set with BOS-only hypothesis; score first beam as 0, others as −∞ 3: for l = 1, . . . , Lmax − 1 do 4: for each active beam k do (k) 5: Compute next-step logits ℓl,: 6: Apply repetition penalty: (k) ℓ(k) γ −cl,v , ℓ(k) ≥ 0, rep (k) l,v l,v ℓ̃l,v ← (k) (k) cl,v (k) ℓl,v γrep , ℓl,v < 0 Disallow PAD for unfinished beams if l ≥ Lcap then Force EOS end if 11: if beam k already finished then 12: Force PAD 13: end if 14: Convert logits to log-probabilities and form candidate continuation scores 15: end for 16: Retain the top K candidates across all beam expansions 17: Update finished-beam indicators 18: Optionally apply length normalization to beam scores 19: end for 20: Select the highest-scoring final beam 21: Pad all positions after EOS with PAD 22: return Tb 7: 8: 9: 10:
Why 3-second contexts? A substantially shorter crop, such as an isolated word, often provides too little temporal context for autoregressive tokenization. Many LibriSpeech (Panayotov et al., 2015) words are short, frequently on the order of a few hundred milliseconds, and such clips can encourage the decoder to behave like a local acoustic classifier rather than a sequence model. They also provide limited evidence for learning dependencies among output tokens, length control, and termination behavior. By contrast, a 3-second excerpt contains richer phonetic, prosodic, and coarticulatory structure. It gives the encoder enough context to form informative conditioning representations and gives the autoregressive decoder a nontrivial sequence over which to model token dependencies. At the same time, the context is short enough to keep cross-attention, teacher-forced likelihood evaluation, EMA-teacher decoding, and inbatch negative comparison computationally practical. Thus, the 3-second setting provides a useful middle ground between isolated lexical crops and long utterance-level modeling. Speech corpus and segment sampling. ings. Let
We train on LibriSpeech (Panayotov et al., 2015) speech recorda ∼ DLS
(103)
denote a sampled recording. From this recording, we extract a fixed-duration segment x = Crop(a; Tctx ), 46
Tctx = 3 s.
(104)
At 16 kHz, this corresponds to 48,000 waveform samples. If the source utterance is longer than the target duration, the crop location is sampled during training; if it is shorter, the signal is padded to the required length. To avoid training on uninformative regions, crop selection uses a silence-aware preference. Candidate crops are scored using short-time RMS energy, and crops containing a sufficient fraction of active speech frames are preferred. This preserves the continuous-speech setting while reducing the probability that a training example is dominated by silence or padding. Positive-pair construction. Within this continuous-speech regime, positive pairs are constructed using a content-preserving corrupted-view relation: x+ i = Aug(xi ).
(105)
The positive view is intended to preserve the underlying spoken content of the anchor while changing nuisance acoustic factors. This produces paired examples that should remain close in the learned token space, even though their waveforms and acoustic features are not identical. Different utterances in the same minibatch provide natural negatives. Thus, for a minibatch of disjoint positive pairs B {(xi , x+ (106) i )}i=1 , the paired view of xi supplies the positive target, while token sequences associated with xj or x+ j , j ̸= i, serve as mismatched alternatives in the later likelihood-contrast objectives. Augmentation design. The augmentation pipeline is intentionally mild. Its role is to expose the model to nuisance variability, not to change the linguistic identity of the segment. We sample a small composition of speech-preserving transformations: Aug(x) = gm ◦ gm−1 ◦ · · · ◦ g1 (x),
(107)
where each gr is drawn from a fixed augmentation family and the composition length m is kept small. The augmentation family includes random gain perturbation, additive noise at controlled SNR, mild bandpass filtering, reverberation, and short time masking. After augmentation, the positive view is RMS-matched back to the anchor and peak-clipped to a safe range. This prevents the model from solving the paired-view task through trivial loudness cues or from being destabilized by large amplitude excursions. Overall, the augmentation is strong enough to create acoustic variation but mild enough to preserve the content that should determine the token sequence. Stage-dependent use of the same paired-view data. The same continuous-audio paired-view construction is used throughout PairAlign, but each stage uses it differently. In Stage I, the encoder is trained to produce robust frame-level representations. The positive view x+ i encourages invariance to content-preserving acoustic perturbations, while unrelated examples in the minibatch provide contrastive negatives. The output of this stage is a contextual encoder and a nearest-centroid VQ tokenizer that serve as the geometric baseline and as the deterministic teacher for the next stage. In Stage II, the encoder and vector quantizer are frozen. The same paired views are then used for deterministic cross-paired token prediction: the token sequence assigned to one view by the frozen VQ tokenizer is evaluated under the conditioning representation of the paired view. This introduces the autoregressive decoder and establishes a stable conditional sequence model before any model-generated targets are used. Because the teacher is fixed, this stage avoids target drift while the decoder learns the pairwise likelihood surrogate. In Stage III, the deterministic teacher is replaced by an EMA teacher over the full encoder–decoder model. The EMA teacher generates adaptive token sequences from each view, and the student learns to predict the paired view’s teacher sequence under the bypass-resistant masked teacher-forcing objective. In-batch 47
likelihood contrast then separates the paired sequence from confusable teacher-generated sequences belonging to unrelated examples. Thus, the same continuous-speech positive-pair relation supports adaptive selfalignment, while the symbolic targets are allowed to evolve with the model. Relation to edit-distance preservation. The continuous-audio paired-view setup provides the training signal needed for the surrogate objective. Since edit distance itself is not directly differentiable with respect to the tokenizer parameters, PairAlign does not optimize edit distance on generated strings inside the training loop. Instead, it encourages a differentiable proxy: the token sequence from one content-preserving view should be likely under the conditioning representation of the other view, and unlikely under unrelated conditioning representations. The use of continuous 3-second excerpts makes this surrogate operate at the same granularity used in evaluation, where token sequences are compared with normalized edit distance for consistency, retrieval, and local sweep analyses. Summary. Overall, PairAlign replaces explicit same-word supervision with a scalable continuous-speech paired-view formulation. Each training example is a realistic 3-second speech context; its positive view is a mild acoustic perturbation of the same content; and unrelated examples in the minibatch provide negatives. This common data regime supports all three stages: robust continuous representation learning, deterministic autoregressive token prediction, and adaptive EMA-teacher self-alignment. The result is a training setup aligned with the intended deployment setting: learning compact, discriminative, and edit-distance-aware symbolic sequences directly from continuous speech. 3.6
Extracting Timing Information from Autoregressive Decoding
Autoregressive tokenization produces a compact symbolic sequence rather than a frame-synchronous label stream. This is useful for compactness and sequence-level comparison, but it removes the native timing information available in frame-level geometric tokenizers. To recover approximate temporal grounding when needed, we exploit the cross-attention scores exposed by the conditional decoder DecAR (·). The procedure described in this section is applied at inference time only. It does not modify the training objective, does not constrain the decoder during learning, and does not backpropagate through the recovered alignments. Instead, it treats decoder–encoder cross-attention as a soft token-to-frame alignment signal and then converts this signal into a monotone segmentation using lightweight post-processing. This design is motivated by the broad observation that speech-to-symbol transduction is approximately monotonic in time. Prior work has used this property directly in model design, including local and monotonic attention for end-to-end speech recognition (Merboldt et al., 2019; Tjandra et al., 2017; Hou et al., 2017), hard monotonic attention with dynamic-programming marginalization (Wu & Cotterell, 2019), monotonic multi-head attention in Transformer-style sequence-to-sequence models (Ma et al., 2019), monotonic location attention for length generalization (Chowdhury & Caragea, 2023), and cross-attention biasing with Gaussian alignment masks for Speech Transformers (Zhao et al., 2020). Our use is more conservative: we do not impose a monotonic attention mechanism during training, but use the monotonicity assumption only to interpret and regularize the attention map after decoding. Cross-attention extraction and slicing.
Given an input window x with encoder latents Z ∈ Rd×Tenc ,
(108)
we decode a token sequence with special symbols, T = [BOS, τ1 , . . . , τL , EOS, PAD, . . . ].
(109)
During decoding, we also request the decoder cross-attention tensor from a selected decoder block, or from an aggregation across multiple blocks: Araw ∈ RH×Tdec ×Tenc , (110) where H is the number of attention heads and Tdec is the decoder length including special tokens. 48
We then remove positions that do not correspond to ordinary decoded tokens: (i) the BOS position, (ii) the EOS position and all padding positions after EOS, and (iii) padded encoder frames. Let peos denote the first EOS index in T . The valid token span is therefore l ∈ {1, . . . , L}, with L = peos − 1. Let menc ∈ {0, 1}Tenc
(111)
be a mask indicating valid non-padded encoder frames. After optional head averaging, layer averaging, and encoder masking, we obtain a token-to-frame attention matrix W ∈ RL×T ,
Wl,t ≥ 0,
T X
Wl,t = 1,
(112)
t=1
P where T = t menc is the number of valid encoder frames. Here Wl,t is interpreted as the attention mass t from decoded token position l to encoder frame t. Light monotonicity bias via a 2D Beta prior. Raw cross-attention can be noisy. Multi-head attention may distribute probability mass over several regions, and different decoder layers may mix local and global information. In speech, however, the correspondence between the input signal and the decoded symbolic sequence is expected to be approximately monotone. We therefore apply a light diagonal prior before extracting timestamps. This prior is inspired by the 2D beta-binomial attention prior used by Neekhara et al. (Neekhara et al., 2024) for improving robustness in LLM-based speech synthesis. In their setting, the prior is used during training to guide cross-attention between text tokens and generated speech tokens toward a monotonic alignment. They show that cross-attention heads in encoder–decoder speech LLMs can implicitly learn text–speech alignment, and that guiding these heads with an attention prior and an alignment loss improves robustness to missing, repeating, and misaligned speech. Our use is deliberately more conservative: we do not use the prior during training, do not add an alignment loss, and do not modify the learned cross-attention mechanism. Instead, we use a 2D Beta-shaped prior only at inference time as a post-hoc denoising bias before monotone Viterbi alignment. Let
t−1 ˜l = l − 1 , , (113) T −1 L−1 for L > 1 and T > 1, where t indexes valid encoder frames and l indexes decoded token positions. We define a row-wise Beta-shaped prior over encoder frames: t̃ =
αl = 1 + a ˜l,
Pl,t ∝ t̃ αl −1 (1 − t̃) βl −1 ,
βl = 1 + b (1 − ˜l),
(114)
followed by row normalization, T X
Pl,t = 1.
(115)
t=1
This produces a near-diagonal prior: earlier decoded tokens are biased toward earlier encoder frames, and later decoded tokens are biased toward later encoder frames. The shape parameters a and b control the sharpness and asymmetry of the prior. We apply the prior multiplicatively: fl,t = P W T
ω Wl,t Pl,t
ω ′ t′ =1 Wl,t Pl,t′ + ε
,
(116)
where ω ∈ [0, 1] controls the strength of the prior and ε > 0 is a numerical stabilizer. We use a small value of ω, so the prior acts only as a gentle regularizer on the extracted attention map rather than imposing a hard alignment constraint. This connects our method to prior attention-guidance approaches, but with a different role. The betabinomial prior in Neekhara et al. (Neekhara et al., 2024) is a training-time mechanism for encouraging 49
robust monotonic cross-attention in speech synthesis. Gaussian or local attention-biasing methods similarly modify attention during model computation to encourage locality or monotonicity (Merboldt et al., 2019; Zhao et al., 2020). By contrast, Eq. 116 is used only after decoding. It does not change the tokenizer, but makes the timestamp extraction step more stable by softly favoring the monotone structure expected in speech. f is token-normalized: each row describes where one Frame-to-token posterior. The smoothed matrix W decoded token attends in the encoder sequence. For timing, we need the inverse perspective: for each encoder frame, which token best explains it? We therefore form a frame-to-token posterior: At,l ≜ PL
fl,t W , fl′ ,t + ε W
A ∈ RT ×L .
(117)
l′ =1
This gives L X
At,l = 1
(118)
l=1
for each encoder frame t. Intuitively, At,l is the posterior probability that encoder frame t belongs to decoded token position l, after attention smoothing and renormalization. Monotone Viterbi alignment. a monotone path through A. Let
To obtain a contiguous segmentation of the encoder timeline, we decode π ∈ {0, . . . , L − 1}T
(119)
be a path assigning each encoder frame to a decoded token index. We constrain the path to be monotone: πt ∈ {πt−1 , πt−1 + 1},
(120)
πT = L − 1.
(121)
with boundary conditions π1 = 0,
The maximum-a-posteriori path is obtained by dynamic programming: π ∗ = arg max
π∈M
T X
log At,πt +1 + ε ,
(122)
t=1
where M denotes the set of valid monotone paths. This step plays a role similar in spirit to monotonic alignment inference in hard-attention and transduction models (Wu & Cotterell, 2019), but it is used here only as a decoding-time post-processor. The model itself is not trained by marginalizing over monotone alignments; the dynamic program simply converts the already decoded cross-attention map into a single monotone token-to-frame assignment. Finally, let the decoded token IDs excluding BOS/EOS/PAD be ul ∈ A.
(123)
t = 1, . . . , T,
(124)
u = (u1 , . . . , uL ), The per-frame token identity is ybt = uπt∗ +1 ,
where the +1 converts the zero-indexed path state into the one-indexed token position. Recovering token-level timestamps. Given the monotone frame assignment π ∗ , each decoded token position l induces a support set of encoder frames: Il = { t ∈ {1, . . . , T } | πt∗ = l − 1 }. 50
(125)
Let the input window begin at absolute time twin-start . Let the encoder frame duration be ∆t =
M , Fs
(126)
where Fs is the sampling rate and M is the encoder downsampling factor. We associate encoder frame t with center time 1 ∆t. (127) time(t) = twin-start + t − 2 The raw start and end times for token l are tstart = min time(t), l
tend = max time(t). l
t∈Il
t∈Il
(128)
We then expand by half a frame so that the timestamp interval covers the occupied encoder support: t̃start = tstart − l l
∆t , 2
t̃end = tend + l l
∆t . 2
(129)
This yields the time-stamped token sequence n
ul , t̃start , t̃end l l
oL
.
(130)
l=1
Interpretation. Equations 116–130 define a simple post-hoc timing recovery procedure. The decoder cross-attention provides a soft token-to-frame association, the 2D Beta prior introduces a weak diagonal preference, and Viterbi decoding extracts a globally monotone segmentation. The resulting timestamps are therefore consistent with both the encoder temporal grid and the decoded token order. The method is intentionally lightweight. Unlike CTC, RNN-T, RNA, hard monotonic attention, monotonic chunkwise attention, or monotonic multi-head attention (Graves et al., 2006; Graves, 2012; Raffel et al., 2017; Chiu & Raffel, 2017a; Ma et al., 2019; Wu & Cotterell, 2019; Sak et al., 2017), it does not introduce an additional alignment latent variable during training. Unlike cross-attention biasing methods for Speech Transformers (Zhao et al., 2020), or training-time beta-binomial attention priors for LLM-based TTS (Neekhara et al., 2024), it does not alter the attention computation used by the model. It only interprets the cross-attention produced by the trained autoregressive tokenizer. Dependence on emergent monotonicity. The reliability of this timing-recovery procedure depends on whether decoder cross-attention contains a meaningful near-monotone correspondence between decoded token positions and encoder frames. When the attention map is locally concentrated and roughly diagonal, the recovered token boundaries are usually temporally coherent. When attention is diffuse, multi-modal, or strongly non-monotonic, the inferred timestamps become less reliable. The post-hoc algorithm cannot create alignment structure that is absent from the attention map; it can only denoise and summarize structure that has already emerged. This limitation is consistent with prior work showing that monotonicity is a useful inductive bias in speech and sequence transduction (Merboldt et al., 2019; Wu & Cotterell, 2019; Ma et al., 2019; Zhao et al., 2020; Neekhara et al., 2024). It also suggests a natural extension of PairAlign. Future versions could replace unconstrained cross-attention with a monotonic, local, chunkwise, location-aware, or attention-priorguided cross-attention mechanism, so that token-to-frame correspondence is encouraged by the architecture or training objective itself rather than recovered after decoding. Such designs may improve timestamp reliability, particularly for segmentation, localization-sensitive retrieval, and streaming applications. Scope. The timestamp extraction procedure is not required for the main PairAlign tokenization objective. For retrieval or symbolic comparison, the compact decoded token sequence can be used directly. Timing recovery is needed only when downstream applications require temporal grounding, such as segmentation, alignment visualization, token-boundary analysis, or time-stamped retrieval. Thus, PairAlign remains a compact autoregressive tokenizer by default, with post-hoc temporal grounding available when required. 51
4
Experiments
Our experimental protocol evaluates PairAlign in the same operating regime used by the method: tokenization of continuous 3-second speech segments. Rather than treating speech as isolated word crops, we ask whether the learned autoregressive tokenizer defines a compact and stable symbolic interface over realistic speech context. The experiments are organized around three questions. First, we test discrete token consistency: do two content-preserving acoustic realizations of the same 3-second segment produce compatible token sequences? Second, we test token-space retrieval: can the learned symbolic sequence support retrieval over a continuousspeech archive using normalized edit distance? Third, we test local sequence evolution: when a 3-second window is moved through speech in small 100 ms steps, does the decoded token sequence change in a controlled way, or does the tokenizer behave like an unstable holistic segment-level code? These three evaluations correspond directly to the methodology. PairAlign is trained to approximate editdistance preservation through cross-paired conditional likelihood rather than by directly optimizing edit distance. The experiments therefore evaluate both intrinsic sequence agreement and downstream edit-distance retrieval. Because the method is designed to move beyond frame-synchronous geometric tokenization, we compare PairAlign primarily against its own Stage I geometric tokenizer: the same encoder followed by nearest-centroid vector quantization. This baseline is important because it isolates the contribution of the autoregressive decoder, pairwise self-alignment, and EMA-teacher refinement from the effect of the acoustic front end. Evaluation domains. Models are trained on LibriSpeech (Panayotov et al., 2015) and evaluated on both LibriSpeech (Panayotov et al., 2015) and TIMIT (Garofolo et al., 1993). LibriSpeech (Panayotov et al., 2015)100 is used to evaluate behavior within the same broad corpus family as training, while TIMIT (Garofolo et al., 1993) is used as a cross-corpus test of whether the learned tokenization transfers to a different speech corpus. This distinction is important: a tokenizer that is merely tuned to LibriSpeech (Panayotov et al., 2015) statistics may show good in-domain consistency but fail to preserve useful sequence structure under a domain shift. What is measured. Across the experiments, we report both sequence-overlap and compactness statistics. For consistency and continuous-sweep analyses, we use unigram Jaccard similarity and normalized edit similarity. Jaccard similarity measures whether the two sequences reuse similar token identities, while edit similarity additionally penalizes substitutions, insertions, deletions, ordering changes, and length mismatch. Since PairAlign produces shorter autoregressive sequences than the geometric tokenizer, we also report sequence length, token rate, archive size, bitrate, and adjacent length change |∆L|. These quantities are essential for interpreting the results: a denser tokenizer can obtain higher overlap simply because many local frame-derived assignments persist, whereas a compact tokenizer may change fewer tokens in absolute terms but incur a larger normalized similarity penalty. In addition, we analyze the token inventory induced by each tokenizer. This includes global vocabulary usage, per-position token entropy, effective vocabulary size, dead-token rate, token-frequency concentration, and position-wise variation in token usage across decoding steps. These statistics test whether PairAlign uses the discrete alphabet broadly or whether its compactness is obtained by shrinking the active token inventory. They also reveal how token uncertainty and token diversity evolve across the decoded sequence. 4.1
Datasets and Preprocessing
Training data. PairAlign is trained on continuous speech excerpts from LibriSpeech (Panayotov et al., 2015). Following the continuous-audio training setup in Section 3.5, each training example is a fixed 3-second segment rather than an isolated lexical crop. Audio is converted to mono when needed and resampled to 16 kHz. Thus, each training segment contains 48,000 waveform samples. Unless otherwise stated, models are trained on LibriSpeech (Panayotov et al., 2015) training splits and selected using held-out LibriSpeech (Panayotov et al., 2015) validation data. The reported analyses use 52
LibriSpeech (Panayotov et al., 2015)-100 for in-domain evaluation and TIMIT (Garofolo et al., 1993) for cross-corpus evaluation. Evaluation data.
We use two evaluation regimes.
For discrete token consistency, we evaluate on LibriSpeech (Panayotov et al., 2015)-100 and TIMIT (Garofolo et al., 1993). For each 3-second segment, we construct an augmented positive view using the same mild content-preserving augmentation family used during training. The anchor and positive view are tokenized independently and compared in token space. For retrieval and continuous-sweep analysis, we use TIMIT (Garofolo et al., 1993) as the main crosscorpus testbed. The retrieval archive is built from overlapping 3-second TIMIT (Garofolo et al., 1993) segments with a 1.5-second hop, yielding 9,461 archive segments. The continuous-sweep analysis uses a finer 100 ms hop on TIMIT (Garofolo et al., 1993), producing 11,137 adjacent window pairs. These two hop sizes test different temporal scales: 1.5 seconds evaluates retrieval over substantially shifted continuous-speech segments, while 100 ms probes local token-sequence evolution under small phone-scale context changes. Segment extraction. Training and evaluation segments are extracted as fixed-duration 3-second windows. When a source utterance is longer than 3 seconds, a crop is selected from the utterance; when it is shorter, the utterance is padded to the target duration. During training, cropping is stochastic so that the model sees varied local contexts across epochs. For validation and reported evaluations, segment construction is deterministic where needed so that comparisons are reproducible. To avoid uninformative crops dominated by silence, training segments are sampled with a silence-aware preference. Candidate crops are scored using short-time RMS energy, and crops with sufficient active speech are preferred. This keeps the training distribution focused on speech-bearing regions without requiring explicit word-level segmentation. Positive-view augmentation. augmentations:
Positive pairs are constructed using mild, content-preserving waveform x+ = Aug(x).
(131)
The augmentation pipeline is intentionally conservative. It includes small gain perturbations, additive noise at controlled SNR, mild filtering, reverberation, and occasional short time masking. At most a small number of transformations are composed for a given example, and the augmented view is RMS-matched back to the anchor and clipped to a safe amplitude range. The purpose of augmentation is to change nuisance acoustic factors while preserving the underlying speech content. This makes the consistency experiment a direct test of whether the tokenizer is stable to contentpreserving acoustic variation. Feature extraction. The main experiments use log-Mel acoustic features derived from the 16 kHz waveform. Each 3-second segment is converted to an 80-bin log-Mel spectrogram using a 25 ms analysis window and a 10 ms hop. The same feature extraction is applied to anchor, positive, query, and archive segments, ensuring that differences in tokenization are due to the learned model rather than inconsistent preprocessing. Tokenization protocols. The Stage I geometric tokenizer maps encoder frames to nearest VQ centroids. Before computing sequence-level metrics, we apply consecutive-token deduplication to the geometric output. This makes the baseline a compact symbolic sequence rather than a raw frame-synchronous stream. PairAlign is evaluated using its decoded autoregressive token sequence. At inference time, decoding is performed with the trained model in free-running mode, without teacher forcing or prefix corruption. Thus, all reported PairAlign results reflect the actual tokenization behavior used at test time. Alignment metadata for retrieval relevance. For retrieval analysis, forced-alignment metadata is used only to define or interpret relevance. The retrieval model itself never uses word or phoneme labels. Tokenization and ranking are performed entirely in token space using normalized edit distance. The alignments 53
allow us to distinguish segment-overlap relevance, phoneme-exact relevance, and phoneme-relaxed relevance in the TIMIT (Garofolo et al., 1993) retrieval experiment. 4.2
Evaluation Overview
Discrete token consistency. We first compare token sequences produced from an anchor segment and its augmented positive view. This evaluates whether the learned tokenizer maps content-preserving acoustic variants to compatible symbolic strings. We report unigram Jaccard similarity, normalized edit similarity, activated vocabulary size, and collapse rate. Cross-corpus retrieval. We then evaluate retrieval on TIMIT (Garofolo et al., 1993) using a 3-second archive with a 1.5-second hop. Each query is an augmented view of an archive segment and is ranked against the full archive using normalized token edit distance. We report Recall@K, MRR, and first-relevant-rank statistics under segment-overlap, phoneme-exact, and phoneme-relaxed relevance definitions. We also report archive compactness, including total token count, average tokens per segment, token rate, and serialized archive size. Continuous-sweep analysis. Finally, we evaluate how token sequences evolve as a 3-second window is swept through TIMIT (Garofolo et al., 1993) with a 100 ms hop. Adjacent windows overlap by 2.9 seconds and differ only in the boundary region entering and leaving the window. For each adjacent pair, we compute normalized edit similarity, unigram Jaccard similarity, and adjacent length change |∆L|. This analysis probes whether the learned tokenization changes coherently under small local shifts in acoustic context. Interpretation. These experiments should be read together. Consistency measures whether tokenization is stable under acoustic nuisance variation. Retrieval measures whether the resulting token space is useful for search over continuous speech. Continuous sweep measures whether token sequences evolve smoothly under fine-grained temporal shifts. The comparison with the deduplicated geometric tokenizer reveals the main trade-off studied in this work: PairAlign produces substantially more compact autoregressive token sequences, while aiming to preserve enough ordered sequence structure for edit-distance comparison and retrieval. 4.3
Discrete Token Consistency
Our first set of experiments evaluates whether the learned tokenizers assign compatible symbolic sequences to acoustically different realizations of the same underlying 3-second speech segment. This is the most direct intrinsic test of whether the learned symbolic representation is stable under nuisance acoustic variation. For each anchor segment xi , we construct a positive view x+ i = Aug(xi ), where Aug(·) is the mild content-preserving augmentation pipeline described in Section 3.5. The positive view is therefore a noisy or acoustically perturbed version of the anchor, while preserving the same underlying spoken content. Both segments are independently tokenized, producing Ti
and
Ti+ .
This experiment is therefore also a direct test of noise-robust tokenization: a useful symbolic interface should map an anchor segment and its noisy content-preserving counterpart to compatible token strings. We evaluate three tokenizers. Stage I Geometric is the base Mamba encoder followed by nearest-centroid VQ and consecutive-token deduplication. Stage I+ Geometric is the same geometric tokenizer after the wav2tokstyle sequence-level strengthening term; it remains a frame-synchronous geometric tokenizer, but receives an explicit order-sensitive CTC-style pairwise constraint. PairAlign is the full autoregressive self-aligned tokenizer, evaluated by free decoding without teacher forcing or prefix corruption. 54
This comparison is diagnostic because all three systems share the same overall front-end lineage, but differ in where sequence structure is imposed. Stage I learns local geometric assignments. Stage I+ adds sequencelevel pressure while keeping token identity tied to frame-level posteriors. PairAlign moves the sequence itself into the learned object: token identity, order, length, and EOS placement are produced by an inputconditioned autoregressive decoder trained through cross-view self-alignment. Unigram Jaccard similarity. Given a token sequence T , we form the corresponding unigram set, G (1) (T ), by discarding repeated occurrences of the same token. For a positive pair (xi , x+ i ), we compute Jaccard(xi , x+ i )=
G (1) (Ti ) ∩ G (1) (Ti+ ) . G (1) (Ti ) ∪ G (1) (Ti+ )
(132)
This metric measures unordered token-set overlap between the anchor and noisy positive view. It captures whether the tokenizer reuses similar symbolic units for the same underlying content, but it ignores token order and repeated structure. Edit-distance-based similarity. Because PairAlign is explicitly motivated by sequence-level preservation under edit distance, we also compute normalized edit similarity between paired token sequences. Let ED(Ti , Ti+ ) denote the Levenshtein edit distance. We define Simedit (Ti , Ti+ ) = 1 −
ED(Ti , Ti+ ) . max |Ti |, |Ti+ |
(133)
This metric complements Jaccard similarity by explicitly penalizing substitutions, insertions, deletions, ordering changes, and length mismatch. It is therefore the primary metric for ordered token-sequence consistency. Exact match rate.
We also report the strict exact-match rate: ExactMatch =
N 1 X ⊮ Ti = Ti+ . N i=1
(134)
This measures the fraction of anchor–positive pairs for which the tokenizer assigns exactly the same token string to the original segment and its noisy content-preserving view. Unlike collapse, high exact match is desirable here because the two inputs represent the same underlying speech content. Mean sequence length.
We report the mean sequence length L=
N 1 X |Ti | + |Ti+ | . 2N i=1
(135)
This measures symbolic compactness. Since edit-distance comparison and sequence storage both scale with length, a tokenizer that preserves consistency while emitting fewer tokens provides a more compact symbolic interface. Edit-operation decomposition. To understand how the anchor and noisy positive tokenizations differ, we also decompose the optimal Levenshtein edit script into substitutions, insertions, and deletions. For each pair, let Si , Ii , Di denote the number of substitutions, insertions, and deletions in an optimal edit alignment between Ti and Ti+ . The total edit distance is ED(Ti , Ti+ ) = Si + Ii + Di . (136) This decomposition separates symbolic relabeling from token birth/death. A high substitution count indicates that the tokenizer preserves length but changes token identities, while high insertion or deletion counts indicate changes in token allocation or segmentation. 55
Why this experiment is necessary. Discrete token consistency is a basic requirement for any symbolic interface over speech. If small acoustic perturbations induce large symbolic changes, then downstream operations such as alignment, memory, retrieval, indexing, or reasoning in token space become unreliable. Because PairAlign is trained through pairwise self-alignment, consistency under noisy content-preserving augmentation provides one of the most direct tests of whether the learning signal has transferred into a stable tokenization function. 4.4
Token Inventory and Collapse Analysis
Beyond pairwise consistency, we analyze the statistical structure of the token inventory induced by each tokenizer. This analysis is necessary because high consistency or compactness can be misleading if it is obtained through pathological collapse. A tokenizer that uses only a small subset of tokens, or that maps many unrelated examples to identical strings, may appear stable under some pairwise metrics while failing as a rich symbolic interface. We therefore examine not only how similar paired sequences are, but also how the token alphabet is used globally, how token usage varies across decoding positions, and whether the tokenizer exhibits either lowdiversity or exact many-to-one collapse. Global token-frequency distribution. Let Deval denote an evaluation set and let Ti = [τi,1 , . . . , τi,Li ] be the token sequence assigned to example i. For each token a ∈ A, we compute its empirical frequency P PLi ⊮[τi,l = a] P . p(a) = i l=1 i Li From this distribution, we compute the global token entropy X Hglobal = − p(a) log p(a),
(137)
(138)
a∈A
and the normalized entropy e global = Hglobal . H log |A|
(139)
The normalized entropy lies in [0, 1], with larger values indicating more uniform use of the vocabulary. We also report the effective vocabulary size Veff = exp(Hglobal ),
(140)
which can be interpreted as the number of equally frequent tokens that would produce the same entropy. Activated and dead vocabulary.
We report the number of activated tokens Vactive = |{a ∈ A : p(a) > 0}| ,
as well as the dead-token rate rdead = 1 −
Vactive . |A|
(141)
(142)
This distinguishes broad vocabulary usage from collapse into a small set of tokens. We also report the cumulative frequency mass of the top q tokens, X Mq = p(a), (143) a∈Topq
for q ∈ {10, 25, 50}. A high top-q mass indicates that the sequence stream is dominated by a small number of symbols even if many tokens are technically active. 56
Position-wise token usage. Because PairAlign generates tokens autoregressively, it is important to examine how token usage varies across decoding positions. For an absolute output position l, let Il = {i : Li ≥ l} denote examples whose decoded sequence is at least length l. We define the position-specific token distribution P ⊮[τi,l = a] pl (a) = i∈Il . (144) |Il | The corresponding position-wise entropy is X Hl = − pl (a) log pl (a), (145) a∈A
with normalized form
Hl . log |A| We also compute the number of distinct tokens used at position l: el = H
Vl = |{a ∈ A : pl (a) > 0}| .
(146)
(147)
e l , and Vl show whether different decoding steps use different parts of the token inventory. For The curves Hl , H PairAlign, this analysis is especially informative because early decoding steps begin with little autoregressive context and must be strongly grounded in the acoustic condition. If the decoder starts many examples with the same small set of generic tokens, this would appear as low early-position entropy and low Vl . Conversely, broad early-position usage indicates that the initial symbolic trajectory is input-dependent rather than dominated by a generic prefix. Relative-position analysis. Absolute position statistics can be biased by sequence length, especially when comparing PairAlign with the geometric tokenizers. We therefore also compute relative-position token statistics. Each token position l in sequence i is assigned to a normalized position bin l−1 b(l, i) = Bpos , (148) Li where Bpos is the number of relative-position bins. For each bin b, we compute the token distribution pb (a), e b , and active token count Vb . This allows us to compare whether the entropy Hb , normalized entropy H beginning, middle, and end of the decoded sequence use the token inventory differently, independent of absolute sequence length. Bigram and transition statistics. To test whether the token inventory is used only as an unordered set or as a structured sequence, we also compute transition statistics. For adjacent token pairs, define the empirical bigram distribution P PLi −1 ⊮[τi,l = a, τi,l+1 = b] . (149) p(a, b) = i l=1 P i (Li − 1) We report bigram entropy, Hbigram = −
X
p(a, b) log p(a, b),
(150)
X
(151)
a,b
and conditional next-token entropy, Hnext = −
X a
p(a)
p(b | a) log p(b | a).
b
These statistics measure whether the tokenizer produces a rich sequence process rather than independent token draws or repetitive loops. A low conditional entropy may indicate strong local continuation regularities, whereas extremely low values together with high repetition would suggest degenerate decoding. 57
Low-diversity sequence collapse. We first evaluate collapse at the level of individual decoded sequences. For a sequence T , define its unique-token ratio as runiq (T ) =
|unique(T )| . |T |
(152)
A sequence is marked as low-diversity collapsed when runiq (T ) ≤ 0.2.
(153)
This criterion detects repetitive or internally degenerate outputs, where a sequence is dominated by a small number of token identities. For anchor and positive streams separately, we report the fraction of sequences that satisfy this low-diversity criterion. We also report the Collapsed Pair Rate, defined as the fraction of anchor–positive pairs in which either the anchor sequence or the positive sequence is low-diversity collapsed: N 1 X ⊮ runiq (Ti ) ≤ 0.2 or runiq (Ti+ ) ≤ 0.2 . N i=1
(154)
This statistic is different from anchor–positive exact match. Exact matching between an anchor and its noisy positive view is desirable because both views represent the same content. Collapsed Pair Rate is undesirable because it indicates that at least one sequence in the pair is internally low-diversity. Within-stream exact-collision collapse. We also evaluate a stronger many-to-one collapse criterion based on exact string collisions across different examples. For the anchor stream, we check whether any anchor token sequence exactly matches another anchor sequence from a different example: N
Collisionanchor =
1 X ⊮ [∃j ̸= i : Ti = Tj ] . N i=1
(155)
The positive-stream collision rate is defined analogously: Collisionpositive =
N 1 X ⊮ ∃j ̸= i : Ti+ = Tj+ . N i=1
(156)
This full-collapse diagnostic distinguishes desirable robustness to noise from undesirable many-to-one assignment across unrelated examples. A high anchor–positive exact-match rate is good; a high within-stream exact-collision rate would indicate that distinct underlying segments are being mapped to the same token string. Interpretation. The token inventory and collapse analyses complement the consistency and retrieval results. If PairAlign improves edit similarity while maintaining broad active vocabulary, high effective vocabulary size, nontrivial position-wise entropy, diverse transition statistics, low low-diversity collapse, and low within-stream exact collision, then the improvement cannot be attributed to trivial collapse. Instead, it indicates that the self-alignment objective reorganizes the token space into a compact but still diverse symbolic sequence interface. By contrast, if gains were accompanied by low entropy, high dead-token rate, high low-diversity collapse, or many exact collisions across unrelated examples, this would suggest that the model is using a narrowed token inventory or generic sequence strategy. 4.5
Long-Form Segment Retrieval on Continuous Audio
To evaluate whether the learned token space is useful beyond pairwise consistency, we perform a retrieval experiment over a large continuous-speech archive. The goal is to test whether token similarity can retrieve linguistically related 3-second speech segments from long-form audio under realistic acoustic variation. This setting is well matched to the regime studied throughout the paper, since both training and evaluation operate on continuous 3-second excerpts rather than on isolated lexical units. 58
Archive construction. We concatenate the full train-clean-100 portion of LibriSpeech (Panayotov et al., 2015) into a long-form searchable archive and segment it using the same sliding-window procedure employed elsewhere in our continuous-audio setup. This yields a collection of overlapping 3-second archive N segments {xn }N n=1 , together with their tokenizations {Tn }n=1 . For analysis purposes, each archive segment is also associated with the word and phoneme content that falls within its temporal extent, obtained from forced alignments. These alignments are used only to define and interpret retrieval relevance; they are not used by the retrieval mechanism itself. Query construction. mented views
To construct queries, we sample a subset of archive segments and generate augqi = Aug(xi ),
(157)
using the same speech-preserving augmentation pipeline as in the intrinsic consistency experiments. Each query is independently tokenized to obtain Tiq , and is then compared against the tokenizations of all archive segments {T1 , . . . , TN }. (158) Retrieval is therefore performed entirely in token space. Token-space ranking. For each query qi , we compute similarity between its token sequence Tiq and every archive token sequence Tn . Similarity may be instantiated using unigram Jaccard, bigram Jaccard, or editbased similarity, consistent with the sequence-comparison measures used throughout the paper. The archive is then ranked from most similar to least similar with respect to the query. Token rate, bitrate, and archive compression. In addition to retrieval accuracy, we measure the compactness of the symbolic archive induced by each tokenizer. For each archive segment xn , let Ln = |Tn | denote the number of emitted tokens. We report the total archive token count Ntok =
N X
Ln ,
(159)
1 X Ln , N n=1
(160)
L , Tctx
(161)
n=1
the average number of tokens per segment, N
L= and the token rate
Rtok =
where Tctx = 3 s in our experiments. Because both the geometric tokenizer and PairAlign use the same vocabulary size |A| = 512, each token can be represented using btok = ⌈log2 |A|⌉ = 9
(162)
bits under a fixed-length code. The corresponding symbolic bitrate is therefore Rbit = Rtok btok .
(163)
We also report the token-count compression ratio between the geometric tokenizer and PairAlign: Ctok =
geo Ntok pa , Ntok
and the relative token reduction rred = 1 −
59
pa Ntok geo . Ntok
(164)
(165)
These quantities are not intended to define an audio-codec bitrate, since the tokens are used here for retrieval and symbolic comparison rather than waveform reconstruction. Instead, they measure the storage and comparison cost of the symbolic archive. This is important because edit-distance retrieval scales with sequence length: a tokenizer that preserves retrieval performance while reducing token rate also reduces the cost of storing and comparing archive entries. Alignment-defined relevance. A key feature of this setup is that relevance is not restricted to exact segment identity. Because word and phoneme alignments are available for the archive, we can define relevance in linguistic terms by examining the content contained within each 3-second segment. Concretely, an archive segment may be treated as relevant if its aligned word content, or alternatively its aligned phoneme content, matches or substantially overlaps that of the source segment from which the query was derived. As a result, two segments may be considered relevant even when they are temporally distant in the archive, provided that they contain sufficiently similar linguistic material. This yields a more meaningful retrieval criterion than simple temporal overlap or exact-segment recovery. Evaluation protocol. For each query, we rank all archive segments according to token-space similarity and evaluate whether linguistically relevant segments are retrieved near the top of the list. We report standard ranking measures such as Recall@K, Mean Reciprocal Rank (MRR), and the rank of the first relevant segment. We also report compactness statistics, including total archive token count, average tokens per segment, token rate, symbolic bitrate, compression ratio, and relative token reduction. When useful, relevance can be evaluated at two levels: (i) a stricter segment-level criterion, where the source segment and its immediate overlapping neighbors are considered relevant, and (ii) a broader content-level criterion, where any archive segment with substantially matching aligned word or phoneme content is treated as relevant. The second criterion is especially informative in continuous speech, because it tests whether the learned token space retrieves linguistically related content rather than merely the original temporal window. Why this experiment matters. This experiment directly tests whether the learned symbolic space is useful for search over continuous speech at the same 3-second segment granularity used throughout the paper. A successful tokenizer must do more than keep augmented views of the same segment close: it should also organize token space so that linguistically related regions of a large archive become retrievable through token similarity alone. At the same time, retrieval should be interpreted together with symbolic compactness. Since edit-distance comparison scales with sequence length, reducing the number of stored tokens can lower both archive size and comparison cost. Thus, the experiment evaluates not only whether PairAlign preserves meaningful retrieval structure, but also whether it does so with fewer tokens and a lower symbolic bitrate. Because relevance is defined independently from the tokenization itself using forced alignments, this experiment provides a principled test of whether the learned sequences preserve linguistically meaningful structure under realistic acoustic variability. 4.6
Compositionality Analysis
Beyond stability and retrieval utility, we examine whether the learned token sequences exhibit locally structured behavior under small changes in acoustic context. In this work, we use a single compositionalityoriented probe: continuous-sweep tokenization analysis. The goal is not to establish compositionality in a formal sense, but to test whether controlled local changes in the input window lead to coherent changes in the decoded token sequence, rather than arbitrary global re-tokenization. 4.6.1
Continuous-Sweep Tokenization Analysis
A learned symbolic tokenizer should not behave as a purely holistic segment-level code. If two input windows share nearly all of their acoustic content, then their token sequences should retain a measurable relationship, even if the small boundary region entering or leaving the window contains linguistically salient material. The continuous-sweep analysis probes this behavior by shifting a fixed-duration window through speech in small steps and comparing the tokenizations of adjacent windows. 60
Importantly, stability under such a sweep is not only a question of sequence length. A tokenizer may preserve length while changing many token identities, or it may change length through a small number of localized insertions and deletions. We therefore analyze adjacent-window behavior not only through overlap scores and length deviation, but also through the edit operations required to transform one token sequence into the next. This gives a more detailed view of how the tokenizer responds to small changes in acoustic context. Window construction.
Let a denote a longer speech recording, and let x(m) = Crop a; tm , Tctx
(166)
be the length-Tctx window extracted from start time tm , where tm = t0 + m∆,
m = 0, 1, . . . , M.
(167)
Here ∆ > 0 is the sweep hop, and Tctx = 3 s in our experiments. Each window is tokenized independently: T (m) = DecAR Enc(x(m) ) .
(168)
Because adjacent windows overlap heavily when ∆ ≪ Tctx , the pair x(m) , x(m+1) provides a controlled local perturbation of the conditioning signal. For each adjacent pair, we compute normalized edit similarity
Adjacent-window sequence similarity.
Sim(m) ≜ Simedit T (m) , T (m+1) ,
(169)
where Simedit is defined in Eq. 133. This metric measures the fraction of the sequence that can be preserved under the optimal edit alignment. We also compute unigram Jaccard similarity to measure unordered tokenset overlap between adjacent windows. Together, these metrics quantify whether neighboring windows reuse similar symbolic material and whether that material appears in a similar ordered sequence. Edit-operation decomposition. Normalized edit similarity gives a single aggregate score, but it does not identify how the sequence changed. We therefore decompose the Levenshtein alignment between adjacent token sequences into substitutions, insertions, and deletions. For the adjacent pair T (m) , T (m+1) , let S (m) ,
I (m) ,
D(m)
(170)
denote the number of substitutions, insertions, and deletions in an optimal edit script, respectively. The total edit distance is then ED T (m) , T (m+1) = S (m) + I (m) + D(m) . (171) We report both absolute operation counts and length-normalized operation rates: (m)
rS
=
S (m) max(L(m) , L(m+1) )
(m)
,
rI
=
I (m) max(L(m) , L(m+1) )
,
(172)
and (m)
rD
=
D(m) . max(L(m) , L(m+1) )
(173)
This decomposition separates different kinds of instability. A high substitution rate indicates that adjacent windows preserve sequence length but change token identities. A high insertion or deletion rate indicates that the tokenizer changes the number or placement of decoded symbols. Thus, edit-operation statistics reveal whether local context motion mainly causes symbolic relabeling, boundary-driven token birth/death, or broader sequence restructuring. 61
We also track decoded sequence length
Adjacent length variation.
L(m) = |T (m) |,
(174)
∆L(m) = L(m+1) − L(m) .
(175)
and adjacent length change Unlike insertions and deletions in the optimal edit script, |∆L| captures only the net change in sequence length. For example, a pair may have |∆L| = 0 while still requiring many substitutions, or even matched insertions and deletions at different positions. Thus, |∆L| is interpreted as a length-control statistic, while the edit-operation decomposition provides a more detailed view of symbolic change. Distributional summaries. For each model, we aggregate statistics over all adjacent sweep pairs. In addition to means, we report medians, standard deviations, and thresholded fractions for |∆L|, edit similarity, and edit-operation counts. For example, we report the fraction of adjacent pairs satisfying |∆L| ≤ k,
k ∈ {0, 1, 2, 3, 5, 10, 20},
(176)
as well as analogous summaries for S (m) ,
I (m) ,
D(m) .
(177)
These distributional summaries are important because averages can hide qualitatively different behavior. A tokenizer may have a moderate mean edit distance because most adjacent pairs change smoothly and a small number undergo large jumps, or because every pair changes by a moderate amount. The distribution distinguishes these cases. Sweep trajectory.
For a single recording, the sweep produces a trajectory n oM −1 Sim(m) , S (m) , I (m) , D(m) , ∆L(m) .
(178)
m=0
Over a set of evaluation recordings, we summarize the trajectory by averaging over all adjacent window pairs: 1 XX (m) Simi , (179) Sim = Npairs i m where Npairs =
X
(Mi − 1).
(180)
i
Similarly, we compute S= and D=
1
XX
Npairs 1
Npairs
,
I=
m
i
XX i
(m)
Si
(m)
Di
,
∆L =
m
1 Npairs 1 Npairs
X X (m) Ii , i
XX i
(181)
m (m)
∆Li
.
(182)
m
Interpreting operation types. The edit-operation decomposition provides a more diagnostic view of tokenizer behavior than length variation alone. If adjacent windows mainly differ by substitutions, then the tokenizer is largely preserving sequence length but changing the symbolic labels assigned to shared acoustic content. This suggests context-sensitive relabeling. If adjacent windows mainly differ by insertions and deletions, then the tokenizer is changing the symbolic segmentation or token rate. This may reflect legitimate boundary effects when new phones enter or leave the window, but frequent large insertion–deletion bursts would indicate unstable sequence allocation. If both substitution and insertion–deletion counts are high, the tokenizer may be undergoing broader re-tokenization under small context shifts. Thus, the desired behavior is not zero edits. A 100 ms boundary shift can legitimately introduce or remove phones, transitions, or short lexical material. Rather, a structured tokenizer should exhibit bounded editoperation counts, controlled length changes, and a distribution dominated by small local edits rather than frequent large edit bursts. 62
Granularity-aware interpretation. The continuous-sweep probe must be interpreted together with sequence length. A dense geometric tokenizer may obtain high adjacent-window edit or Jaccard similarity because many frame-derived token assignments persist across overlapping windows. A compact autoregressive tokenizer may represent the same 3-second context with far fewer symbols. In that lower-rate regime, changing only a small number of tokens can produce a larger drop in normalized edit or Jaccard similarity. Therefore, overlap-based similarity, edit-operation rates, and absolute operation counts should be interpreted jointly. This distinction is especially important for PairAlign. Because PairAlign emits substantially shorter sequences, a small number of substitutions or insertions can have a large normalized effect. For this reason, absolute operation counts reveal whether the tokenizer is changing by a small number of symbolic edits, while normalized rates reveal the relative severity of those edits at the tokenizer’s own symbolic scale. What the probe tests. This experiment tests whether local acoustic changes produce locally coherent symbolic changes. Strong behavior in this probe would be indicated by nontrivial adjacent-window similarity, controlled adjacent length variation, and edit-operation distributions concentrated at small counts. Conversely, very low adjacent similarity together with large substitution, insertion, and deletion bursts would suggest that the tokenizer is sensitive to small context shifts and may be closer to a holistic segment-level code. The goal is therefore not strict invariance, but a controlled edit trajectory: neighboring windows should differ through a limited number of interpretable symbolic operations rather than arbitrary global re-tokenization. Relation to retrieval. This analysis is complementary to the long-form retrieval experiment. Retrieval evaluates whether token-space similarity can recover relevant segments from a larger archive, where neighboring archive windows may differ substantially. Continuous sweep instead examines a finer local regime, where adjacent windows share most of their acoustic content and differ only by a small temporal shift. Together, the two experiments test whether the learned token space is useful both for coarse archive-level search and for local sequence evolution under small changes in context. 4.7
Baselines and Ablations
Our evaluation compares PairAlign against representative geometric and sequence-consistency-based tokenization baselines that discretize frame-level speech representations. The primary baseline is the Stage I Mamba encoder + nearest-centroid VQ tokenizer used to initialize PairAlign. This model uses the same front-end encoder family and the same discrete vocabulary as PairAlign, but assigns tokens through local nearest-centroid decisions in the encoder space. It therefore serves as the most controlled geometric baseline: differences between this tokenizer and PairAlign isolate the effect of the later autoregressive sequence modeling, cross-view self-alignment, adaptive target generation, and learned length control. We also include a Stage I+ baseline, corresponding to the wav2tok-style extension of the geometric tokenizer. This baseline augments the Stage I representation with a sequence-consistency objective, encouraging paired realizations of the same speech content to produce more compatible token sequences while retaining the frame-level geometric tokenization structure. Stage I+ is therefore an important intermediate comparison: it tests whether improvements can be obtained simply by adding a sequence-consistency signal to a geometric tokenizer, without introducing the full autoregressive conditional tokenization and EMA-teacher self-alignment used by PairAlign. For geometric tokenizers, we report results after consecutive-token deduplication unless otherwise stated. This is important because nearest-centroid frame-level tokenizers can emit long runs of repeated symbols due to local temporal continuity. Deduplication removes this trivial redundancy and gives the geometric baselines a stronger and more compact symbolic representation for edit-distance comparison. Consequently, when PairAlign produces shorter sequences or lower archive token counts, the comparison is not merely against an uncompressed frame-level trace, but against a deduplicated geometric token stream. Unless otherwise stated, we evaluate a single PairAlign variant throughout the main experiments. This model uses sample-specific token budgets given by a simple length constraint of approximately 15% of the 63
Dataset
Model
Jaccard Edit Exact Mean Active Similarity Similarity Match Rate Sequence Vocabulary Length Size
Stage I Geometric LibriSpeech-100 Stage I+ Geometric PairAlign
0.718 0.738 0.719
0.609 0.629 0.630
0.264 0.265 0.291
92.09 75.61 35.55
512 512 512
Stage I Geometric Stage I+ Geometric PairAlign
0.742 0.750 0.753
0.616 0.643 0.691
0.267 0.267 0.301
78.65 58.79 26.19
456 420 430
TIMIT
Table 1: Discrete token consistency and compactness on LibriSpeech-100 and TIMIT. Each model independently tokenizes an anchor 3-second segment and a noisy content-preserving positive view. Jaccard Similarity measures unordered unigram token-set overlap. Edit Similarity measures normalized Levenshtein similarity. Exact Match Rate is the fraction of anchor–positive pairs with identical token strings. Mean Sequence Length measures symbolic compactness. Active Vocabulary Size measures how many codebook entries are used. conditioning representation length. We do not include an additional phoneme-count-supervised variant in the revised setup. This keeps the experimental study centered on the core PairAlign formulation and avoids introducing extra supervision beyond the main method. Interpretive role of the experiments. The experiments are designed to evaluate the main claims of PairAlign in a targeted and complementary manner. Discrete token consistency assesses whether the learned tokenizer assigns stable symbolic sequences to content-preserving acoustic views of the same continuous speech segment. Collapse and token-inventory analyses test whether improved consistency or compactness is achieved through genuine symbolic structure rather than pathological many-to-one degeneration or narrow vocabulary use. Long-form retrieval evaluates whether the learned token space remains useful in a realistic archive setting, where related regions of continuous audio must be retrieved from token similarity alone. The archive compactness analysis measures the storage and comparison cost of the resulting symbolic sequences, including token count, token rate, symbolic bitrate, and relative token reduction. The continuous-sweep compositionality probe tests how token sequences evolve when the acoustic context is shifted in small controlled steps. Because this probe is sensitive to symbolic granularity, its interpretation uses both overlap-based similarity metrics and absolute edit-trajectory statistics, including adjacent length variation and edit-operation counts. Together, these experiments compare three levels of symbolic-token learning: a deduplicated geometric tokenizer, a wav2tok-style Stage I+ tokenizer with sequence consistency added to geometric discretization, and the full PairAlign model with autoregressive conditional tokenization and self-alignment. This structure allows us to separate the effect of the encoder and VQ initialization from the effect of sequence-level consistency training, and from the additional effect of PairAlign’s autoregressive token structuring and adaptive self-alignment.
5
Results and Discussion
5.1
Discrete Token Consistency, Collapse, and Token-Inventory Analysis
We now evaluate whether the three tokenizers produce stable, compact, and non-degenerate symbolic sequences under noisy content-preserving perturbations. The results are reported on LibriSpeech-100 and TIMIT, comparing Stage I Geometric, Stage I+ Geometric, and PairAlign. The positive view in this experiment is a noisy version of the anchor, so high anchor–positive agreement indicates robustness to nuisance acoustic variation rather than agreement between unrelated examples. Stage I+ strengthens the geometric tokenizer but remains frame-synchronous. The Stage I+ results show that the wav2tok-style sequence-level term has the intended intermediate effect. Relative to Stage I Geometric, Stage I+ increases normalized edit similarity from 0.609 to 0.629 on LibriSpeech-100 and 64
Dataset
Model
Anchor Positive Collapsed Anchor Positive LowLowPair ExactExactDiversity Diversity Rate Collision Collision Sequence Sequence Rate Rate Rate Rate
Stage I Geometric LibriSpeech-100 Stage I+ Geometric PairAlign
0.0269 0.0263 0.0000
0.0381 0.0329 0.0000
0.0500 0.0450 0.0000
0.0000 0.0000 0.0001
0.0000 0.0000 0.0001
Stage I Geometric Stage I+ Geometric PairAlign
0.0240 0.0136 0.0000
0.0719 0.0223 0.0000
0.0803 0.0290 0.0000
0.0000 0.0000 0.0004
0.0000 0.0000 0.0028
TIMIT
Table 2: Collapse behavior of the tokenizers. Low-Diversity Sequence Rate is the fraction of sequences whose unique-token ratio is at most 0.2. Collapsed Pair Rate is the fraction of anchor–positive pairs in which either sequence is low-diversity collapsed. Exact-Collision Rate is the fraction of sequences in a stream that exactly duplicate another sequence from a different example in the same stream. Low-diversity collapse detects repetitive internal degeneration, whereas exact-collision collapse detects many-to-one assignment across examples. from 0.616 to 0.643 on TIMIT. Jaccard similarity also improves, from 0.718 to 0.738 on LibriSpeech-100 and from 0.742 to 0.750 on TIMIT. At the same time, mean sequence length decreases from 92.09 to 75.61 tokens on LibriSpeech-100 and from 78.65 to 58.79 tokens on TIMIT. Thus, adding an explicit order-sensitive sequence constraint to the geometric tokenizer improves cross-view agreement with the noisy positive view while reducing some frame-level redundancy. This behavior is important for the interpretation of PairAlign. Stage I+ is not a weak baseline: it already introduces a direct pairwise sequence-level criterion, but it still induces tokens through frame-level geometric posteriors. Its improvements therefore show that part of the desired noise-robust sequence structure can be injected into a geometric tokenizer. PairAlign tests a stronger hypothesis: that the token sequence itself should be generated and aligned as a sequence, rather than remaining a deduplicated stream of frame assignments. PairAlign changes the operating point from dense overlap to compact ordered consistency. PairAlign produces much shorter sequences than both geometric systems. On LibriSpeech-100, the mean length decreases from 92.09 tokens for Stage I and 75.61 tokens for Stage I+ to 35.55 tokens for PairAlign. On TIMIT, the mean length decreases from 78.65 and 58.79 tokens to 26.19 tokens. This corresponds to a reduction of about 61% relative to Stage I on LibriSpeech-100 and about 67% on TIMIT. The comparison with Stage I+ is also substantial: PairAlign is about 53% shorter on LibriSpeech-100 and about 55% shorter on TIMIT. Despite this compression, PairAlign preserves or improves the stricter order-sensitive metric. On LibriSpeech100, PairAlign obtains an edit similarity of 0.630, slightly above Stage I+ and above Stage I. On TIMIT, the improvement is larger: PairAlign reaches 0.691, compared with 0.643 for Stage I+ and 0.616 for Stage I. The TIMIT result is particularly important because it shows that the autoregressive self-alignment objective does not merely fit the LibriSpeech training domain; it yields a more stable ordered token sequence for noisy positive views under cross-corpus evaluation. The Jaccard results should be read at the correct granularity. On LibriSpeech-100, Stage I+ gives the highest unigram Jaccard score, while PairAlign is close to Stage I. This is not a failure of sequence organization. Jaccard discards order and rewards reuse of the same token identities; dense geometric tokenizers can perform well under this metric because many local frame-derived symbols are reused across augmented views. PairAlign emits far fewer tokens, so each token has a larger representational burden and small identity changes have a larger effect on set overlap. The fact that PairAlign nevertheless matches or exceeds the geometric systems in edit similarity indicates that its compact sequence preserves ordered compatibility even when unigram set overlap is not maximized. On TIMIT, PairAlign improves both Jaccard and edit similar65
Figure 1: Summary of discrete token consistency, compactness, and collapse. Stage I+ improves the geometric tokenizer while remaining in the frame-synchronous regime. PairAlign produces the shortest sequences, achieves the strongest cross-corpus edit consistency on TIMIT, and removes measured sequence-diversity collapse under the reported criterion. ity, giving the strongest evidence that sequence-level self-alignment improves cross-corpus noise-robust token consistency. Exact-match and collapse behavior distinguish robustness from degeneracy. The exact-match rate provides the strictest view of noise robustness. PairAlign achieves an exact-match rate of 0.291 on LibriSpeech-100 and 0.301 on TIMIT, compared with roughly 0.264–0.267 for the geometric systems. This means that the full autoregressive tokenizer more often assigns exactly the same token string to an anchor and its noisy positive view, despite emitting substantially shorter sequences. However, exact anchor–positive agreement must be interpreted together with collapse diagnostics. High exact-match rate is desirable only if it reflects content-preserving robustness, not many-to-one degeneration. The collapse results in Table 2 make this distinction explicit. Under the low-diversity criterion runiq ≤ 0.2, PairAlign has zero measured collapsed-pair rate on both datasets. By contrast, Stage I Geometric has collapsed-pair rates of 0.0500 on LibriSpeech-100 and 0.0803 on TIMIT. Stage I+ reduces these rates to 0.0450 and 0.0290, showing that the wav2tok-style sequence term already improves collapse behavior within the geometric regime. PairAlign removes this measured low-diversity collapse in the reported evaluations. The within-stream exact-collision diagnostic addresses the stronger full-collapse question: whether different examples are mapped to identical token strings. The geometric systems show no exact within-stream collisions in these evaluations. PairAlign shows extremely small exact-collision rates: 0.0001 for both anchor and positive streams on LibriSpeech-100, and 0.0004 for anchors and 0.0028 for positives on TIMIT. These 66
Figure 2: Edit-operation decomposition for anchor–positive token consistency. PairAlign requires far fewer absolute edit operations because it operates in a shorter sequence space. The reduction is visible across substitutions, insertions, and deletions, indicating that improved compactness is not obtained through unstable token birth/death.
values are several orders of magnitude below what would indicate many-to-one collapse, but they are worth reporting because PairAlign produces much shorter sequences and therefore has a higher chance of exact string collisions. The key point is that PairAlign’s higher anchor–positive exact-match rate is not accompanied by broad within-stream exact collision. Thus, the model improves noise robustness without collapsing the dataset into a small number of repeated token strings. Edit-operation decomposition shows bounded symbolic change under noise. The aggregate edit similarity can be decomposed into substitutions, insertions, and deletions, as shown in Figure 2. On LibriSpeech-100, the mean edit distance decreases from 39.25 operations for Stage I to 31.17 for Stage I+ and to 14.58 for PairAlign. On TIMIT, it decreases from 33.72 to 23.90 and then to 9.54. This reduction is not confined to one operation type. PairAlign has fewer substitutions, insertions, and deletions than both geometric systems on both datasets. This matters because a compact tokenizer could in principle obtain short sequences while becoming unstable in segmentation, producing many insertions and deletions under noisy positive views. The opposite pattern is observed. PairAlign reduces insertion and deletion counts sharply: on TIMIT, insertions decrease from 8.57 in Stage I and 5.72 in Stage I+ to 1.77 in PairAlign, while deletions decrease from 6.79 and 6.36 to 3.39. Substitutions also decrease from 18.36 and 11.81 to 4.37. The anchor–positive difference is therefore not dominated by large token birth/death events. PairAlign produces a shorter symbolic sequence whose response to noise is bounded in absolute edit operations. Vocabulary usage rules out trivial compactness. The token-inventory statistics in Figure 3 show that PairAlign’s compactness is not obtained by shrinking the vocabulary to a small set of generic symbols. On LibriSpeech-100, all three models activate the full 512-token vocabulary. PairAlign has lower normalized global entropy than the two geometric systems, 0.965 compared with 0.984 for Stage I and 0.980 for Stage I+, and a lower effective vocabulary size, approximately 411 compared with 464 and 453. This is expected: an autoregressive tokenizer operating at a much lower token rate will generally impose stronger sequence-level structure and a less uniform marginal token distribution. Crucially, the entropy remains high and the top-10 token mass remains below 0.09, which is incompatible with a small-token collapse explanation. The cross-corpus TIMIT inventory is more revealing. Stage I Geometric has a normalized entropy of 0.786 and an effective vocabulary of about 135; Stage I+ has 0.766 and about 119; PairAlign increases these to 67
Figure 3: Global token-inventory diagnostics. PairAlign remains broad-vocabulary rather than collapsed. On LibriSpeech-100 it uses the full 512-token alphabet with high normalized entropy; on TIMIT it has higher normalized entropy and a larger effective vocabulary than both geometric systems, despite much shorter sequences. 0.813 and about 160. It also lowers top-10 token mass relative to both geometric systems. Thus, under TIMIT shift, PairAlign does not become narrower than the geometric tokenizers; it uses a broader and less concentrated token distribution while also producing much shorter sequences. This is strong evidence that the learned autoregressive interface remains active and discriminative under cross-corpus evaluation. Native-position plots expose the difference in decoding regimes. Figures 4 and 5 plot token entropy, active-token count, and dominant-token mass at the models’ original output length scales. These plots should not be interpreted as if all models had the same number of decoding steps. Stage I and Stage I+ produce long deduplicated frame-derived sequences, while PairAlign produces a much shorter autoregressive sequence whose maximum length is capped at 44 in these results. The native-position curves show a characteristic PairAlign pattern. The first few positions have lower entropy and a higher dominant-token mass, especially on TIMIT. This does not mean that these positions are unconditioned or driven only by a generic autoregressive prior. In PairAlign, every decoding position is conditioned on the input through cross-attention to the encoder states and through the encoder-summary bias injected into the decoder stream. The lower early-position entropy is therefore better interpreted as a structured input-conditioned commitment: the decoder uses the acoustic condition to select an initial symbolic trajectory, after which token usage broadens across the body of the sequence. On LibriSpeech-100, PairAlign reaches high position-wise entropy through the body of the sequence, with hundreds of active tokens at later positions. On TIMIT, the same qualitative pattern remains, though the 68
Figure 4: Native-position token-inventory diagnostics on LibriSpeech-100. The models are shown at their own absolute output-length scales. PairAlign has a shorter decoding horizon and a characteristic earlyposition entropy ramp, after which token usage becomes broad across the compact sequence.
entropy level is lower because of cross-corpus shift and fewer evaluation examples. This behavior is preferable to a generic-prefix failure mode. If the autoregressive decoder were bypassing the acoustic condition, the early positions would remain dominated by a very small set of reusable tokens and entropy would remain low across the sequence. Instead, the plots show input-conditioned early commitment followed by broad token usage. The entropy ramp is therefore better interpreted as a structured conditional decoding effect than as collapse. Relative-position analysis separates compactness from positional collapse. The relative-position plots in Figures 6 and 7 normalize each sequence to a common beginning–middle–end axis. This is necessary because native-position plots otherwise confound sequence length with positional diversity. The normalized 69
Figure 5: Native-position token-inventory diagnostics on TIMIT. PairAlign emits fewer positions, but maintains substantial token diversity across those positions. The lower early entropy reflects the difficulty and importance of initial symbolic commitments when decoding begins from BOS. view shows that PairAlign does not place all meaningful variability in only one part of the sequence. On LibriSpeech-100, PairAlign entropy rises from the first relative bin into the middle of the sequence and remains high through the later bins. On TIMIT, PairAlign again shows an early lower-entropy region, but then maintains competitive or stronger normalized entropy than the geometric systems through much of the middle of the sequence. This is important for the claim that PairAlign learns a compact symbolic interface rather than a short code with a fixed prefix and a small variable suffix. The relative-position active-token plot shows that many tokens are available across the normalized sequence. In particular, PairAlign’s TIMIT relative-position behavior is stronger than one would expect from a collapsed English-trained autoregressive tokenizer under cross-corpus evaluation: active coverage remains broad, and global entropy is higher than that of both geometric systems. 70
Figure 6: Length-normalized position-wise token entropy. Relative-position bins allow comparison of the beginning, middle, and end of each sequence independent of absolute length. PairAlign retains broad token usage across the normalized sequence, while showing a clearer early-position commitment region.
Figure 7: Length-normalized active-token coverage. PairAlign uses fewer absolute positions, but each relative region still activates a large portion of the vocabulary. On TIMIT, PairAlign preserves stronger active-token coverage than Stage I+ over much of the normalized sequence. Thus, the compactness of PairAlign is distributed across the sequence, rather than being achieved by a small number of position-specific symbols. Overall interpretation. The results support a more nuanced conclusion than a simple claim that one tokenizer dominates on every metric. Stage I Geometric is a strong local-assignment baseline. Stage I+ Geometric confirms that adding a wav2tok-style sequence-level term improves the geometric tokenizer: it increases Jaccard and edit similarity, reduces sequence length, and lowers sequence-diversity collapse while remaining frame-synchronous. PairAlign then moves to a different operating point. It emits far fewer tokens, obtains the best order-sensitive edit similarity on TIMIT, matches or slightly exceeds Stage I+ on LibriSpeech-100 edit similarity, improves exact-match rate on both datasets, and removes measured sequence-diversity collapse under the reported criterion. 71
The key interpretation is that PairAlign improves robustness to noisy content-preserving views without relying on degenerate many-to-one behavior. Anchor–positive exact matching is desirable because the positive view is a noisy version of the same segment. Collapsed-pair rate and within-stream exact-collision rate are different safeguards: they test whether the tokenizer is producing low-diversity sequences or assigning identical token strings to different examples. PairAlign achieves higher anchor–positive exact matching while keeping within-stream exact collisions extremely rare. The inventory and position-wise plots are essential for interpreting these numbers. PairAlign’s compactness is not explained by vocabulary collapse: it uses all 512 tokens on LibriSpeech-100, maintains high entropy, and under TIMIT shift has higher normalized entropy and effective vocabulary size than both geometric systems. Its native-position plots show a plausible conditional autoregressive structure: an input-conditioned early commitment phase, supported by cross-attention and encoder-summary bias at every decoding position, followed by broad token usage across the body of the sequence. Together, these results support the central PairAlign hypothesis. Sequence-level self-alignment does not merely remove frame-level redundancy. It reorganizes the symbolic interface so that related clean/noisy acoustic realizations produce compact, ordered, and discriminative token sequences. The geometric systems retain strong dense local overlap, while PairAlign trades that redundancy for a much shorter autoregressive sequence with stronger cross-corpus edit consistency, better exact string agreement under noise, zero measured sequence-diversity collapse, and broad token-inventory usage.
5.2
Retrieval Comparison with the Geometric Tokenizer
We next compare PairAlign against the Stage I geometric tokenizer on a cross-corpus retrieval task. This evaluation is important because the PairAlign formulation is explicitly motivated by the goal of mapping continuous speech segments to discrete token sequences whose similarity remains meaningful under edit distance. If the learned token space preserves sequence-level structure, then retrieval using normalized token edit distance should remain effective even after moving beyond frame-synchronous geometric discretization. At the same time, this experiment should not be interpreted as a comparison between two tokenizers with identical representational granularity. The Stage I geometric tokenizer assigns tokens through local nearestcentroid decisions in the encoder space and therefore remains closely tied to local frame-level continuity, even after consecutive-token deduplication. PairAlign instead learns a shorter autoregressive token sequence whose length, ordering, and cross-realization consistency are shaped through sequence-level self-alignment. Retrieval with normalized edit distance is therefore used here as a diagnostic: it tests whether this more compact sequence-level representation preserves enough symbolic structure to support search. To make the comparison more informative than an in-domain evaluation alone, we conduct the experiment on TIMIT (Garofolo et al., 1993), which is not used for training. Both models are trained on LibriSpeech (Panayotov et al., 2015) and evaluated cross-corpus on the same TIMIT archive. Concretely, we construct a retrieval archive of 9,461 overlapping 3-second speech segments using a sweep hop of 1.5 seconds, and evaluate with 300 augmented query segments generated from source segments in the same corpus. Thus, adjacent archive segments overlap by 50% of their duration. This retrieval setting is intentionally less local than the continuous-sweep analysis in Section 5.3, where the hop is only 100 ms. The retrieval experiment therefore evaluates whether the token space supports search across substantially shifted 3-second contexts, rather than merely testing near-identical neighboring windows. Each query and each archive segment is tokenized independently. Ranking is then performed directly in token space using normalized edit distance. That is, if Tq denotes the query token sequence and Ta denotes an archive token sequence, then retrieval is based on the Levenshtein edit distance between Tq and Ta , normalized by sequence length as in Eq. 133. Lower normalized edit distance corresponds to greater symbolic similarity. A central issue in this evaluation is the definition of relevance. Because the retrieval units are continuous 3-second segments rather than isolated words, relevance is not a single binary notion. We therefore report results under three complementary relevance definitions. 72
Relevance definition
Model
R@1
R@5
R@10
R@20
MRR
Mean FRR
Segment overlap
Geometric PairAlign
0.75 0.71
0.83 0.79
0.85 0.80
0.87 0.82
0.78 0.74
36.40 53.99
Phoneme exact
Geometric PairAlign
0.75 0.71
0.83 0.78
0.85 0.80
0.86 0.82
0.78 0.74
48.22 73.40
Phoneme relaxed
Geometric PairAlign
0.75 0.71
0.83 0.79
0.85 0.80
0.87 0.84
0.78 0.74
38.55 48.95
Table 3: Retrieval comparison on TIMIT (Garofolo et al., 1993) between the Stage I geometric tokenizer and PairAlign. Models are trained on LibriSpeech (Panayotov et al., 2015) and evaluated cross-corpus on TIMIT. The retrieval archive is constructed from 3-second windows with a 1.5-second hop. Queries are ranked against the same TIMIT archive using normalized edit distance over token sequences. R@K denotes Recall@K, MRR denotes Mean Reciprocal Rank, and FRR denotes First Relevant Rank. Median first relevant rank is 1 for both models under all three relevance definitions, and HitRate is 1.0 throughout; these quantities are omitted for compactness.
Segment-overlap relevance. Under this criterion, an archive segment is considered relevant if it is the same source segment as the query or if it overlaps sufficiently with that source segment in time. This is the least linguistically restrictive relevance definition. It asks whether the tokenizer retrieves the original local region of continuous audio together with its immediate temporal neighbors. Phoneme-exact relevance. Under this criterion, an archive segment is considered relevant only if the phoneme sequence aligned to that segment matches the phoneme sequence of the query source segment exactly. This is the strictest relevance definition reported here: it requires exact segment-level phonemic agreement rather than merely temporal overlap or broad phonetic similarity. Phoneme-relaxed relevance. Under this criterion, an archive segment is considered relevant if its aligned phoneme content is sufficiently close to that of the query source segment under a more tolerant matching rule. This criterion captures cases where two continuous segments are phonetically very similar without being identical under the strict exact-match condition. Thus, the three relevance definitions probe retrieval at increasing levels of linguistic specificity: segmentoverlap relevance tests recovery of the local continuous-audio region, phoneme-exact relevance tests strict phonemic identity, and phoneme-relaxed relevance tests strong but not necessarily exact phonetic similarity. Retrieval behavior under normalized edit distance. Table 3 shows that both tokenizers support retrieval under normalized edit-distance ranking, but the geometric tokenizer is stronger at the top of the ranking. For segment-overlap relevance, Recall@1 is 0.75 for the geometric tokenizer and 0.71 for PairAlign. The same pattern appears under phoneme-exact relevance, where Recall@1 is 0.75 for the geometric tokenizer and 0.71 for PairAlign, and under phoneme-relaxed relevance, where the corresponding values are again 0.75 and 0.71. MRR follows the same trend: the geometric tokenizer obtains 0.78, while PairAlign obtains approximately 0.74 across the three relevance definitions. This result clarifies the cost of moving from a dense frame-derived tokenizer to a compact autoregressive tokenizer. PairAlign is designed to learn token sequences whose edit-distance similarity remains meaningful, and the retrieval results show that this property is preserved to a substantial degree. However, the geometric tokenizer remains advantageous for immediate rank-1 retrieval. This is expected because a frame-derived tokenizer retains more dense local correspondence, and normalized edit-distance retrieval benefits from such local token persistence when query and archive segments share fine-grained acoustic structure. Retrieval coverage is preserved, but the tail becomes heavier. Although PairAlign is weaker at the earliest ranks, it does not lose retrieval coverage. HitRate is 1.0 for both models under all relevance 73
Archive statistic
Geometric
PairAlign
Number of segments Window length / hop Total tokens Ntok Average tokens / segment L Token rate Rtok Bits / token btok Symbolic bitrate Rbit Token-count compression Ctok Relative token reduction rred Minimum tokens / segment Maximum tokens / segment Archive cache size
9,461 3.0s / 1.5s 800,611 84.62 28.21 tok/s 9 253.89 bit/s – – 12 154 5.90 MB
9,461 3.0s / 1.5s 360,723 38.13 12.71 tok/s 9 114.39 bit/s 2.22× 54.94% 5 44 4.85 MB
Table 4: Archive compactness comparison on TIMIT (Garofolo et al., 1993) between the Stage I geometric tokenizer and PairAlign. Both archives contain the same 9,461 continuous 3-second segments and therefore the same underlying speech content. Both tokenizers use a vocabulary of size |A| = 512, so each token requires btok = ⌈log2 512⌉ = 9 bits under a fixed-length symbolic code. The symbolic bitrate is not an audio-codec bitrate; it measures the storage and comparison rate of the token sequence used for retrieval. PairAlign reduces the archive token count by 54.94%, corresponding to a 2.22× reduction in stored token sequence length.
definitions, and the median first relevant rank is 1 throughout. Thus, for a typical query, a relevant segment is still retrieved at the top of the ranked list. The main difference is in the tail of harder queries. The mean first-relevant-rank statistics expose this tail behavior. For segment-overlap relevance, PairAlign has a higher mean FRR than the geometric tokenizer, 53.99 versus 36.40. For phoneme-exact relevance, the gap is larger: PairAlign has mean FRR 73.40, compared with 48.22 for the geometric tokenizer. For phoneme-relaxed relevance, PairAlign has mean FRR 48.95, compared with 38.55 for the geometric tokenizer. These values indicate that PairAlign preserves typical retrieval behavior but has more difficult outlier queries, especially under strict phoneme-exact matching. This distinction is important. PairAlign should not be described as matching the geometric tokenizer uniformly across the full ranking distribution. Rather, it remains effective and fully covers the query set, while giving up some top-rank sharpness and producing a heavier tail. The heavier tail is consistent with PairAlign’s lower-rate representation: because each token carries more information, a small number of token changes can have a larger effect on normalized edit distance, which can move difficult queries further down the ranking.
Compactness of the symbolic archive. The retrieval results should be interpreted together with the archive statistics in Table 4. Both archives contain the same 9,461 TIMIT segments and are constructed with the same 3-second window and 1.5-second hop. The only difference is the symbolic representation used to encode those segments. Under this controlled comparison, PairAlign reduces the total number of stored tokens from 800,611 to 360,723, a reduction of approximately 54.9%. The average sequence length decreases from 84.62 tokens per segment to 38.13 tokens per segment. Equivalently, the average token rate decreases from approximately 28.21 tokens per second to 12.71 tokens per second. Since both tokenizers use the same vocabulary size, |A| = 512, this reduction also translates directly into a lower fixed-length symbolic bitrate. Under a 9-bit code per token, the geometric tokenizer corresponds to approximately 253.89 bit/s, whereas PairAlign corresponds to approximately 114.39 bit/s. Equivalently, PairAlign gives a 2.22× reduction in stored token count relative to the geometric tokenizer. These values should not be interpreted as waveform compression rates, since the tokens are not used here to reconstruct audio. Rather, they quantify the size of the symbolic representation over which edit-distance retrieval is performed. 74
The serialized archive cache also decreases from 5.90 MB to 4.85 MB. This decrease is more modest than the raw token-count reduction because the cache includes metadata and storage overhead in addition to token indices. For this reason, the token count, token rate, and symbolic bitrate are the cleaner measures of retrieval-time sequence compactness. PairAlign therefore performs retrieval in a much smaller sequence space, with less than half as many stored tokens. The comparison is especially meaningful because the geometric tokenizer is already evaluated after consecutive-token deduplication. Thus, the compactness gain is not merely due to removing repeated frame labels; it arises from the autoregressive sequence-level tokenization itself. Retrieval reflects two different tokenization styles. The gap between the two tokenizers is not only a difference in retrieval score; it reflects two different styles of tokenization. The geometric tokenizer behaves like a dense, locally grounded symbolic trace. Its tokens are obtained from local nearest-centroid assignments in encoder space, so nearby or overlapping acoustic regions tend to reuse local token patterns. This style is naturally favorable for normalized edit-distance retrieval, especially at rank 1, because overlapping or phonetically similar segments can share many local symbols. PairAlign behaves differently. It is a lower-rate autoregressive tokenizer that generates the complete token sequence from BOS under an acoustic condition. Its sequence is not a deduplicated frame trace; it is a compact ordered representation whose length and token ordering are learned through cross-realization sequence alignment. The token-inventory analysis in Section 5.1 shows that this compactness is not achieved by collapse. PairAlign maintains broad vocabulary usage under TIMIT evaluation and does not reduce the archive to a small set of generic strings. Instead, it uses fewer positions and places more representational burden on each token decision. The position-wise inventory diagnostics further explain the retrieval behavior. Both tokenizers show lowentropy early positions, meaning that the beginning of the token sequence is more restricted than later regions. PairAlign shows a stronger early commitment, which is expected for an autoregressive decoder that must choose an initial symbolic trajectory from BOS. After this early commitment, token usage broadens across the sequence. This decoding style can produce compact and discriminative token strings, but it also means that early symbolic decisions can influence the rest of the sequence. Under normalized edit-distance retrieval, such early trajectory differences may have a larger effect than in a dense frame-derived tokenizer, where the representation is distributed over many more local symbols. Temporal scale of the retrieval setting. The retrieval archive is constructed from 3-second windows with a 1.5-second hop, so neighboring archive entries are half-overlapping rather than near-duplicates. This is a relatively coarse segmentation of the continuous speech stream: shifting the window by 1.5 seconds changes a substantial part of the phonetic and lexical content contained in the segment. The retrieval experiment therefore tests whether token sequences support search across partially overlapping continuousspeech contexts, rather than merely testing robustness to small local shifts. This temporal scale is important for interpreting the comparison. A frame-derived geometric tokenizer retains many local assignments and can therefore benefit when two segments share overlapping acoustic regions. PairAlign operates in a lower-rate symbolic space: its archive contains approximately 12.71 tokens per second on average, compared with 28.21 tokens per second for the deduplicated geometric tokenizer. Thus, PairAlign performs retrieval with fewer than half as many symbols, while still retrieving relevant material for all queries. The reduction in top-rank sharpness should therefore be read together with the large reduction in symbolic rate: the geometric tokenizer retains denser local detail, whereas PairAlign provides a compact sequence-level representation that preserves useful but less redundant edit-distance structure. Connection to the continuous-sweep analysis. The continuous-sweep experiment in Section 5.3 uses the same TIMIT domain but a much smaller sweep hop of 100 ms. This 100 ms hop is approximately phoneme-scale or sub-syllabic: it is small enough that neighboring windows differ by only a small local portion of speech, while still potentially changing one or more phone-level acoustic events. The sweep analysis therefore probes a finer form of local stability than the retrieval experiment. Retrieval with a 1.575
second hop tests search over substantially shifted segments, whereas continuous sweep with a 100 ms hop tests how the symbolic sequence evolves under small phonetic-scale context changes. The two experiments give a consistent picture when interpreted at the correct symbolic scale. In the sweep setting, PairAlign produces substantially shorter token sequences than the geometric tokenizer: 25.48 tokens per 3-second window on average, compared with 85.08 for the geometric tokenizer. It also shows smaller absolute adjacent length changes, with median |∆L| = 4 compared with 7 for the geometric tokenizer. The edit-operation decomposition strengthens this interpretation. The median adjacent edit distance is 15 operations for PairAlign, compared with 36 for the geometric tokenizer, and 65.8% of PairAlign adjacent transitions require at most 20 edit operations, compared with only 13.2% for the geometric tokenizer. Thus, PairAlign’s local trajectory is more bounded in absolute edit activity, even though its normalized edit similarity is lower. This distinction helps explain the retrieval behavior observed here. A dense geometric sequence can preserve many local token assignments across overlapping or phonetically related segments, which favors sharp rank-1 retrieval under normalized edit distance. PairAlign compresses the same 3-second context into fewer tokens, so each substitution, insertion, or deletion has a larger effect on normalized edit distance. This makes strict nearest-neighbor ranking more sensitive to small symbolic changes, especially under the phonemeexact relevance criterion. The reduction in rank-1 sharpness and the heavier mean-FRR tail are therefore consistent with the compact sequence scale at which PairAlign operates, rather than evidence of vocabulary collapse or arbitrary tokenization. The edit-operation analysis also rules out a stronger negative interpretation. PairAlign is not producing arbitrary segment-level re-tokenizations under small context shifts. Most adjacent changes remain within a moderate edit distance and are dominated by substitutions, with smaller insertion and deletion counts than the geometric tokenizer. For PairAlign, the mean substitution, insertion, and deletion counts are 9.22, 3.91, and 4.35, respectively, compared with 23.22, 9.56, and 9.79 for the geometric tokenizer. This suggests context-sensitive symbolic relabeling plus limited token birth/death, rather than frequent large sequence restructuring. Consequently, the retrieval results, sweep results, and token-inventory diagnostics point to the same trade-off: PairAlign gives up some dense frame-local overlap, but preserves a controlled and comparison-friendly symbolic trajectory in a much shorter sequence space.
Compactness-aware retrieval interpretation. The retrieval, sweep, and inventory analyses together suggest that PairAlign should be understood as a compactness-aware sequence tokenizer. It does not preserve local frame-level redundancy as strongly as the geometric tokenizer, and therefore it is not expected to dominate every edit-distance retrieval metric, especially at rank 1. However, it preserves enough sequencelevel structure to remain effective in cross-corpus retrieval, while reducing the archive token count by more than half. This is the relevant trade-off for the PairAlign objective: the model gives up some dense local overlap in exchange for a shorter and more explicitly sequence-modeled symbolic representation. The token-inventory analysis is essential for interpreting this trade-off. A shorter sequence would not be useful if it were produced by collapse to a small set of frequent symbols or by assigning generic strings to many different segments. That is not the observed behavior. PairAlign maintains broad vocabulary usage under TIMIT evaluation, while its position-wise statistics show an autoregressive structure: stronger early commitment followed by broader token usage across later positions. This supports the view that PairAlign compresses the symbolic sequence by changing the style of tokenization, not by discarding discriminative capacity. This also clarifies the role of edit distance in the method. PairAlign is not claiming that a shorter autoregressive token sequence will always produce higher raw edit similarity than a dense frame-derived sequence. Rather, it aims to make edit-distance comparison meaningful in a compact sequence space. The TIMIT retrieval results support this goal in a qualified but important sense: despite the large reduction in token count, PairAlign preserves full HitRate, median first relevant rank of 1, and reasonable Recall@K under all three relevance definitions. The cost is reduced top-rank sharpness and a heavier tail of difficult queries, especially under phoneme-exact relevance. The sweep analysis explains why this is possible: PairAlign’s 76
token sequence is not maximally redundant, but its local edit trajectory is bounded and structured rather than arbitrary. Summary. Overall, the retrieval experiment shows that PairAlign remains effective for cross-corpus tokenspace search after moving beyond frame-synchronous geometric discretization, but it does not match the geometric tokenizer at the top of the ranking. The geometric tokenizer is stronger at Recall@1, Recall@5, and MRR, especially under strict phoneme-exact relevance. PairAlign preserves full retrieval coverage and median first relevant rank of 1, but has a heavier tail of difficult queries. This is achieved while reducing the symbolic archive by approximately 55% in token count. Together with the continuous-sweep and token-inventory analyses, these results support a compactnessaware interpretation. The geometric tokenizer behaves like a dense frame-local symbolic trace, which is advantageous for nearest-neighbor sharpness under normalized edit distance. PairAlign behaves like a lowerrate autoregressive symbolic interface: it has stronger early trajectory commitment, broad token usage across the compact sequence, bounded local edit trajectories, and substantially lower archive token count. The updated sweep results make this trade-off clearer: PairAlign does not preserve normalized adjacentwindow overlap as strongly as the geometric tokenizer, but it changes through fewer absolute edit operations and smaller absolute length shifts. Thus, PairAlign should not be viewed as a replacement for dense framelocal tokenization on every overlap or rank-1 retrieval metric. Rather, it provides a compact sequencelevel representation that preserves meaningful edit-distance retrieval structure while substantially reducing sequence length. 5.3
Continuous-Sweep Tokenization and Compositionality Analysis
We next examine how the learned tokenization changes as the acoustic context is varied continuously. Unlike the retrieval experiment above, which constructs a 3-second archive with a 1.5-second hop, the continuoussweep analysis uses a much finer hop of 100 ms on the same TIMIT (Garofolo et al., 1993) corpus. This creates a stricter local perturbation setting: adjacent 3-second windows share almost all of their duration, but the 100 ms shift is still large enough to affect phone-scale acoustic content. Since phoneme durations in continuous speech are typically on the order of tens to hundreds of milliseconds, this sweep hop probes whether tokenization changes coherently under small linguistically meaningful shifts rather than only under nearly identical waveform perturbations. Because neighboring windows share most of their acoustic content, this setting provides a useful probe of local tokenization behavior. A structured tokenizer should preserve some symbolic material across adjacent windows, while still allowing changes due to speech entering and leaving the window boundaries. We compare PairAlign with the Stage I geometric tokenizer. For the geometric tokenizer, consecutive-token deduplication is applied before computing all metrics, so the comparison is against the compact symbolic sequence used for matching rather than the raw frame-synchronous token stream. PairAlign is evaluated directly using its autoregressively decoded token sequences. In addition to normalized edit similarity, unigram Jaccard similarity, and adjacent length change, we decompose the optimal Levenshtein alignment between adjacent windows into substitutions, insertions, and deletions. This is important because length variation alone is an incomplete measure of local stability. A tokenizer may preserve length while changing many token identities, or it may change length through a small number of localized insertions and deletions. The edit-operation decomposition therefore lets us distinguish context-sensitive symbolic relabeling from token-rate or segmentation changes. Fine-grained sweep resolution. The continuous-sweep analysis uses the same TIMIT (Garofolo et al., 1993) domain as the retrieval experiment, but with a much finer hop of 100 ms. Adjacent sweep windows therefore overlap by 2.9 seconds out of 3 seconds. This creates a stringent local perturbation setting: the two windows are nearly identical at the segment level, yet the 100 ms boundary replacement can still contain phone-scale acoustic material. The experiment therefore probes whether token sequences evolve coherently under small linguistically meaningful changes in context. 77
Metric
Geometric
PairAlign
Dataset Window length / hop Adjacent pairs Mean sequence length Approx. tokens / second Edit similarity Unigram Jaccard Adjacent length change |∆L| Length ratio Relative length change
TIMIT 3.0s / 0.1s 11,137 85.08 28.36 0.536 ± 0.217 0.595 ± 0.227 10.30 ± 11.18 0.888 ± 0.111 12.1%
TIMIT 3.0s / 0.1s 11,137 25.48 8.49 0.414 ± 0.218 0.479 ± 0.211 6.21 ± 6.95 0.793 ± 0.182 24.4%
Table 5: Continuous-sweep tokenization statistics on TIMIT (Garofolo et al., 1993) for adjacent 3-second windows with a 100 ms sweep hop. Edit similarity is normalized Levenshtein similarity between adjacent token sequences. Unigram Jaccard measures token-set overlap. Adjacent length change is the absolute difference in decoded sequence length between neighboring windows. Relative length change is computed as mean |∆L| divided by mean sequence length. The geometric tokenizer is evaluated after consecutive-token deduplication. Distributional statistic
Geometric
PairAlign
Median edit similarity Edit similarity ≥ 0.5 Edit similarity ≥ 0.4
0.596 70.1% 80.3%
0.409 36.9% 52.2%
Median |∆L| |∆L| = 0 |∆L| ≤ 1 |∆L| ≤ 2 |∆L| ≤ 3 |∆L| ≤ 5 |∆L| ≤ 10 |∆L| ≤ 20
7 4.2% 12.3% 19.8% 27.1% 41.0% 66.3% 88.0%
4 15.6% 28.2% 38.9% 48.1% 61.6% 80.0% 94.1%
Table 6: Distributional view of adjacent-window length behavior in the continuous-sweep experiment. PairAlign produces much shorter sequences and smaller absolute adjacent length changes, while the geometric tokenizer preserves stronger token-overlap similarity under edit and Jaccard metrics. These metrics capture complementary aspects of stability and should be interpreted together with sequence length, edit-operation counts, and sweep hop. This differs from the retrieval setting, where the archive hop is 1.5 seconds and neighboring windows replace half of the segment. Retrieval tests whether the token space supports search across substantially shifted continuous-speech contexts. Continuous sweep instead tests local sequence behavior under fine-grained context evolution. Together, the two experiments evaluate the same TIMIT-domain token space at two different temporal scales: coarse half-window retrieval and local 100 ms sequence evolution. Adjacent-window behavior and symbolic granularity. Tables 5 and 6 show that the two tokenizers occupy different symbolic regimes. The geometric tokenizer has higher adjacent-window overlap under normalized edit similarity and unigram Jaccard similarity. Its mean edit similarity is 0.536, compared with 0.414 for PairAlign, and its mean unigram Jaccard similarity is 0.595, compared with 0.479 for PairAlign. Thus, if local continuity is measured only by token-overlap similarity between adjacent 100 ms shifts, the geometric tokenizer preserves a larger fraction of the neighboring token sequence. This result is consistent with the denser, more local nature of the geometric tokenizer. Even after consecutivetoken deduplication, it produces an average of 85.08 tokens per 3-second window, corresponding to roughly 28.36 tokens per second. PairAlign produces 25.48 tokens on average, corresponding to roughly 8.49 tokens 78
Edit-operation statistic
Geometric
PairAlign
Edit distance ED Median ED
42.57 ± 24.06 36
17.48 ± 10.47 15
Substitutions S Insertions I Deletions D
23.22 ± 20.43 9.56 ± 8.84 9.79 ± 9.46
9.22 ± 7.32 3.91 ± 5.90 4.35 ± 5.64
Substitution rate rS Insertion rate rI Deletion rate rD
0.253 ± 0.204 0.314 ± 0.197 0.104 ± 0.088 0.127 ± 0.165 0.107 ± 0.095 0.145 ± 0.157
ED ≤ 10 ED ≤ 20 ED ≤ 30 S ≤ 10 I≤5 D≤5
1.0% 13.2% – 28.1% 35.7% 35.6%
31.3% 65.8% 85.7% 67.2% 77.1% 74.3%
Table 7: Edit-operation decomposition for adjacent windows in the continuous-sweep experiment. S, I, and D denote substitutions, insertions, and deletions in an optimal Levenshtein edit script. Rates are normalized by the maximum of the two adjacent sequence lengths. PairAlign has lower normalized overlap, but its absolute edit trajectory is more bounded: the median edit distance is 15 operations, compared with 36 for the geometric tokenizer. Most PairAlign transitions remain within moderate substitution, insertion, and deletion counts, indicating controlled symbolic change rather than frequent large sequence restructuring.
per second. The two tokenizers therefore do not merely differ in metric values; they differ in symbolic scale. The geometric tokenizer expresses the segment through many local symbols, whereas PairAlign expresses it through a much shorter ordered sequence. This granularity difference is central for interpreting compositionality. A dense frame-derived tokenizer can preserve many local assignments across adjacent windows simply because most frames are shared. PairAlign, by contrast, makes fewer symbolic decisions, and each token carries a larger representational burden. Consequently, a small number of token changes can produce a larger penalty under normalized edit similarity and Jaccard overlap. Lower normalized overlap therefore should not be interpreted directly as unstable tokenization; it partly reflects the fact that PairAlign operates at a coarser symbolic granularity.
Absolute length stability under 100 ms shifts. The length statistics show that PairAlign has smaller absolute sequence-length variation under fine-grained window motion. PairAlign has an adjacent length change of 6.21 ± 6.95 tokens, compared with 10.30 ± 11.18 tokens for the geometric tokenizer. The median adjacent length change is 4 tokens for PairAlign and 7 tokens for the geometric tokenizer. Since the two windows differ only by a 100 ms boundary replacement, this smaller median length change indicates that PairAlign maintains more controlled decoded lengths under small context changes. The full distribution strengthens this conclusion. For PairAlign, 48.1% of adjacent pairs have |∆L| ≤ 3, compared with 27.1% for the geometric tokenizer. Similarly, 61.6% of PairAlign transitions stay within five tokens, whereas this is true for 41.0% of geometric-tokenizer transitions. At wider thresholds, 80.0% of PairAlign adjacent pairs have |∆L| ≤ 10, compared with 66.3% for the geometric tokenizer, and 94.1% have |∆L| ≤ 20, compared with 88.0%. PairAlign also produces exactly unchanged adjacent lengths more often (15.6% versus 4.2%). Thus, PairAlign is less prone to large absolute length fluctuations as the window moves through speech. Although the relative length-change value is larger for PairAlign, this is a direct consequence of its much shorter sequences: a four-token change is a larger fraction of a 25-token sequence than of an 85-token sequence. For this reason, relative length change should not be interpreted without the absolute values. In absolute terms, PairAlign shows stronger length control under the 100 ms sweep. 79
Edit-operation decomposition gives a sharper stability diagnosis. Length variation alone cannot tell us whether the sequence is changing through localized token birth/death, symbolic relabeling, or broader re-tokenization. Table 7 therefore decomposes the adjacent edit script into substitutions, insertions, and deletions. This reveals a substantially different picture from normalized overlap metrics alone. The geometric tokenizer has a mean edit distance of 42.57±24.06 and a median edit distance of 36 operations between adjacent 100 ms shifts. PairAlign has a much smaller mean edit distance of 17.48 ± 10.47 and a median edit distance of 15 operations. Thus, although PairAlign has lower normalized edit similarity, the absolute number of operations needed to transform one adjacent tokenization into the next is far smaller. This distinction is central: normalized edit similarity asks what fraction of a sequence changes, whereas the absolute edit script asks how many symbolic operations the tokenizer actually changes under the local perturbation. The thresholded edit-distance distribution makes the contrast especially clear. Only 13.2% of geometrictokenizer adjacent pairs have ED ≤ 20, whereas 65.8% of PairAlign adjacent pairs fall below this threshold. Similarly, 31.3% of PairAlign transitions have ED ≤ 10, compared with only 1.0% for the geometric tokenizer. Moreover, 85.7% of PairAlign transitions have ED ≤ 30. This indicates that PairAlign’s local trajectory is usually bounded by a moderate number of symbolic operations, while the geometric tokenizer often undergoes many more local token changes because it operates at a much higher token rate. Substitutions dominate PairAlign’s local changes. The edit-operation decomposition also clarifies the type of change induced by a 100 ms shift. For PairAlign, the mean number of substitutions is 9.22, compared with 3.91 insertions and 4.35 deletions. The median counts follow the same pattern: 7 substitutions, 2 insertions, and 3 deletions. Thus, most PairAlign changes are token-identity changes under a relatively stable sequence allocation, rather than large insertion/deletion bursts. This suggests that adjacent windows often preserve a comparable symbolic skeleton while relabeling some positions in response to the changed acoustic context. The distribution supports this interpretation. For PairAlign, 67.2% of adjacent transitions have at most 10 substitutions, 77.1% have at most 5 insertions, and 74.3% have at most 5 deletions. Moreover, 49.8% of PairAlign transitions have at most one insertion, and 37.9% have at most one deletion. Thus, token birth/death is usually limited relative to the geometric tokenizer, although it is more frequent than in the earlier shorter PairAlign sweep. The main source of change is still not uncontrolled expansion or contraction of the decoded sequence, but a bounded number of substitutions together with moderate insertion/deletion adjustments. This is the desired behavior for a compact autoregressive tokenizer. A 100 ms boundary shift can legitimately change the local phonetic evidence available to the model, especially near the window boundaries. The goal is therefore not zero edits. Rather, the desired behavior is a controlled edit trajectory: bounded symbolic updates, limited token birth/death, and no frequent large edit bursts. PairAlign matches this pattern more closely than the geometric tokenizer in absolute operation counts. Interpreting sweep metrics with token-inventory evidence. The token-inventory analysis in Section 5.1 is important for interpreting these sweep results. PairAlign’s compactness is not explained by vocabulary collapse or by reuse of a small set of generic symbols. In the discrete token analysis, PairAlign maintains broad token usage under TIMIT evaluation, with high normalized entropy and a large effective vocabulary despite emitting far fewer tokens than the geometric tokenizer. This matters because a short sequence alone is not evidence of a useful symbolic interface: a degenerate autoregressive tokenizer could also be short if it collapsed to a narrow set of repeated or generic outputs. The inventory diagnostics rule out this explanation. The position-wise inventory diagnostics add a second nuance. Both tokenizers show low-entropy behavior at the earliest positions, followed by broader token usage at later positions. For the geometric tokenizer, this early concentration can arise from regularity in the first frame-derived assignments after deduplication. For PairAlign, the effect is stronger: the autoregressive decoder must choose an initial symbolic trajectory from BOS, and the first few token decisions are correspondingly more concentrated. This should not be 80
interpreted as an unconditioned decoder prior, because every PairAlign decoding position is conditioned on the input through cross-attention and the encoder-summary pathway. Rather, the early low-entropy region indicates a stronger input-conditioned commitment to a starting symbolic pattern, after which token usage broadens across the compact sequence. This observation is important for the compositionality analysis. Whole-sequence edit and Jaccard metrics do not reveal where the overlap occurs inside the decoded string. Some overlap may come from shared low-entropy starting patterns rather than from a shifted preservation of the acoustic content common to adjacent windows. Therefore, high adjacent-window overlap should not automatically be interpreted as evidence that a tokenizer performs a clean local shift of the symbolic sequence. Conversely, lower overlap for PairAlign should not automatically be interpreted as a loss of structure, because PairAlign operates with fewer positions, stronger early trajectory commitment, and broad token usage through the rest of the sequence. What the sweep analysis does and does not test. The continuous-sweep analysis should not be interpreted as testing strict shift-equivariant token behavior. In the strongest form of local compositionality, a 100 ms shift would transform a token sequence approximately as (a, b, c, d) → (b, c, d, e), where tokens corresponding to the shared acoustic region are preserved as an interior subsequence, the exiting boundary material is removed, and the entering boundary material is added at the appropriate end. The present experiment does not verify this stronger property. It compares whole decoded sequences using edit similarity, Jaccard overlap, length change, edit distance, and edit-operation counts. These metrics measure whether adjacent windows remain symbolically close, but they do not localize which token positions correspond to the shared, entering, or exiting acoustic regions. This distinction matters because the beginning of the decoded sequence may be special for both tokenizers. The token-inventory analysis shows that early positions have lower entropy and higher dominant-token mass in both systems, with PairAlign showing an even stronger early commitment. In both cases, apparent agreement near the beginning of the sequence may partly reflect common starting patterns, not necessarily a localized mapping from shared acoustic subregions to shifted token subsequences. Therefore, the sweep analysis supports a weaker but still meaningful claim: tokenizations of neighboring windows remain related under fine-grained context motion. It does not prove token-to-phone alignment, strict subsequence tracking, or shift-equivariant behavior of the form (a, b, c, d) → (b, c, d, e). Establishing that stronger claim would require additional analyses that align token positions to time, inspect whether preserved subsequences move by the expected offset, or use the post-hoc timing recovery mechanism to test whether edits concentrate near the entering and exiting boundary regions. The present analysis is therefore best understood as a structural stability probe, not as a proof of localized compositional segmentation. Why normalized rates and absolute counts tell different stories. The normalized operation rates are higher for PairAlign in all three categories: substitution rate is 0.314 for PairAlign versus 0.253 for the geometric tokenizer; insertion rate is 0.127 versus 0.104; and deletion rate is 0.145 versus 0.107. At first glance, this might suggest that PairAlign is less stable. However, these rates are normalized by sequence length, and PairAlign emits roughly 30% as many tokens as the geometric tokenizer. A small or moderate absolute number of edits therefore occupies a much larger fraction of the PairAlign sequence. This is precisely why both absolute and normalized measures are necessary. The normalized rates show that, at PairAlign’s own symbolic scale, a 100 ms shift affects a substantial fraction of the compact sequence. The absolute counts show that the actual number of symbolic operations remains much smaller than for the geometric tokenizer. These two observations are not contradictory. They indicate that PairAlign is a lowrate tokenizer whose symbols carry more context per token. When one such symbol changes, the normalized penalty is large, but the edit trajectory can still be bounded and interpretable. The inventory results sharpen this interpretation. PairAlign’s lower normalized overlap is not accompanied by vocabulary collapse or a narrow token distribution. Instead, PairAlign combines stronger early-position 81
commitment with broad token usage across the compact sequence. The sweep results should therefore be read as evidence of a compactness–locality trade-off: PairAlign gives up dense frame-level token reuse, but retains a broad symbolic inventory and bounded local edit behavior. Token-overlap metrics and sequence scale. The edit and Jaccard similarities measure whether the same token identities and local ordering are preserved across adjacent windows. On these overlap-based metrics, the geometric tokenizer is stronger. This is expected because it produces a much denser sequence and remains tied to local frame-level assignments. When adjacent windows share 2.9 seconds of audio, many frame-derived token assignments can persist, even after deduplication. PairAlign has lower edit and Jaccard similarity, but this should not be read as poor sweep behavior. These metrics are strongly affected by sequence length and token count. PairAlign emits only 25.48 tokens per 3-second window on average, compared with 85.08 tokens for the geometric tokenizer. In such a short sequence, each substitution, insertion, or deletion has a larger effect on normalized edit similarity, and each token identity change has a larger effect on unigram Jaccard. The lower overlap scores therefore reflect both genuine context sensitivity and the compact sequence scale at which PairAlign operates. Importantly, PairAlign still obtains a mean edit similarity of 0.414, and 52.2% of adjacent pairs have edit similarity at least 0.4. Given the short length of the PairAlign sequences, this indicates nontrivial symbolic continuity under 100 ms window motion. The tokenization is not being re-formed arbitrarily from one window to the next. Rather, a meaningful portion of the ordered token sequence is preserved, while a bounded number of token identities and local sequence positions adapt to the boundary content entering or leaving the window. Length change versus insertion–deletion behavior. The difference between |∆L| and the insertion/deletion counts is also informative. The net length change for PairAlign is moderate: median |∆L| = 4 and 61.6% of adjacent pairs satisfy |∆L| ≤ 5. The insertion and deletion distributions show that this is not merely the result of large compensating insertion–deletion events that cancel in length. Instead, insertions and deletions themselves are much smaller than for the geometric tokenizer: 77.1% of transitions have I ≤ 5 and 74.3% have D ≤ 5. Therefore, PairAlign’s length stability reflects controlled token-rate changes, not only cancellation between large opposing edit operations. For the geometric tokenizer, both net length changes and insertion/deletion counts are larger. The mean insertion and deletion counts are 9.56 and 9.79, respectively, and only about 35.7% and 35.6% of transitions have I ≤ 5 and D ≤ 5. This indicates that the dense geometric sequence undergoes more frequent token birth/death events under the same local sweep. Some of this behavior is expected from a frame-derived representation: a small boundary shift can alter many local nearest-centroid assignments after deduplication. However, it also means that the geometric tokenizer’s higher normalized overlap comes with substantially larger absolute edit activity. Compositionality at different symbolic granularities. The continuous-sweep experiment should therefore be interpreted as a granularity-sensitive probe of compositionality. A strong, fine-grained form of compositionality would require the shared acoustic region between adjacent windows to map to a nearly unchanged token subsequence, while the entering and exiting 100 ms regions would correspond to localized insertions or deletions. The present metrics do not establish this strong form. They do not check whether a token string shifts as (a, b, c, d) → (b, c, d, e), nor whether preserved tokens occupy the expected interior positions after the acoustic window moves. They only measure whole-sequence similarity, length behavior, and edit cost. Under these whole-sequence metrics, the geometric tokenizer is closer to fine-grained local persistence. Its symbols remain tied to local frame-derived assignments, and adjacent windows share most of their frames. Its higher edit and Jaccard similarities therefore reflect stronger local token reuse. However, the position-wise inventory analysis cautions against interpreting this as direct evidence of clean subsequence shifting. The geometric tokenizer also has low-entropy starting positions, so part of its apparent overlap can arise from common initial token patterns rather than from a localized acoustic-to-symbolic shift. 82
PairAlign appears to operate at a coarser symbolic granularity. Its tokens are fewer, and individual token identities may not correspond to single short acoustic fragments. Instead, ordered combinations of tokens may encode broader phonetic structure. Under a 100 ms shift, the relevant combination can change even if the underlying representation remains structured. This makes strict token-level overlap a demanding criterion for PairAlign. The observed edit similarity in the 0.4 range, together with smaller absolute length changes and bounded edit-operation counts, is therefore consistent with a weaker but still meaningful form of local compositionality: neighboring windows preserve part of the symbolic sequence, while the token composition adapts through a moderate number of substitutions and smaller insertion/deletion changes. This is not a proof that PairAlign learns phonemes or word-like units. The experiment does not explicitly align token subsequences to phone spans, nor does it establish that every local acoustic change maps to a localized symbolic operation. However, it does show that the learned tokenization is not purely holistic. If PairAlign were assigning unrelated segment-level codes to each window, adjacent 100 ms shifts would produce much lower symbolic continuity, less controlled length behavior, and larger or more erratic editoperation bursts. Instead, PairAlign retains measurable edit similarity, more controlled sequence length, and a bounded edit script while operating at a substantially lower token rate. Relation to the geometric tokenizer. The comparison with the geometric tokenizer therefore reveals a difference in symbolic organization rather than a simple dominance relation. The geometric tokenizer provides stronger fine-grained local persistence under overlap-based metrics. This is useful for normalized edit similarity, Jaccard similarity, and rank-1 retrieval, and it is expected from a representation derived from local nearest-centroid assignments. However, this comes with a much longer symbolic sequence, substantially larger absolute length variation, and many more edit operations under the same 100 ms sweep. PairAlign provides a different operating point. It uses far fewer tokens, maintains stronger absolute length control, and requires fewer edit operations between adjacent windows. Its lower edit and Jaccard overlap indicate that token identities are more context-sensitive at the compact symbolic scale, but the edit-operation statistics show that this context sensitivity is bounded. The dominant change mode is substitutions, with smaller insertion/deletion counts than the geometric tokenizer. This is consistent with a coarser symbolic decomposition in which combinations of tokens, rather than individual frame-local symbols alone, carry the relevant phonetic information. The token-inventory analysis further shows that this coarser organization is not a collapse mode. PairAlign commits more strongly at the earliest decoding positions, but it does not remain restricted to a small set of symbols throughout the sequence. Later positions show broader token usage, and the global inventory remains active under TIMIT evaluation. Thus, PairAlign’s lower local-overlap scores should be interpreted together with its shorter length, bounded edit operations, and non-collapsed token inventory. Connection to retrieval. These sweep results clarify the retrieval behavior reported earlier. Retrieval is performed on the same TIMIT (Garofolo et al., 1993) corpus but with a 1.5-second hop, whereas sweep uses a 100 ms hop. The retrieval setting therefore evaluates larger shifts in continuous-speech content, while the sweep setting evaluates fine-grained local sequence evolution. A dense geometric tokenizer is naturally strong under strict local-overlap and rank-1 matching criteria because its token sequence retains fine-grained framederived structure. PairAlign, by contrast, is less redundant and operates at a coarser symbolic granularity. As a result, small changes in token identity or ordering can have a larger effect on normalized edit similarity, which can reduce exact nearest-neighbor sharpness under strict rank-1 or local normalized-overlap metrics. At the same time, the sweep analysis shows that PairAlign is not changing arbitrarily. Most adjacent transitions remain within a moderate edit distance, and length changes are smaller in absolute token count than for the geometric tokenizer. This helps explain why PairAlign remains useful for edit-distance retrieval despite lower local token-overlap similarity: its compact sequences preserve enough ordered symbolic structure to support matching, while substantially reducing the number of stored archive tokens. Thus, the retrieval and sweep analyses support a consistent interpretation. The geometric tokenizer behaves more like a dense framesynchronous symbolic trace, preserving stronger fine-grained local overlap. PairAlign is better understood as a compact sequence-level retrieval substrate: it is not intended to maximize token reuse at every 100 ms 83
shift, but to learn a lower-rate symbolic interface whose sequence structure remains stable, discriminative, and useful under edit-based comparison. Overall interpretation. The continuous-sweep analysis shows that PairAlign behaves consistently under fine-grained context shifts, but at a different symbolic scale from the geometric tokenizer. With a 100 ms hop, adjacent windows differ only in a small boundary region while sharing 2.9 seconds of acoustic context. In this setting, PairAlign changes its decoded length by a median of 4 tokens, compared with 7 tokens for the geometric tokenizer. Nearly half of PairAlign transitions stay within three tokens (|∆L| ≤ 3), and 61.6% stay within five tokens. The corresponding values for the geometric tokenizer are much lower, 27.1% and 41.0%, respectively. This indicates that PairAlign has stronger absolute length control under small window shifts. The edit-operation decomposition strengthens this conclusion. PairAlign has a median edit distance of 15 operations between adjacent windows, compared with 36 for the geometric tokenizer. Moreover, 65.8% of PairAlign transitions require at most 20 edit operations, whereas only 13.2% of geometric-tokenizer transitions satisfy the same bound. PairAlign’s edits are dominated by substitutions, with substantially smaller insertion and deletion counts than the geometric tokenizer. This suggests that the compact sequence usually changes through bounded context-sensitive relabeling and limited token birth/death, rather than through large unstable re-tokenization events. The overlap-based metrics tell a complementary story. The geometric tokenizer obtains higher edit and Jaccard similarity, reflecting its denser frame-derived structure and stronger fine-grained local persistence. PairAlign obtains lower overlap scores, but it also represents each 3-second window with far fewer tokens. The token-inventory analysis is essential for interpreting this result. Both tokenizers show low-entropy starting positions, and PairAlign commits even more strongly at the beginning of the sequence. Therefore, wholesequence overlap should not be treated as direct evidence of strict shifted-subsequence behavior. The sweep experiment does not show that token sequences transform as (a, b, c, d) → (b, c, d, e), nor does it localize edits to the entering and exiting acoustic boundaries. What the sweep experiment does show is more modest but still important. PairAlign retains nontrivial symbolic continuity, smaller absolute length changes, fewer edit operations, and limited token birth/death under 100 ms window motion, while maintaining broad token usage under TIMIT evaluation. This rules out the interpretation that its compactness comes from holistic segment-level codes or vocabulary collapse. The appropriate interpretation is therefore that PairAlign learns a coarser sequence-level symbolic interface: one with stronger early autoregressive commitment, broad token usage over the compact sequence, bounded local edit trajectories, and sufficient ordered structure for edit-distance retrieval. It does not preserve dense framelevel redundancy as strongly as the geometric tokenizer, but it provides a lower-rate symbolic representation whose sequence structure remains meaningful at a coarser granularity.
6
Conclusion
This paper introduced PairAlign, a framework for learning compact symbolic token sequences from continuous audio through sequence-level self-alignment. The central premise is that tokenization should not be treated only as a local assignment problem. In many existing audio tokenizers, discrete symbols are induced by quantizing encoder frames, clustering self-supervised features, or reconstructing acoustic latents through codec objectives. These approaches have produced powerful neural audio codecs, semantic speech units, hierarchical semantic–acoustic tokenizers, and token streams for audio language models. However, the global behavior of the resulting symbolic sequence—its length, ordering, stability across realizations, edit-distance geometry, and usefulness for comparison—is usually inherited indirectly from local quantization, encoder stride, reconstruction losses, or downstream post-processing. PairAlign addresses this gap by making the symbolic sequence itself the object of learning. PairAlign formulates tokenization as conditional sequence generation. An encoder maps a speech segment to a continuous conditioning representation, and an autoregressive decoder generates the complete symbolic tokenization from BOS. The decoder therefore learns not only which symbols to emit and in what order, but also how many symbols are needed and when to terminate with EOS. This is a fundamental departure 84
from fixed-stride, frame-synchronous, or CTC-collapse-based tokenization, where output length is largely inherited from input duration, encoder stride, or collapse rules. In PairAlign, sequence length is part of the learned symbolic interface. Given two content-preserving views of the same segment, the token sequence induced from one view is trained to receive high likelihood under the conditioning representation of the other. Unrelated examples provide competing symbolic sequences through in-batch likelihood contrast. Thus, PairAlign uses crossview predictability not merely to align continuous embeddings, but to induce and stabilize the discrete sequence interface itself: token identity, token order, output length, and termination are all shaped by the self-supervised alignment objective. This places PairAlign at a distinct point in the landscape of self-supervised learning. Masked-prediction and denoising methods learn by reconstructing hidden or corrupted content. Contrastive and non-contrastive multiview methods learn by making continuous representations agree across augmentations. EMA-teacher methods stabilize self-supervised targets through slowly moving teacher networks. JEPA-style predictive approaches emphasize learning representations by predicting abstract latent structure rather than reconstructing raw sensory detail. PairAlign is compatible with this broader predictive philosophy, but moves the prediction target into a different space. The target is not a waveform, a spectrogram, a masked frame, a continuous latent vector, or a fixed cluster label. It is an ordered symbolic sequence whose alphabet usage, length, and termination are themselves learned. In this sense, PairAlign can be interpreted as a sequence-symbolic analogue of JEPA-style learning: the model predicts an abstract target associated with another view, but the target is a learned variable-length token string rather than a fixed-dimensional continuous representation. Thus, PairAlign extends self-supervised learning from representation alignment to symbolic-interface induction. PairAlign is also related to neural sequence transduction, but changes the role of transduction. CTC and RNN-T learn to map an input sequence to an externally specified target sequence while marginalizing over unknown monotonic alignments. Attention-based encoder–decoder models learn soft input–output alignments for tasks such as speech recognition and translation. wav2tok brought this transduction view closer to retrieval-oriented tokenization by using a CTC-style sequence constraint to make paired speech views agree as token strings. PairAlign inherits this sequence-level motivation, but removes the assumption that the symbolic target is fixed by a frame-posterior tokenizer. The decoder directly generates the token sequence assigned to the input, and cross-paired likelihood is used to make paired realizations mutually predictable. The objective therefore shifts from explaining a given target sequence under unknown alignment to inducing the target sequence space itself. This distinction also separates PairAlign from standard language modeling, denoising sequence-to-sequence learning, and LLM alignment. Unlike GPT-style continuation, PairAlign does not continue a prompt. Unlike BART-style denoising, it does not reconstruct a corrupted symbolic source. Unlike LLM post-training alignment, it does not adjust behavior over an already fixed tokenizer using external preferences. At inference time, there is no text prompt, no partial acoustic-token prefix, and no externally supplied target-side context. The decoder starts from BOS and produces the tokenization assigned to the acoustic input. Alignment in PairAlign therefore has a different meaning: it is not post-hoc policy shaping over a fixed symbolic interface, but a representation-learning principle for constructing that interface. The experimental design is intended to expose this distinction rather than obscure it. A conventional evaluation of tokenizers might report only reconstruction, classification, or retrieval scores. Here, we instead evaluate the symbolic interface from several complementary angles: cross-view token consistency, collapse behavior, token-inventory usage, position-wise entropy, edit-operation decomposition, long-form retrieval, and continuous-sweep behavior under small temporal shifts. This suite of experiments is central to the contribution. It tests not only whether PairAlign produces short sequences, but whether those sequences remain stable, discriminative, non-degenerate, and meaningful under edit-based comparison. The discrete token consistency experiments show that PairAlign improves or preserves order-sensitive agreement between noisy content-preserving views while producing substantially shorter sequences. The collapse diagnostics then rule out the simplest failure mode: the gains are not achieved by mapping many inputs to low-diversity or repeated strings. The token-inventory analysis further strengthens this conclusion. PairAlign
85
maintains broad vocabulary usage, high entropy, and nontrivial position-wise diversity, including under crosscorpus TIMIT evaluation. Its position-wise behavior reveals a characteristic autoregressive symbolic regime: stronger early input-conditioned commitment followed by broader token usage across the compact sequence. This is precisely the kind of diagnostic needed for a learned tokenizer whose outputs are generated from BOS, because high agreement is meaningful only if it is not obtained through decoder bypass or genericprefix collapse. The retrieval and continuous-sweep experiments reveal the operating point of the learned representation. The geometric tokenizer remains stronger for dense local overlap and sharp rank-1 retrieval, as expected from a frame-derived symbolic trace. PairAlign instead trades part of this fine-grained redundancy for a lowerrate sequence-level interface. In retrieval, it reduces the archive token count substantially while retaining meaningful edit-distance search behavior, full coverage, and typical first relevant ranks at the top of the list. In continuous sweep, it has lower normalized adjacent-window overlap, but its absolute length changes and edit-operation trajectories are more bounded than those of the geometric tokenizer. These results do not prove phoneme-like units or strict shift-equivariant behavior of the form (a, b, c, d) → (b, c, d, e). Rather, they show that the learned sequence is not a holistic segment code: it changes coherently, uses a broad inventory, controls its length, and remains comparison-friendly at a coarser symbolic granularity. The value of these experiments is that they make the trade-off visible. PairAlign is not presented as a tokenizer that dominates dense geometric tokenization on every metric. That would be the wrong standard for a lower-rate autoregressive symbolic interface. Instead, the experiments show what is gained and what is lost when tokenization moves from frame-level assignment to sequence-level self-alignment. The geometric tokenizer preserves local redundancy and native timing. PairAlign gives up some of that redundancy, but obtains shorter sequences, learned length control, bounded edit trajectories, stronger explicit sequence modeling, and a symbolic space whose consistency is shaped directly by cross-view likelihood. This is the more informative comparison: not a claim of universal dominance, but a characterization of a new operating regime. PairAlign should therefore be understood as complementary to existing audio tokenizers, not as a replacement for all of them. Codec tokenizers optimize a discrete interface for reconstruction, compression, and downstream generation. Semantic tokenizers improve invariance and linguistic abstraction. Hierarchical and disentangled tokenizers organize information across code streams. Retrieval-oriented geometric tokenizers preserve localization-friendly frame-synchronous structure. Audio language models demonstrate the value of modeling over discrete audio tokens once such a tokenizer exists. PairAlign asks a different question: can the symbolic sequence interface itself be learned by self-alignment, so that compactness, order, length, consistency, and discriminability are optimized directly in token space? The results suggest that the answer is yes, with a clear trade-off between dense local persistence and compact sequence-level organization. The limitations of this formulation are equally important. Because PairAlign generates compact autoregressive sequences, it does not provide native frame-to-token timing in the way a frame-synchronous tokenizer does. The present timing recovery procedure is post-hoc and approximate. The compact representation also reduces redundancy, which can weaken strict rank-1 retrieval and local normalized-overlap metrics. The continuous-sweep analysis demonstrates bounded symbolic change under small context shifts, but does not establish localized phonetic segmentation or exact subsequence tracking. Future work should therefore explore monotonic or alignment-aware decoders, stronger temporal grounding, multilingual and lexically disjoint evaluation, adaptive-rate symbolic coding, larger-scale audio domains, and downstream systems that exploit the learned symbolic interface for retrieval, generation, editing, memory, and reasoning. The broader contribution of PairAlign is to reframe tokenization as a self-supervised sequence-transduction problem. It uses the predictive structure of multiview learning, the stabilizing logic of teacher-based selfdistillation, and the expressive machinery of neural transduction, but directs them toward a different object: the discovery of a compact symbolic language for continuous signals. The learned tokens are not merely local codebook indices, reconstruction residuals, clustered SSL units, or fixed language-model symbols. They are induced by the requirement that different realizations of the same underlying signal remain mutually predictable as ordered sequences, while different signals remain separable. This is the core novelty of PairAlign: alignment is used not after the symbolic interface is given, but to construct the symbolic interface itself.
86
In this view, PairAlign is a step toward a next generation of self-supervised predictive and neural transduction methods. The aim is not only to learn representations in continuous latent space, nor only to transduce inputs into externally defined label sequences. The aim is to learn the symbolic coordinates in which continuous perceptual experience can be compressed, compared, searched, aligned, generated from, and eventually reasoned over. Viewed through the lens of JEPA-style learning, PairAlign replaces continuous latent prediction with symbolic sequence prediction: it predicts an abstract object associated with another view, but that object is the token interface itself. PairAlign shows that such coordinates can be induced without transcripts, without reconstruction as the primary objective, and without a fixed tokenizer as a prerequisite. They can emerge from cross-view sequence predictability, discriminative self-alignment, learned sequence length, and controlled autoregressive token generation.
7
Broader Impact
PairAlign is motivated by a broader question that extends beyond audio tokenization: how can continuous signals be converted into compact symbolic interfaces without relying on human annotation, reconstruction alone, or a fixed pre-existing vocabulary? Many domains contain signals that are naturally continuous, high-dimensional, and temporally structured: speech, music, environmental sound, video, motion, biological recordings, neural activity, sensor streams, physical simulations, robot trajectories, medical signals, and multimodal interaction traces. In such settings, the central challenge is not only to learn useful continuous embeddings, but also to discover discrete structures that are stable enough to compare, store, retrieve, compose, transmit, and reason over. PairAlign contributes to this direction by showing that self-supervised learning can be used not only to align representations, but also to induce the symbolic sequence interface itself. The immediate impact is in audio and speech. A compact, self-aligned token sequence can support retrieval, indexing, matching, and comparison over long audio archives without requiring transcripts. This is relevant for spoken document search, low-resource speech processing, acoustic monitoring, speech analytics, and audio organization in settings where manual labels or reliable ASR are unavailable. Because PairAlign learns from content-preserving views rather than explicit transcripts, it is naturally aligned with low-resource and multilingual directions, where the cost of annotation is high and the linguistic structure of the data may not be fully captured by existing supervised systems. The learned-length property of PairAlign also suggests impact in low-rate and adaptive-rate coding. Unlike fixed-stride tokenizers, PairAlign can in principle decide how many symbols to allocate to a segment. This opens the possibility of symbolic codecs that spend more tokens on complex or information-dense regions and fewer tokens on redundant regions. Such representations need not optimize only waveform reconstruction. For many applications, the relevant objective is relational: preserve what is needed for retrieval, indexing, recognition, generation, alignment, or reasoning. PairAlign therefore points toward semantic and relational compression, where the goal is not simply to reproduce the signal sample by sample, but to preserve the structure needed for downstream use. The same principle is relevant to generation. Modern audio, speech, music, and multimodal generative models increasingly rely on discrete token interfaces. In most current systems, the tokenizer is trained separately and then treated as a fixed input to a language model, diffusion model, or multimodal generator. PairAlign suggests a complementary direction: learn token sequences whose structure is already shaped by sequence-level consistency, compactness, learned termination, and cross-view predictability. Such tokenizers may provide more stable symbolic substrates for generative audio models, controllable editing systems, multimodal agents, and long-context audio-language models. The goal is not merely to reduce bitrate, but to learn tokens whose sequence geometry is meaningful for downstream modeling. More broadly, PairAlign points toward a new generation of self-supervised predictive methods in which the output space is not assumed in advance. Classical SSL often learns continuous representations by predicting masked content, future states, clustered targets, or teacher embeddings. JEPA-style learning sharpens this predictive view by emphasizing prediction of abstract representations rather than reconstruction of raw sensory detail. PairAlign can be interpreted as a sequence-symbolic analogue of this principle: it predicts an abstract target associated with another view, but the target is a learned variable-length token sequence 87
rather than a fixed-dimensional continuous latent. This changes the role of self-supervision. The model is not only learning invariance or latent predictability; it is learning the discrete coordinates in which future comparison, compression, memory, retrieval, and reasoning may occur. The same idea could be explored for video, robotics, biological recordings, wearable-sensor streams, climate signals, physical systems, and scientific time-series data, where meaningful symbolic units are rarely given a priori. This perspective is especially important for multimodality. Multimodal systems require interfaces through which different sensory streams can be compared, fused, retrieved, translated, generated, and acted upon. Text already has a discrete symbolic interface; audio, vision, motion, and physical signals usually do not. A method that learns compact, aligned symbolic sequences from continuous data could help reduce this asymmetry. Rather than forcing all modalities into text-like labels or dense embeddings, future systems may learn modality-specific or shared symbolic interfaces that are self-aligned, comparable, composable, and adaptive in length. Viewed through the lens of JEPA-style predictive learning, this suggests a shift from predicting only continuous latent states to predicting structured symbolic event sequences across views, time, tasks, or modalities. PairAlign is a step in this direction: it treats the token sequence not as an engineering byproduct, but as a learned bridge between continuous perception and symbolic computation. The same idea may also influence how future audio-language and multimodal models represent intermediate reasoning. Different tasks can require different surface outputs while sharing underlying structure. For example, answering a spoken question, retrieving a relevant audio segment, summarizing an acoustic scene, and generating an explanation may involve related intermediate abstractions even when their final outputs differ. PairAlign suggests that such intermediate structures could themselves be aligned: views, modalities, or tasks that refer to the same underlying event could be trained to induce mutually predictable symbolic or latent reasoning traces. This would extend self-supervised learning from representation alignment to process alignment, where models learn not only what to represent, but how to organize the sequence of abstractions used for perception, generation, and reasoning. The potential scientific impact is also significant. Many scientific fields seek compact descriptions of complex continuous phenomena. In neuroscience, one may ask whether neural activity contains reusable symbolic patterns across trials, subjects, or behavioral contexts. In biology, one may seek discrete event structures in continuous cellular, physiological, or behavioral measurements. In physics and climate science, one may wish to identify recurring regimes, transitions, or dynamical motifs in long continuous signals. In medicine, one may want stable symbolic summaries of ECG, EEG, ultrasound, or other time-varying measurements. PairAlign-like objectives could provide a general framework for discovering ordered symbolic abstractions without requiring the symbolic vocabulary or sequence length to be manually specified. The broader methodological implication is that predictive learning need not end at continuous latent states: it can also be used to induce discrete symbolic interfaces that summarize the structure of complex temporal phenomena. At the same time, the societal implications require care. Self-supervised tokenization can make large-scale audio and multimodal data easier to search and index. This can benefit accessibility, archival search, education, scientific discovery, low-resource language technology, and communication-efficient processing. However, it can also increase the ease of surveillance or unauthorized analysis of private recordings. Any deployment of retrieval-oriented audio tokenization should therefore respect privacy, consent, data governance, and domainspecific regulations. Compact symbolic representations are not automatically anonymous; they may still retain information about speech content, speaker identity, environment, or other sensitive attributes. There are also risks in generative use. Better symbolic interfaces can improve controllable generation, editing, and multimodal synthesis, but the same capabilities could be misused for deceptive audio generation, impersonation, or manipulation. PairAlign itself is not a generative audio model, but it may contribute to future generative pipelines by providing more structured audio tokens. Responsible use therefore requires pairing such methods with provenance tracking, watermarking where appropriate, consent-aware datasets, and safeguards against impersonation or unauthorized voice use. The limitations of the current work should temper its broader claims. PairAlign is demonstrated on speech tokenization and retrieval-oriented analysis, not on every modality or downstream task. The learned tokens are not proven to correspond to phonemes, words, events, or human-interpretable units. The current formulation also trades native frame-level timing for compact sequence-level structure. Thus, the broader
88
impact of PairAlign is best understood as methodological rather than immediately universal: it proposes a way to learn symbolic interfaces from continuous data, but each domain will require its own validation, interpretability analysis, privacy analysis, and safety considerations. Overall, PairAlign suggests a direction in which self-supervised learning moves beyond continuous representation learning alone. Instead of only mapping continuous signals into continuous embeddings, future systems may learn discrete symbolic interfaces that are compact, stable, predictive, adaptive in length, and useful for comparison across views. In JEPA-style terms, PairAlign replaces continuous latent prediction with symbolic sequence prediction: the abstract object predicted across views is not only a representation, but the token interface itself. Such interfaces could become important building blocks for retrieval, generation, multimodal reasoning, scientific discovery, communication-efficient coding, and human-centered AI. The broader promise is that symbolic structure need not always be imposed from outside the data. It can be induced by the predictive relationships among different realizations of the same underlying phenomenon.
8
Future Work
PairAlign opens several directions for future research in sequence-level tokenization, self-supervised symbolic representation learning, and low-rate perceptual coding. The present work establishes a first step: a tokenizer whose symbolic sequence is not only inherited from local quantization, but shaped directly by cross-view predictability, learned length control, and discriminative sequence-level alignment. Future work can extend this principle along several complementary axes. Learning sequence length as part of the symbolic interface. A distinctive property of PairAlign is that output sequence length is not fixed by encoder stride, frame rate, or a CTC-style collapse rule. The decoder starts from BOS, emits a variable-length token sequence, and decides when to terminate with EOS. Thus, PairAlign learns not only token identities and token ordering, but also how many symbols are needed to represent a segment. This makes length itself part of the learned symbolic interface. Future work should study this property more directly: when does the model choose shorter or longer sequences, how does length correlate with acoustic, phonetic, semantic, or temporal complexity, and how can length decisions be regularized without forcing either under-compression or over-compression? Understanding learned length control is central to developing tokenizers that adapt their symbolic rate to the information content of the input. Scaling beyond paired corrupted views. The present work constructs paired views through controlled content-preserving augmentations. This provides a clean setting for studying sequence-level self-alignment, but it is not the only possible source of positive structure. A broader goal is to learn symbolic token spaces directly from large continuous audio streams, where related views are not explicitly given. Future work could discover positives through temporal recurrence, nearest-neighbor mining, speaker-normalized matching, weak semantic similarity, cross-modal co-occurrence, or teacher-based retrieval over large unlabeled corpora. This would move PairAlign toward a more general self-supervised setting in which the model discovers not only the token sequence, but also the correspondence structure that should organize the symbolic space. Learning symbolic interfaces at scale. PairAlign currently studies short continuous speech segments, which provide enough context for nontrivial autoregressive tokenization while keeping training and evaluation controlled. A natural next step is to scale the method to longer audio, more diverse acoustic domains, multilingual speech, music, environmental sound, and mixed audio scenes. Scaling raises new questions: how should symbolic length grow with signal duration, how should EOS be controlled over long contexts, and how should the model avoid both over-compression and redundant local tokenization? Studying these questions may reveal whether sequence-level self-alignment can produce stable symbolic interfaces for largescale audio and multimodal foundation models. Low-bitrate symbolic coding of perceptual data. One of the most promising directions is low-bitrate symbolic coding. Most low-bitrate audio codecs optimize reconstruction fidelity under waveform, spectral, or perceptual losses. PairAlign suggests a complementary objective: optimize the symbolic code for consistency, 89
comparability, learned length control, and predictive structure across views. In this setting, bitrate is not only determined by frame rate or codebook size; it is also determined by the number of tokens the model chooses to emit. Because PairAlign learns termination, it can potentially allocate a variable number of symbols to different inputs. This opens a different design space for compression. The goal need not always be high-fidelity waveform reconstruction. For retrieval, indexing, monitoring, multimodal grounding, agent memory, and scientific analysis, it may be more useful to preserve content-level relations at very low symbolic rate. Future work could therefore study PairAlign-like tokenizers as semantic or relational codecs: compact representations optimized to preserve what an application needs to compare, search, generate, or reason over, rather than everything required to reconstruct the original signal. Rate–distortion objectives for learned symbolic sequences. Future work should make the compactness–fidelity trade-off explicit through rate–distortion-style objectives for symbolic tokenization. Here, the rate is the cost of the representation: the number of emitted tokens, the number of bits needed to store them, or the computational cost of modeling the sequence. The distortion is not limited to waveform reconstruction error. For PairAlign-style tokenizers, distortion can measure loss of cross-view consistency, edit-distance stability, retrieval accuracy, phonetic or semantic content, downstream task performance, or generative controllability. This view changes the question from “how well can we reconstruct the waveform at a given bitrate?” to “how few learned symbols are needed to preserve the relations that matter?”. For retrieval, the relevant distortion may be degradation in nearest-neighbor ranking. For generation, it may be loss of controllability or long-range coherence. For multimodal grounding, it may be loss of alignment between audio, vision, text, and action. For reasoning systems, it may be loss of task-relevant state information. A rate–distortion formulation would therefore allow PairAlign to become an adaptive-rate symbolic tokenizer: simple or redundant inputs could receive shorter sequences, while complex or information-dense inputs could receive longer ones. Adaptive-rate and learned-length coding. Because PairAlign learns EOS placement, it naturally suggests adaptive-rate coding. Unlike a fixed-stride tokenizer, which emits a predetermined number of symbols per second, PairAlign can in principle allocate more tokens to complex segments and fewer tokens to simple or redundant ones. Future work could explicitly regularize this behavior by adding length costs, entropy costs, or information-budget constraints to the self-alignment objective. The goal would be to learn tokenizers that spend symbolic capacity only when it improves consistency, retrieval, generation, or downstream reasoning. Such adaptive learned-length coding may be especially useful for long-form audio, multimodal streams, and scientific time-series data, where information density varies substantially over time. Semantic and task-aware compression. PairAlign compresses audio into symbolic sequences without requiring transcripts or labels. Future work could study whether different kinds of information can be selectively preserved under self-alignment. For example, one tokenizer might preserve lexical content while discarding speaker and channel variation; another might preserve speaker identity, prosody, acoustic scene information, or affective cues. This would connect PairAlign to semantic compression: instead of asking only how many bits are needed to reconstruct the signal, we can ask how many bits are needed to preserve a chosen relation over signals. Such task-aware symbolic coding may be useful for privacy-preserving indexing, low-resource speech search, acoustic monitoring, and multimodal memory systems. Tighter sequence-level objectives. In this work, edit-distance preservation is encouraged indirectly through cross-paired conditional likelihood and likelihood contrast. This makes training tractable, but it does not optimize edit distance directly. Future work may explore differentiable relaxations of edit distance, soft dynamic-programming losses, optimal-transport sequence losses, edit-aware contrastive objectives, or minimum Bayes risk criteria over generated token strings. Such objectives could more directly control the geometry of the symbolic sequence space. They may also reduce the gap between teacher-forced likelihood and free-running tokenization quality. 90
Beyond edit distance: richer symbolic geometries. Edit distance is a useful first metric because it captures order-sensitive sequence similarity. However, learned token spaces may support richer geometries. Future work could study n-gram similarity, subsequence kernels, tree-structured edit distances, alignmentaware token embeddings, learned sequence metrics, or hybrid symbolic-continuous similarity functions. This is especially important if PairAlign tokens encode broader acoustic or phonetic regions rather than framelocal units. A mature theory of learned perceptual tokenization should distinguish when token identity, token order, token subsequence structure, token length, and token timing each matter. Architectures with stronger alignment structure. The present PairAlign instantiation uses an unconstrained Transformer decoder with cross-attention and recovers approximate temporal grounding post hoc. Future work should study architectures with stronger monotonic or alignment-aware inductive biases. Monotonic attention, monotonic chunkwise attention, hard monotonic transduction, transducer-style decoders, or duration-explicit decoders could make token-to-time correspondence part of the model rather than a post-processing step. Such architectures may be especially important for segmentation, keyword localization, streaming retrieval, and applications where compact symbolic tokens must remain temporally grounded. Streaming and online PairAlign. A compact autoregressive tokenizer is attractive for offline indexing, but many applications require online operation. Future work could develop streaming PairAlign variants that emit tokens incrementally as audio arrives. This would require controlling latency, revising EOS or boundary decisions, and preventing unstable re-tokenization when additional context becomes available. Streaming PairAlign would connect sequence-level tokenization to online speech search, real-time acoustic monitoring, on-device indexing, and low-bandwidth communication. Hierarchical and multi-granular tokenization. The current formulation produces a single token sequence for each input segment. A natural extension is to learn hierarchical or multi-stream symbolic representations operating at different temporal and semantic scales. For speech, one stream might capture short-term phonetic structure, another might capture longer-range lexical or prosodic organization, and another might encode speaker or channel information. For music, streams could separate rhythm, timbre, melody, harmony, and performance style. For video or robotics, streams could represent local motion, objectlevel events, and long-horizon action structure. A hierarchical PairAlign model could therefore provide a bridge between low-level perception and high-level symbolic reasoning. Disentangled sequence-level tokenization. Related to hierarchical modeling is the possibility of disentangled symbolic sequences. Existing factorized audio tokenizers often separate information through codebook design, architectural factorization, or supervised factors. PairAlign suggests another route: different token streams could be aligned under different view transformations. For example, augmentations that preserve lexical content but alter speaker identity could train a content stream, while augmentations that preserve speaker identity but alter linguistic content could train a speaker stream. More generally, transformation-aware self-alignment could induce symbolic factors without requiring explicit labels for every factor of variation. Improved stabilization of autoregressive token learning. Autoregressive tokenization introduces failure modes that are different from ordinary frame-level quantization. The decoder may overuse target-side prefixes, emit generic high-probability starts, repeat tokens, collapse sequence length, or drift as the teacher evolves. PairAlign addresses these issues through cross-paired likelihood, prefix corruption, structured selfattention dropout, encoder-summary bias, EMA targets, and in-batch likelihood contrast. Future work could investigate stronger stabilization methods: adaptive teacher update rates, confidence-filtered teacher targets, entropy-controlled decoding, contrastive decoding, curriculum schedules over sequence length, or explicit anti-repetition and anti-collapse regularizers. The goal is to make autoregressive token induction as reliable as conventional quantization, while retaining its sequence-level advantages. Better negative selection and hard-example mining. PairAlign uses in-batch likelihood contrast to prevent unrelated examples from becoming equally likely under the same conditioning representation. 91
Future work could improve this component by using semantically hard negatives, phonetic near-neighbors, speaker-matched negatives, cross-lingual negatives, or retrieval-mined confusions. The choice of negatives is likely to shape what the symbolic interface preserves. Easy negatives may only prevent collapse, whereas hard negatives may force the tokenizer to represent finer distinctions in phonetic, lexical, speaker, or acoustic structure. Sharper structural evaluation of learned token spaces. The continuous-sweep analysis probes local stability under small temporal shifts, but it does not prove strict shifted-subsequence behavior of the form (a, b, c, d) → (b, c, d, e), nor does it establish token-to-phone alignment. Future work should develop sharper structural probes. These could include token-to-time alignment diagnostics, localized edit attribution, subsequence tracking under controlled shifts, phoneme-boundary correlation, token mutual information with linguistic units, and intervention tests that modify specific acoustic regions and measure the resulting symbolic edits. Such evaluations would clarify whether learned tokens are holistic segment codes, coarse acoustic events, phonetic-like units, or something more abstract. Interpretable symbolic units. PairAlign does not assume that its tokens correspond to phonemes, syllables, words, or any human-defined unit. This is a strength, but it also creates an interpretability challenge. Future work should study what individual tokens, token n-grams, and token subsequences represent. Useful tools may include nearest-neighbor analysis, token activation maps, cross-attention timing, phonetic enrichment tests, clustering of token contexts, and controlled generation or editing experiments. The goal is not necessarily to force the tokens to match known linguistic categories, but to understand the symbolic organization that emerges from self-alignment. Multilingual and lexically disjoint evaluation. A key question is whether PairAlign learns languagespecific symbolic patterns or more general acoustic-phonetic structure. Future work should evaluate Englishtrained tokenizers on lexically disjoint languages, multilingual speech, and typologically diverse corpora. Strong performance in such settings would suggest that the tokenizer captures reusable acoustic structure beyond memorized English lexical patterns. Conversely, failure modes may reveal where language-specific sequence statistics dominate. Such experiments are important for understanding whether PairAlign can become a general speech-token interface rather than a corpus-specific retrieval code. Applications to sequence-level speech processing. In this work, retrieval serves as a downstream test of whether the learned token space is compact, stable, and comparison-friendly. Future work should study tasks that directly benefit from these properties: spoken-content search, archive indexing, segmentation, temporal alignment, query-by-example retrieval, keyword discovery, spoken term discovery, symbolic speech editing, and low-resource speech organization. PairAlign may also be useful for building compact memory systems over speech archives, where symbolic sequences provide a searchable intermediate layer between raw audio and text transcripts. Interfaces for generative audio models. Discrete audio tokens are now central to speech, music, and general-audio generation. Most generative systems model over token streams produced by a separately trained codec or semantic tokenizer. PairAlign opens the possibility of training tokenizers whose sequence geometry is already organized for generation: compact length, controlled termination, reduced redundancy, stable cross-view behavior, and broad token usage. Future work could use PairAlign tokens as inputs to audio language models, diffusion models, masked generative models, or editing systems. This would test whether sequence-level self-alignment improves not only retrieval, but also controllability, long-context consistency, and sample efficiency in generation. Aligning reasoning traces across tasks and modalities. PairAlign also suggests a broader direction for aligning intermediate reasoning or computation traces across tasks. In language and audio-language models, different tasks can require different surface outputs while sharing underlying reasoning structure. For example, answering a spoken question, summarizing an audio scene, retrieving a matching segment, and generating an explanation may require related intermediate steps even though their final outputs differ. A PairAlign-like objective could be used to align such intermediate traces: two views, tasks, or modalities 92
that express the same underlying event or problem should induce mutually predictable latent or symbolic reasoning sequences. This would extend PairAlign from perceptual tokenization to process alignment. Instead of aligning only final answers or output labels, future systems could align the internal sequence of abstractions used to solve related tasks. For audio-language models, this might mean aligning acoustic event tokens, textual rationales, retrieval steps, and generation plans. For general multimodal models, it could mean aligning reasoning traces across vision, audio, language, and action. The key idea is the same as in PairAlign: alignment is not applied only after a symbolic interface is fixed; it can help induce the intermediate symbolic structures through which models compare, explain, and act. From representation alignment to reasoning-trace alignment. A longer-term direction is to extend the PairAlign principle from perceptual tokens to reasoning traces. Current alignment methods often supervise final outputs, preferences, or answers. PairAlign points to a different possibility: align the intermediate symbolic sequences that connect inputs to outputs. Across two related tasks, two modalities, or two views of the same problem, the surface forms may differ, but the underlying reasoning trajectory may share structure. Future self-supervised systems could exploit this by training models so that the reasoning trace induced in one task is predictable from the representation or trace induced in another. This is especially relevant for audio-language and multimodal models. A spoken query, a visual scene, a text instruction, and an action trajectory may all refer to the same underlying situation. Rather than forcing them into a single fixed token vocabulary immediately, a PairAlign-style objective could encourage their latent symbolic traces to become mutually predictable. Such an approach would move self-supervision beyond invariance in embedding space toward alignment of structured internal computations. In this sense, PairAlign may be viewed as an early instance of a broader class of methods that learn not only what to represent, but how to organize the sequence of abstractions used for perception, retrieval, generation, and reasoning. Multimodal symbolic interfaces. The PairAlign principle is not specific to speech. A promising direction is to learn symbolic interfaces across audio, vision, text, video, touch, motion, and robotics. Text already has a discrete symbolic interface, while most perceptual modalities are represented as continuous embeddings or dense token grids. PairAlign-like objectives could help learn modality-specific or shared symbolic sequences that support retrieval, translation, grounding, generation, and reasoning. For example, an audio event, a video segment, and a textual description could be aligned through mutually predictable symbolic sequences without forcing all modalities into a single fixed vocabulary at the outset. Connection to predictive world models and JEPA-style learning. PairAlign can be viewed as a sequence-symbolic analogue of JEPA-style predictive representation learning. JEPA-style models emphasize prediction in representation space rather than reconstruction of raw sensory input: a context representation is trained to predict an abstract target representation of another view. PairAlign shares this predictive-abstraction principle, but changes the nature of the predicted object. Instead of predicting a fixed-dimensional continuous latent target, PairAlign predicts a learned, variable-length discrete token sequence produced from a paired view of the same underlying audio segment. This distinction is central to the framework. The target in PairAlign is not merely a latent vector; it has explicit sequence structure, an induced vocabulary, an EOS decision, and an edit-distance geometry. The model must therefore learn not only what information should be predictable across content-preserving views, but also how that information should be serialized into a compact symbolic form. In this sense, PairAlign extends the JEPA-like idea from continuous latent prediction to symbolic sequence prediction. This sequence-symbolic formulation provides affordances that standard latent-prediction objectives do not directly expose: indexability, length adaptivity, symbolic comparison, and compatibility with sequence-level retrieval. PairAlign therefore provides one concrete audio instantiation of symbolic JEPA-style learning, while also suggesting a broader class of predictive systems that learn compact discrete event sequences across time, views, tasks, or modalities. 93
Symbolic memory and retrieval for agents. Autonomous agents require memory systems that can store, retrieve, and compare past perceptual experience. Dense embeddings are useful, but they are often difficult to edit, compose, inspect, and search structurally. PairAlign-like tokenizers could provide compact symbolic traces of audio, video, motion, or interaction histories. These traces could support long-term memory, event-level retrieval, temporal abstraction, and cross-modal grounding. In this view, learned perceptual tokenization becomes a bridge between raw sensory streams and the symbolic memory structures needed for planning and reasoning. Scientific time-series and non-audio data. The same framework may be useful beyond machine learning benchmarks. Many scientific domains involve continuous temporal signals with unknown discrete structure: neural recordings, physiological signals, climate trajectories, seismic measurements, molecular dynamics, animal behavior, financial streams, and physical simulations. PairAlign-like self-alignment objectives could be used to discover compact symbolic motifs, regimes, or event sequences in such data. The goal would not be to impose a human-defined vocabulary, but to learn symbolic units that remain stable across different views, conditions, sensors, or experimental repetitions. Privacy-preserving and communication-efficient representations. Low-rate symbolic tokenization may enable communication-efficient and privacy-aware processing. Instead of transmitting raw audio or highdimensional features, devices could transmit compact token sequences optimized for a specific downstream relation, such as retrieval or event detection. However, this direction requires care: symbolic tokens may still encode sensitive information such as speech content, speaker identity, or environmental context. Future work should study what information PairAlign tokens preserve, what they discard, and how privacy constraints can be incorporated into the alignment objective. This may require adversarial objectives, factorized token streams, explicit rate limits on sensitive attributes, or controllable disentanglement of content and identity. Toward a general theory of self-supervised symbolic induction. PairAlign provides an audiogrounded formulation of symbolic induction: useful token sequences should be stable under contentpreserving acoustic variation, discriminative across unrelated utterances, compact in length, and organized by a sequence geometry such as edit distance. At the same time, the framework raises broader theoretical questions that go beyond the present audio instantiation. What properties must positive views preserve for cross-view prediction to induce stable symbols? What negatives are needed to prevent collapse without destroying invariance? How do sequence length, EOS placement, vocabulary size, entropy, and edit-distance geometry interact? When does cross-view predictability produce compositional symbolic units rather than holistic segment codes? How should rate, distortion, invariance, and discriminability be jointly formalized for learned token sequences whose length is itself a model output rather than a fixed consequence of frame rate? Developing such a theory would connect PairAlign-like systems to representation learning, information bottlenecks, predictive coding, neural transduction, semantic compression, and symbolic abstraction. Broader role of symbolic perceptual tokenization. More broadly, PairAlign raises the question of when perceptual signals should be processed through learned symbolic sequences rather than only through dense continuous representations. Continuous embeddings are powerful, but symbolic sequences offer a different set of affordances: they can be compared by edit operations, indexed, stored, compressed, composed, and inspected. The long-term opportunity is to learn symbolic interfaces that retain the flexibility of neural representation learning while gaining some of the structure of discrete languages. PairAlign is one step in that direction: a self-supervised method for inducing compact token strings whose identities, ordering, and lengths are shaped by prediction, alignment, and discrimination rather than by labels or reconstruction alone. Toward self-supervised symbolic systems. The long-term opportunity is to move from self-supervised representation learning to self-supervised symbolic system learning. PairAlign shows that a model can learn not only which symbols to emit, but also how many symbols are needed, how they should be ordered, and how they should remain predictable across related views. This suggests a broader class of methods in which symbolic interfaces, reasoning traces, memory codes, and multimodal event sequences are induced rather 94
than manually specified. Such systems would combine the flexibility of neural representation learning with the compactness, comparability, and compositional affordances of symbolic sequences.
References Alberto Abad, Eugénio Ribeiro, Fábio N Kepler, Ramón Fernández Astudillo, and Isabel Trancoso. Exploiting phone log-likelihood ratio features for the detection of the native language of non-native english speakers. In INTERSPEECH, pp. 2413–2417, 2016. Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023. Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15619–15629, 2023. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447–4455. PMLR, 2024. Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910.05453, 2019. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 12449–12460, 2020. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International conference on machine learning, pp. 1298–1312. PMLR, 2022. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. Adhiraj Banerjee and Vipul Arora. wav2tok: Deep sequence tokenizer for audio retrieval. In The Eleventh International Conference on Learning Representations, 2022. Adhiraj Banerjee and Vipul Arora. Enc-dec rnn acoustic word embeddings learned via pairwise prediction. In Proc. Interspeech 2023, pp. 1478–1482, 2023. Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021. Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. arXiv preprint arXiv:2404.08471, 2024. 95
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 2523–2533, 2023. Doğan Can and Murat Saraclar. Lattice indexing for spoken term detection. IEEE Transactions on Audio, Speech, and Language Processing, 19(8):2338–2347, 2011. Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132–149, 2018. Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912–9924, 2020. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650–9660, 2021. William Chan, Navdeep Jaitly, Quoc V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint arXiv:1508.01211, 2015. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PmLR, 2020. Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750–15758, 2021. Yi-Chen Chen, Sung-Feng Huang, Chia-Hao Shen, Hung-Yi Lee, and Lin-Shan Lee. Phonetic-and-semantic embedding of spoken words with applications in spoken content retrieval. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 941–948. IEEE, 2018. Chung-Cheng Chiu and Colin Raffel. Monotonic chunkwise attention. arXiv preprint arXiv:1712.05382, 2017a. Chung-Cheng Chiu and Colin Raffel. Monotonic chunkwise attention. arXiv preprint arXiv:1712.05382, 2017b. Jishnu Ray Chowdhury and Cornelia Caragea. Monotonic location attention for length generalization. In International Conference on Machine Learning, pp. 28792–28808. PMLR, 2023. Yu-An Chung, Chao-Chung Wu, Chia-Hao Shen, Hung-Yi Lee, and Lin-Shan Lee. Unsupervised learning of audio segment representations using sequence-to-sequence recurrent neural networks. In Proc. Interspeech, pp. 765–769, 2016. Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. arXiv preprint arXiv:1904.03240, 2019. Yu-An Chung, Hao Tang, and James Glass. Vector-quantized autoregressive predictive coding. arXiv preprint arXiv:2005.08392, 2020. Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pretraining. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244–250. IEEE, 2021. 96
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in neural information processing systems, 36:47704–47720, 2023. Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013. Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037, 2024. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186, 2019. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32, 2019. Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673, 2023. Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 591–595. IEEE, 2024. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017, 2020. Zhengcong Fei, Mingyuan Fan, and Junshi Huang. A-jepa: Joint-embedding predictive architecture can listen. arXiv preprint arXiv:2311.15830, 2023. John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett. Darpa timit acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1. NASA STI/Recon technical report n, 93:27403, 1993. Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711, 2012. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369–376, 2006. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020. Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021. 97
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), volume 2, pp. 1735–1742. IEEE, 2006. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729–9738, 2020. Wanjia He, Weiran Wang, and Karen Livescu. Multi-view recurrent neural acoustic word embeddings. arXiv preprint arXiv:1611.04496, 2016. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170–11189, 2024. Junfeng Hou, Shiliang Zhang, and Li-Rong Dai. Gaussian prediction based attention for online end-to-end speech recognition. In Proc. Interspeech 2017, pp. 3692–3696, 2017. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460, 2021. Yushi Hu, Shane Settle, and Karen Livescu. Acoustic span embeddings for multilingual query-by-example search. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 935–942. IEEE, 2021. Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel PW Ellis. Mulan: A joint embedding of music audio and natural language. arXiv preprint arXiv:2208.12415, 2022. Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117–128, 2010. Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408.16532, 2024. Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100, 2024. Herman Kamper, Weiran Wang, and Karen Livescu. Deep convolutional acoustic word embeddings using word-pair side information. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4950–4954. IEEE, 2016. Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11:1703–1718, 2023. Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352, 2022. 98
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36:27980– 27993, 2023. Yann LeCun et al. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62, 2022. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 7871–7880, 2020. Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6419–6423. IEEE, 2020. Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Semanticodec: An ultra low bitrate semantic audio codec for general sound. IEEE Journal of Selected Topics in Signal Processing, 18(8):1448–1461, 2024. Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference optimization through learning-torank. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 2404–2420, 2025a. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1412–1421, 2015. Xutai Ma, Juan Pino, James Cross, Liezl Puzon, and Jiatao Gu. Monotonic multihead attention. arXiv preprint arXiv:1909.12406, 2019. Jonathan Mamou, Bhuvana Ramabhadran, and Olivier Siohan. Vocabulary independent spoken term detection. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 615–622, 2007. Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235, 2024. Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023. André Merboldt, Albert Zeyer, Ralf Schlüter, and Hermann Ney. An analysis of local monotonic attention variants. In Interspeech, pp. 1398–1402, 2019. David RH Miller, Michael Kleber, Chia-Lin Kao, Owen Kimball, Thomas Colthurst, Stephen A Lowe, Richard M Schwartz, and Herbert Gish. Rapid and accurate spoken term detection. In Interspeech, volume 7, pp. 314–317, 2007. 99
Pooneh Mousavi, Gallil Maimon, Adel Moumen, Darius Petermann, Jiatong Shi, Haibin Wu, Haici Yang, Anastasia Kuznetsova, Artem Ploujnikov, Ricard Marxer, et al. Discrete audio tokens: More than a survey! arXiv preprint arXiv:2506.10274, 2025. Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. Advances in Neural Information Processing Systems, 37:108877–108901, 2024. Paarth Neekhara, Shehzeen Hussain, Subhankar Ghosh, Jason Li, Rafael Valle, Rohan Badlani, and Boris Ginsburg. Improving robustness of llm-based speech synthesis by learning monotonic alignment. arXiv preprint arXiv:2406.17957, 2024. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206–5210. IEEE, 2015. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pp. 28492–28518. PMLR, 2023. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. Colin Raffel, Minh-Thang Luong, Peter J Liu, Ron J Weiss, and Douglas Eck. Online and linear-time attention by enforcing monotonic alignments. In International conference on machine learning, pp. 2837– 2846. PMLR, 2017. Dhananjay Ram, Lesly Miculicich, and Hervé Bourlard. Cnn based query by example spoken term detection. In Interspeech, pp. 92–96, 2018. Dhananjay Ram, Lesly Miculicich, and Hervé Bourlard. Neural network based end-to-end query by example spoken term detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1416– 1427, 2020. Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925, 2023. Hasim Sak, Matt Shannon, Kanishka Rao, and Françoise Beaufays. Recurrent neural aligner: An encoderdecoder neural network model for sequence to sequence mapping. In Interspeech, volume 8, pp. 1298–1302, 2017. Murat Saraclar and Richard Sproat. Lattice-based search for spoken utterance retrieval. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pp. 129–136, 2004. 100
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Ravid Shwartz Ziv and Yann LeCun. To compress or not to compress—self-supervised learning and information theory: A review. Entropy, 26(3):252, 2024. Anup Singh, Kris Demuynck, and Vipul Arora. Attention-based audio embeddings for query-by-example. arXiv preprint arXiv:2210.08624, 2022. Anup Singh, Kris Demuynck, and Vipul Arora. Simultaneously learning robust audio embeddings and balanced hash codes for query-by-example. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2023. Anup Singh, Kris Demuynck, and Vipul Arora. Flowhash: Accelerating audio search with balanced hashing via normalizing flow. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024. Anup Singh, Kris Demuynck, and Vipul Arora. Best-std: Bidirectional mamba-enhanced speech tokenization for spoken term detection. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2025a. Anup Singh, Kris Demuynck, and Vipul Arora. Best-std2. 0: Balanced and efficient speech tokenizer for spoken term detection. arXiv preprint arXiv:2512.16395, 2025b. Anup Singh, Kris Demuynck, and Vipul Arora. Language-agnostic speech tokenizer for spoken term detection with efficient retrieval. In Proc. Interspeech 2025, pp. 2630–2634, 2025c. Hubert Siuzdak, Florian Grötschla, and Luca A Lanzendörfer. Snac: Multi-scale neural audio codec. arXiv preprint arXiv:2410.14411, 2024. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18990–18998, 2024. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence to sequence pretraining for language generation. arXiv preprint arXiv:1905.02450, 2019. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020. Yi Su, Jisheng Bai, Qisheng Xu, Kele Xu, and Yong Dou. Audio-language models for audio-centric tasks: A survey. arXiv preprint arXiv:2501.15177, 2025. Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827–6839, 2020. Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Local monotonic attention mechanism for endto-end speech and language processing. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 431–440, 2017. TJ Tsai. Segmental dtw: A parallelizable alternative to dynamic time warping. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 106–110. IEEE, 2021. Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self-supervised learning from a multi-view perspective. arXiv preprint arXiv:2006.05576, 2020. 101
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096–1103, 2008. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023a. Dong Wang, Joe Frankel, Javier Tejedor, and Simon King. A comparison of phone and grapheme-based spoken term detection. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4969–4972. IEEE, 2008. Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. Viola: Unified codec language models for speech recognition, synthesis, and translation. arXiv preprint arXiv:2305.16107, 2023b. Xiaofei Wang, Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as a versatile speech transformer. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3355–3364, 2024. Haibin Wu, Xuanjun Chen, Yi-Cheng Lin, Kai-wei Chang, Ho-Lam Chung, Alexander H Liu, and Hung-yi Lee. Towards audio language modeling–an overview. arXiv preprint arXiv:2402.13236, 2024. Shijie Wu and Ryan Cotterell. Exact hard monotonic attention for character-level transduction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1530–1537, 2019. Yi-Chiao Wu, Israel D Gebru, Dejan Marković, and Alexander Richard. Audiodec: An open-source streaming high-fidelity neural audio codec. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2023. Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9653–9663, 2022. Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of lowbitrate neural speech codec. arXiv preprint arXiv:2409.05377, 2024. Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765, 2023. Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Haohan Guo, Xuankai Chang, Jiatong Shi, Jiang Bian, Zhou Zhao, et al. Uniaudio: Towards universal audio generation with large language models. In Forty-first International Conference on Machine Learning, 2024. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019. Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. 102
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310–12320. PMLR, 2021. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 495–507, 2021. Tianyu Zhang, Xin Luo, Li Li, and Dong Liu. Stablecodec: Taming one-step diffusion for extreme image compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17379– 17389, 2025. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308.16692, 2023a. Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03926, 2023b. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023. Yingzhu Zhao, Chongjia Ni, Cheung-Chi Leung, Shafiq R Joty, Eng Siong Chng, and Bin Ma. Cross attention with monotonic alignment for speech transformer. In Interspeech, pp. 5031–5035, 2020.
103