AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Preprint

AUDIOTETRIS: T OKEN E LIMINATION VIA

arXiv:2610.13600v1 [cs.SD] 9 Oct 2026

T RANSITIONAL I MPORTANCE S IGNALS FOR E FFICIENT AUDIO LLM S Shreya Sajid, Anna Vettoruzzo & Joaquin Vanschoren Department of Mathematics & Computer Science Eindhoven University of Technology Eindhoven, Netherlands {s.sajid, a.vettoruzzo, j.vanschoren}@tue.nl

A BSTRACT Large Audio Language Models (LALMs) achieve strong performance on speechtext tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder’s representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top-k token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.

1

I NTRODUCTION

Large Audio Language Models (LALMs) have achieved state-of-the-art results across several speech-text tasks like speech recognition, audio QA, and general audio understanding (Chu et al., 2024; Radford et al., 2023; Chu et al., 2023; Wang et al., 2025a; Yang et al., 2024). Typically, these models use a pretrained audio encoder to map the input waveform into a sequence of continuous frame-level embeddings (audio tokens), which are projected into the Large Language Model (LLM) embedding space and prepended to text prompts. However, this approach creates a severe computational bottleneck: standard speech encoders generate hundreds of tokens for just a few seconds of audio (Lin et al., 2025; Chu et al., 2024; Ghosh et al., 2025a). Because decoder self-attention scales quadratically with sequence length, this large token footprint drives up inference costs (Tay et al., 2022). Furthermore, uniform audio tokenization is inherently redundant (Radford et al., 2023; Chu et al., 2024; Ghosh et al., 2025a; Wang et al., 2025b; Zhang et al., 2026). It clashes with the nature of speech, where information is concentrated during rapid phonetic transitions rather than steadystate regions (Furui, 1986a; Stevens, 2002; Aylett & Turk, 2004; Zhang et al., 2025a). Reducing token count without losing these acoustically rich transition regions is vital for efficient LALM 1

Preprint

Figure 1: Overview of AudioTETRIS. A phoneme-transition-aware acoustic prior wt is computed from the audio encoder’s hidden states via the transitional distance dt = ∥ht − ht−1 ∥22 , preserving phoneme transitions yhard (§4; left). Token pruning occurs in two stages: Stage 1 injects wt as a static acoustic prior from the encoder’s hidden states; Stage 2 computes a depth-adaptive LLM attention score, fusing intra-audio attention in early layers with cross-modal (text-to-audio) attention in later layers (right). The two scores are fused into a single importance score, and top-k selection retains the most salient tokens at each pruning layer ℓ. deployment. While recent methods apply token pruning or merging (Lin et al., 2025; Bhati et al., 2025; Gong et al., 2026; Behera et al., 2024), they share a substantial flaw: they rely solely on isolated decoder attention or local feature similarity, ignoring the structural phoneme transitions that could be already captured by the encoder (Behera et al., 2024; Lin et al., 2025; Gong et al., 2026; Jung et al., 2026; Bhati et al., 2025). We hypothesize that speech token compression should be guided by the underlying acoustic structure of the signal, rather than post-hoc decoder attention alone. Our approach is based on a simple insight: regions of rapid spectral transition in the encoder’s hidden states coincide with phonetic transitions, where essential acoustic details reside (Furui, 1986a; Stevens, 2002). We empirically validate this hypothesis in Section 4, showing a strong correlation between representation variance in encoder space and actual phonetic segment transitions. Compression methods that ignore this structure could risk discarding high-frequency acoustic details and degrading downstream performance (Zhang et al., 2026; Libera et al., 2026). To solve this, we introduce AudioTETRIS (Figure 1), a training-free framework that compresses audio tokens by jointly exploiting speech-specific acoustic structures and decoder-side attention. First, AudioTETRIS derives a phoneme-transition-aware prior from the temporal derivative of hidden states, isolating regions of high spectral change to protect information-dense transition segments. Second, this prior is propagated into the LLM decoder and fused with a depth-adaptive attention score derived from intra-audio attention in early layers and cross-modal text-to-audio attention in middle layers. By coupling acoustic salience with task-conditioned attention, token survival is driven by both structural and semantic relevance. We evaluate AudioTETRIS across backbones like Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3 (Chu et al., 2024; Microsoft et al., 2025; Ghosh et al., 2025a) on tasks including Automatic Speech Recognition (ASR), Speech Emotion Recognition, Audio Question Answering, and Speech Translation. AudioTETRIS retains near-lossless accuracy while keeping as few as 25% of tokens, outperforming other training-free baselines, and even surpassing full-sequence models at moderate compression rates.

2

Preprint

Our main contributions are: • Acoustic transition prior: We show that temporal variance in encoder hidden states serves as a reliable proxy for phonetic transitions, capturing high-information regions without fine-tuning or external alignments. • AudioTETRIS framework: We propose a training-free compression framework that propagates this acoustic prior into the LLM decoder, dynamically guiding self-attention to preserve both acoustic and semantic details. • State-of-the-art efficiency: Extensive experiments demonstrate near-lossless performance at aggressive compression levels (25% token retention), outperforming strong training-free baselines across diverse benchmarks.

2

R ELATED W ORK

Large Audio Language Models. LALMs typically couple a pretrained speech encoder, such as Whisper (Radford et al., 2023), with an LLM backbone (Chu et al., 2024; Ghosh et al., 2025b;a; Microsoft et al., 2025; Tang et al., 2024; Deshmukh et al., 2023), projecting audio features into the text embedding space. However, this paradigm yields audio token sequences that are orders of magnitude longer than the accompanying text, creating severe computational overhead during prefill and generation. Reducing this token count without sacrificing the dense lexical content of the underlying signal remains a key challenge (Shao et al., 2026; He et al., 2025). Training-Free Token Compression. Training-free compression methods eliminate redundant audio tokens without modifying model weights. One direction seeks to prune or merge sequence tokens prior to or during LLM processing (Bolya et al., 2023; Behera et al., 2024; Chen et al., 2024; Jung et al., 2026). Across vision-language models (VLMs), methods like SparseVLM (Zhang et al., 2025b) leverage self-attention logits to select text tokens as ‘raters’ that adaptively prune visual tokens across layers (see also Shang et al., 2025; Yang et al., 2025; Xing et al., 2025). In the speech domain, SpeechPrune (Lin et al., 2025) compresses long-form speech (≥30s) tokens using approximated attention scores from the LLM’s initial transformer layer alongside speech-text feature similarity. However, relying on early-layer LLM dynamics forces the model to evaluate query relevance before building a comprehensive contextual representation of the signal. Alternatively, encoder-level methods like Local Temporal Bipartite Merging (LTBM) (Luo et al., 2026) merge nearby tokens under a temporal window constraint (Li et al., 2023), relying purely on local feature similarity rather than task-specific or phonetic cues. Training-Based & Multimodal Token Compression. Rather than relying on training-free pruning, alternative strategies preserve performance by introducing extra trainable parameters or secondary modalities (Kavaki & Mandel, 2025). For instance, encoder-level pooling methods often require low-rank adapters (LoRA) (Hu et al., 2022) to fine-tune the backbone and recover downstream accuracy (Bhati et al., 2025). Although effective, this training dependency eliminates the module’s utility as a plug-and-play solution for unseen backbones or domain shifts. Other frameworks, such as EchoingPixels (Gong et al., 2026), achieve substantial token reduction by exploiting crossmodal alignment across visual and audio streams, but do not extend naturally to audio-language tasks. Across both directions, training-free methods bypass acoustic structure while structure-aware methods require fine-tuning or secondary modalities - a trade-off AudioTETRIS resolves with a training-free, acoustically grounded importance signal (§3).

3

M ETHODOLOGY

We present AudioTETRIS, a training-free framework for compressing audio tokens in LALMs. AudioTETRIS operates in two stages: Stage 1 (§3.2) extracts a phoneme-transition-aware acoustic prior directly from the audio encoder’s hidden states, and Stage 2 (§3.3) propagates this prior into the LLM decoder, fusing it with a depth-adaptive attention score to progressively prune audio tokens across decoder layers. Figure 1 illustrates the pipeline. 3

Preprint

Early Layers

Audio attention mass

Middle Layers

Later Layers

Text attention mass

Audio token share of sequence (85.0%)

Figure 2: Distribution of decoder attention mass allocated to audio tokens across model depth in Qwen2-Audio on LibriSpeech. Despite audio tokens comprising 84.98% of the input sequence (dashed ring), early layers are dominated by intra-audio attention (76.91%), which drops sharply in middle (46.73%) and late layers (40.60%) as attention shifts to text-conditioned representations. This shift motivates our layer-dependent pooling rule in Eq. 3.

3.1

P RELIMINARIES

Given an input waveform, the audio encoder outputs a sequence of hidden states {h1 , . . . , hT } ∈ RT ×d representing T audio tokens. These tokens are projected into the LLM embedding space and concatenated with the query embeddings to yield the initial decoder sequence X = [xa1 , . . . , xaT , xq1 , . . . , xqM ], with a and q referring to audio and query tokens. We index decoder layers as ℓ = 0, . . . , Ldec − 1, where Ldec is the number of decoder layers, and denote the layer-ℓ (ℓ) (ℓ) self-attention weight matrix as A(ℓ) ∈ RH×N ×N , where H is the number of heads and N (ℓ) is the sequence length at layer ℓ (which shrinks as audio tokens are pruned). 3.2

S TAGE 1: P HONEME T RANSITION S CORE

Motivation. Linguistic information in speech is concentrated near acoustic-phonetic transitions (Furui, 1986b). Because these transitions carry the most acoustic and phonetic information, we hypothesize that tokens spanning them undergo significantly larger representational shifts in latent space than tokens within stable phonetic segments. We leverage this geometric property to construct an acoustic gate that identifies informative tokens without external supervision. Phonetic Transition Score Formulation. For each audio token t, we compute the token-to-token derivative dt as the squared Euclidean distance between consecutive hidden states ht , ht−1 ∈ Rd taken from a single intermediate encoder layer (layer selection in §5.1): dt = ∥ht − ht−1 ∥22 ,

d1 := 0,

(1)

where the first token is assigned zero derivative in the absence of a preceding token. The squared L2 norm amplifies sharp representational shifts over intra-segment noise at a linear computational cost of O(d) per token. To modulate attention during pruning, we convert dt into a normalized acoustic weight wt : dt wt = 1 PT

i=1 di

T

,

(2)

where T is the total number of audio tokens in the utterance. The denominator enforces unit mean (Et [wt ] = 1). 3.3

S TAGE 2: D EPTH -A DAPTIVE ATTENTION S CORING

Motivation. Attention mass allocation evolves significantly across LLM depth (Figure 2). Early decoder layers are dominated by intra-audio attention (audio tokens attending to audio tokens) to refine structural audio representations. In contrast, middle and late layers reallocate attention toward cross-modal interaction, where text queries attend directly to audio tokens. Consequently, a single, depth-invariant attention metric fails to capture token importance uniformly across the network. 4

Preprint

(ℓ)

Layer-Adaptive Scoring. We address this depth dependence by defining the attention score p̃t for audio token t at layer ℓ over a depth-specific subset of query rows:  H X  X 1  (ℓ)   Ah,i,t , ℓ ∈ Learly (intra-audio filter)   H · |Qa | (ℓ) h=1 i∈Qa p̃t = (3) H X X  1  (ℓ)  A , ℓ ∈ L (cross-modal filter)  later h,i,t   H · |Qq | h=1 i∈Qq

where Qa and Qq denote the index sets of audio-query and text-query rows, H is the number of attention heads, and {Learly , Llater } partition the decoder layers. Automated Crossover Discovery. Rather than hand-crafting the layer partition {Learly , Llater }, we determine the crossover layer ℓ† from backbone attention dynamics on a small calibration set, as the first depth at which mean cross-modal attention mass across (ℓ) overtakes mean intra-audio attention mass aintra (ℓ): ℓ† = min{ℓ : across (ℓ) ≥ aintra (ℓ)}, (4) This unsupervised procedure identifies ℓ† in a single calibration pass per backbone; full derivation and per-layer statistics are in Appendix A.2.2. (ℓ)

Score Fusion. At each pruning layer ℓ, we fuse the layer-adaptive attention score p̃t with the static acoustic gate wt : (ℓ) (ℓ) st = p̃t · wt . (5) (ℓ)

At each pruning layer ℓ, we rank tokens by st and retain a top fraction kℓ , discarding the rest (see §5.1 for the pruning schedule and how kℓ is set per compression rate).

4

A NALYSIS : VALIDATION OF ACOUSTIC S TATE DYNAMICS

In Section 3.2, we hypothesized that token-to-token encoder state transitions dt track actual phonetic transitions. Here, we validate this hypothesis both quantitatively against ground-truth forced alignments and label-free via distributional kurtosis. Phoneme Transition Correlation. To evaluate whether encoder distance spikes align with physical phoneme transitions, we define a continuous diagnostic transition prior bt . Because audio encoders distribute context across neighboring tokens (Radford et al., 2023), we convolve dt with a Gaussian kernel of width σ = 2 and clip to [0, 1]: P  2 bt = clip (6) τ dτ · N (t − τ ; 0, σ ), 0, 1 . We evaluate bt against forced alignments on LibriSpeech (Panayotov et al., 2015; CorentinJ, 2019). Standard alignment tools yield discrete phonetic transitions yhard . To model continuous speech coarticulation (Furui, 1986b), we apply identical Gaussian smoothing to yhard , producing a continuous target ysoft . We measure the Pearson correlations rhard = corr(bt , yhard ) and rsoft = corr(bt , ysoft ). Figure 8 in Appendix A.2.3 illustrates this end-to-end pipeline. We provide additional dataset validations in Appendix A.2.4. As shown in Figure 3 (top), transition alignment varies predictably across depth, peaking in midnetwork layers across all backbones: Qwen2-Audio (layer 16, rsoft = 0.68) (Chu et al., 2024), Phi-4-Multimodal (layer 14, rsoft = 0.59) (Microsoft et al., 2025), and Audio Flamingo 3 (layer 15, rsoft = 0.62) (Ghosh et al., 2025a). Appendix A.2.5 provides an ablation over σ. Label-Free Diagnostic via Kurtosis. To verify this mid-network correlation peak without relying on external alignments, we measure the distributional kurtosis (κ) of dt across layer depth. High kurtosis indicates that transitions stand out as sharp, heavy-tailed outliers above background intrasegment noise. As shown in Figure 3 (bottom), kurtosis peaks at the exact same mid-network layers (κ ≈ 24.0-26.5), providing an unsupervised method to verify transition isolation. Trajectory Visualization. A qualitative t-SNE analysis of per-token hidden states (Appendix A.2.6, Figure 11) corroborates this pattern: mid-network layers show tight phoneme clustering separated by sudden trajectory jumps at transitions, while early and late layers show progressively less phonetically consolidated structure. 5

0.75 0.60 0.45 0.30 0.15 0.00

Kurtosis (norm)

Pearson Correlation

Preprint

30 24 18 12 6 0

Qwen2-Audio

0.68

Phi-4-Multimodal

0.75 0.60 0.45 0.30 0.15 0.00

0.35

26.5

0 4 8 12 16 20 24 28 Layer Index

30 24 18 12 6 0

rsoft

0.59 0.31

24.0

0

4

0.75 0.60 0.45 0.30 0.15 0.00 30 24 18 12 6 0

Audio-Flamingo

0.62 0.33

26.3

8 12 16 20 0 4 8 12 16 20 24 28 Layer Index Layer Index rhard Kurtosis (norm)

Figure 3: Layer-wise diagnostics across Qwen2-Audio, Phi-4-Multimodal, and Audio Flamingo 3. Top: Mean correlation between bt and ground-truth phoneme transitions (rsoft and rhard ). Bottom: Kurtosis (κ) of the distance distribution dt . Both metrics identify an intrinsic mid-network sweet spot (shaded) where phonetic transitions are maximally accentuated.

5

E XPERIMENTS & R ESULTS

5.1

E XPERIMENTAL S ETUP

Backbones and Datasets. We evaluate AudioTETRIS on three LALM backbones with distinct encoder-decoder architectures: Qwen2-Audio-7B (∼8.4B) (Chu et al., 2024), Phi-4-Multimodal (5.6B) (Microsoft et al., 2025), and Audio Flamingo 3 (∼8.3B) (Ghosh et al., 2025a). Our evaluation spans four core tasks: automatic speech recognition on LibriSpeech test-other (Panayotov et al., 2015), emotion recognition on IEMOCAP (Busso et al., 2008), audio question answering on MMAU-mini (Sakshi et al., 2025), and speech translation (English-to-Chinese) on CoVoST 2 (Wang et al., 2021). Together, these benchmarks span lexical transcription, cross-lingual generation, paralinguistic classification, and complex audio reasoning, allowing us to test whether preserving phoneme-transition regions specifically benefits tasks that require precise lexical recovery. Baselines. We compare against SpeechPrune (Lin et al., 2025), SparseVLM (Zhang et al., 2025b), and LTBM (Luo et al., 2026), representing the two families of prior work discussed in §2. We additionally include a Random Pruning baseline that retains a uniformly random token subset at matched compression rates, providing a lower-bound sanity check. All baselines are implemented under matched compression schedules using released hyperparameters where available. Implementation Details. We evaluate four target token compression rates - 0.36, 0.51, 0.64, and 0.75 - representing the fraction of audio tokens removed by the final decoder layer. These correspond to uniform per-stage retention factors r ∈ {0.8, 0.7, 0.6, 0.5} across two pruning layers (1 − r2 ) in our experiments, ranging from light reduction to aggressive 4× compression. Pruning is scheduled at layer 11 and at a backbone-specific later layer ℓ† (Eq. 4) via progressive top-k retention on the (ℓ) fused score st (Eq. 5). The acoustic prior wt is computed from a single mid-network encoder layer per backbone: layer 16 for Qwen2-Audio, layer 14 for Phi-4-Multimodal, and layer 15 for Audio Flamingo 3. These layers are selected by the label-free kurtosis criterion of §4 (the depth maximizing κ of dt ); forced alignments are used only to validate this choice post hoc and are not required to apply AudioTETRIS to a new backbone. The diagnostic prior bt is computed from the same layer. All experiments are executed on a single NVIDIA A100 GPU. Layer choices, calibration, and alternative schedules are detailed in Appendices A.2.2 and A.3.3. 1 Decoder layers are indexed starting from 0 throughout this paper (matching the axis convention used in Figures 2 and 7); e.g., ‘layer 1’ denotes the second decoder layer.

6

Preprint

5.2

M AIN R ESULTS

We report overall performance and efficiency across four compression rates on Qwen2-Audio in Table 1. Three key observations emerge from these results: AudioTETRIS maintains high accuracy even under heavy compression. At low compression (rate 0.36), AudioTETRIS matches or improves upon the uncompressed baseline on many benchmarks (4.52 WER, 58.04% IEMOCAP, 42.36% MMAU-mini vs. 4.61, 55.11%, 42.20% uncompressed). Even at rate 0.75, WER stays at 8.50 while IEMOCAP, MMAU-mini, and CoVoST2 remain within 2 points of uncompressed baselines, confirming that our method removes redundant tokens without disrupting essential lexical and semantic transitions. Performance gaps widen as compression grows aggressive. At rate 0.36, several baselines remain competitive; by rate 0.75, baseline methods degrade sharply (random pruning WER climbs to 31.59), lacking a mechanism to safeguard acoustic transitions. AudioTETRIS outperforms the best baseline by over 1.2 WER points at the highest compression setting. ASR is inherently more vulnerable to token pruning. Fine-grained transcription (WER) degrades far faster under aggressive compression than high-level understanding (MMAU-mini) or emotion recognition (IEMOCAP): at rate 0.75, IEMOCAP and MMAU-mini stay nearly intact while WER roughly doubles. High-level tasks rely on coarse acoustic features and tolerate token dropout, whereas exact lexical recovery needs fine-grained transition preservation - a vulnerability that speech-specific methods (AudioTETRIS, SpeechPrune) mitigate far better than general-purpose vision baselines or non-speech heuristics. 5.3

E FFICIENCY A NALYSIS

Pruning within the LLM backbone yields direct computational savings, dropping total TFLOPs from 7.89 to 4.32 at a 0.75 compression rate, with latency falling from 1.31 s to 0.62 s over the same range. In-LM methods (AudioTETRIS, SparseVLM) incur slightly higher TFLOPs than single-shot methods (SpeechPrune, LTBM) since early layers still process the full, unpruned sequence before the first pruning stage; this difference is minimal relative to AudioTETRIS’s accuracy gains. Realworld latency scales somewhat less than the FLOP drop due to RoPE reindexing and KV-cache trimming overhead at intermediate layers (Appendix A.1.4). 5.4

G ENERALIZATION ACROSS BACKBONES

To test whether AudioTETRIS’s gains are specific to Qwen2-Audio’s architecture, we repeat the main comparison on Phi-4-Multimodal and Audio Flamingo 3 at compression ratio 0.51 (Figure 4). On both backbones, AudioTETRIS remains within 1 point of uncompressed WER and attains the strongest or near-strongest accuracy on MMAU-mini, while baselines that match it on one metric (e.g., WER) fall sharply behind on others (e.g., SpeechPrune loses over 17 BLEU points on CoVoST2); Full numbers are given in Table 4 (Phi-4-Multimodal) and Table 5 (Audio Flamingo 3). These results suggest the phonetic transition prior transfers across encoders with different subsampling rates and architectures (Phi-4-Multimodal’s Conformer encoder vs. the Whisper-based encoders of Qwen2-Audio and Audio Flamingo 3), consistent with encoder-state transitions being a general property of speech representations rather than an artifact of one backbone. Extended results across all four compression ratios and benchmarks are in Appendix A.3.1, where similar patterns hold.

6

A BLATION S TUDIES

We isolate the contribution of each component of AudioTETRIS’s scoring function and validate the choice of distance metric underlying the transition score §3.2. 6.1

D ISTANCE M ETRIC FOR THE T RANSITION S CORE

The transition signal dt requires a distance function to measure the transition magnitude between adjacent encoder hidden states. We compare four standard choices: Euclidean (L2), cosine, Manhat7

Preprint

Table 1: Performance and efficiency comparison across compression ratios on Qwen2Audio evaluated on LibriSpeech (ASR), IEMOCAP (Emotion Recognition), MMAU-mini (Audio Understanding, MCQ), and CoVoST2 (Speech Translation). WER (%) is lower-is-better (↓); Accuracy (%) and BLEU are higher-is-better (↑). Baselines: SpeechPrune (Lin et al., 2025), SparseVLM (Zhang et al., 2025b), LTBM (Luo et al., 2026). Method

LibriSpeech WER ↓ IEMOCAP Acc. ↑ MMAU-mini Acc. ↑ CoVoST2 BLEU ↑ Latency (s) TFLOPs 4.61±0.12

No Compression

55.11±1.06

42.20±0.96

33.03±0.72

1.31

7.89

29.10±0.02 31.34±0.07 31.38±0.12 29.97±0.06 31.96±0.04

1.15 1.21 1.16 1.25 1.19

6.93 6.99 7.28 6.93 7.33

26.95±0.01 31.16±0.15 31.17±0.09 29.55±0.03 31.45±0.06

0.97 1.14 1.15 1.15 1.00

5.61 5.67 6.10 5.62 6.13

24.64±0.02 30.12±0.05 30.79±0.12 29.47±0.06 31.24±0.11

0.84 0.99 0.95 1.03 0.86

4.47 4.53 5.08 4.49 5.26

22.82±0.01 25.77±0.05 30.66±0.02 24.26±0.05 31.33±0.07

0.61 0.69 0.71 0.70 0.62

3.52 3.58 4.23 3.52 4.32

Compression Ratio 0.36 Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

9.91±1.67 6.75±1.44 8.40±1.08 5.90±0.48 4.52±0.21

53.10±0.37 56.43±0.35 52.37±0.21 55.90±0.27 58.04±0.84

Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

14.02±0.33 6.52±1.14 9.13±1.15 6.92±0.04 5.77±0.33

52.08±0.49 57.39±0.74 47.10±0.69 54.58±0.42 57.03±0.41

Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

24.08±0.26 8.62±0.07 12.40±0.55 8.60±0.18 6.62±0.29

50.22±1.10 56.10±1.09 47.27±1.74 53.38±0.86 57.90±0.76

Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

31.59±0.24 9.75±0.10 17.30±0.70 10.58±0.42 8.50±0.18

47.67±0.71 54.17±0.92 46.10±3.05 47.84±1.05 55.34±1.25

40.18±0.23 41.17±0.60 41.09±0.73 40.67±0.67 42.36±0.21

Compression Ratio 0.51 39.20±0.43 41.83±0.06 38.80±0.27 40.33±0.40 42.91±0.36

Compression Ratio 0.64 39.23±0.41 40.30±0.69 37.66±0.66 40.70±0.78 41.83±0.15

Compression Ratio 0.75

† ‡

38.57±1.16 40.93±0.34 35.83±0.12 39.23±0.45 41.46±0.32

In-LM pruning (token elimination performed inside the LLM backbone). Originally developed for the vision modality, adapted here for speech/audio.

Phi-4-Multimodal

Audio Flamingo 3

LibriSpeech (WER ↓ ) MMAU-mini (Acc. ↑ ) LibriSpeech (WER ↓ ) MMAU-mini (Acc. ↑ ) No Compression Random SpeechPrune SparseVLM LTBM

AudioTETRIS

0

5

10 50.0

52.5

55.0 0

5

57.5

60.0

62.5

Figure 4: Comparison of LibriSpeech WER (↓) and MMAU-mini accuracy (↑) at compression ratio 0.51, for Phi-4-Multimodal and Audio Flamingo 3. Error bars show standard deviation across runs. AUDIOTETRIS is highlighted in the darker shade. tan (L1), and Chebyshev (L∞ ) distance, holding all other hyperparameters fixed at a compression ratio of 0.51. Results are summarized in Figure 5. We find that L2 distance is the strongest choice on all four benchmarks, with the largest gap on ASR (cosine increases WER by over 50%: 9.04% vs. 5.77%); hence we use it as the default throughout. 6.2

C OMPONENT AND S TAGE A BLATION A NALYSIS (ℓ)

(ℓ)

The fused score st = p̃t · wt (Eq. 5) combines the depth-adaptive attention term with the acoustic prior. Figure 6 isolates each stage’s contribution against the full pipeline at compression ratio 0.51; extended results across all ratios and remaining benchmarks are in Table 6 (Appendix A.3.2). 8

Preprint

LibriSpeech (WER )

MMAU-mini (Acc. )

60

10.0 7.5

55

5.0 2.5

IEMOCAP (Acc. )

L2 Cos L1

L

50

L2 Cos L1

L

34

44

32

42

30

40

28 L2 Cos L1

L

26

CoVoST2 (BLEU )

L2 Cos L1

L

Figure 5: WER/accuracy/BLEU across distance metrics for the transition score, at compression ratio 0.51. Error bars show standard deviation over runs. LibriSpeech (WER ↓ )

MMAU-mini (Acc. ↑ )

45 We find that both stages combined consistently 10 outperforms either stage alone across all ratios 5 and benchmarks, with the gap widening sharply 40 under aggressive compression (at ratio 0.75, 0 Only Only Stage 1 + Only Only Stage 1 + Stage 2 alone degrades to 15.56% WER vs. Stage 1 Stage 2 Stage 2 Stage 1 Stage 2 Stage 2 8.50% combined). This reflects complemen- Figure 6: Ablation of AudioTETRIS stages at tary signals: the phonetic prior wt preserves compression ratio 0.51. phonetically contiguous spans, while the attention term targets task relevance - most visible on LibriSpeech, where ASR’s sensitivity to fine-grained transitions makes the two-stage advantage largest.

7

C ONCLUSION

We presented AudioTETRIS, a training-free framework for compressing audio tokens in LALMs. By leveraging the audio encoder’s hidden-state dynamics as a phoneme-transition prior and fusing it with depth-adaptive decoder attention, AudioTETRIS preserves acoustically rich and semantically critical regions without requiring auxiliary training or fine-tuning. Experiments across Qwen2Audio, Phi-4-Multimodal, and Audio Flamingo 3, demonstrate that AudioTETRIS retains nearlossless performance at light compression, while widening its advantage over training-free baselines as compression gets aggressive. Furthermore, our findings highlight that ASR is highly sensitive to token loss making speech-aware structural priors essential for exact lexical recovery. Ultimately, this work highlights a broader principle for multimodal architecture design and token-compression methods: intermediate representations computed by modality-specific encoders contain valuable structural geometry that should be propagated forward rather than discarded at the LLM interface. L IMITATIONS AudioTETRIS’s acoustic prior assumes speech-encoder hidden states exhibit large-norm transitions at phonetic boundaries - validated for Whisper-based encoders (Qwen2-Audio, Audio Flamingo 3) and the Conformer-based Phi-4-Multimodal encoder, but not confirmed for encoders trained with different objectives or aggressive temporal downsampling (e.g., HTSAT-style audio-event encoders). Second, while the early pruning layer is fixed by construction, the later crossover layer ℓ† requires a short backbone-specific calibration pass (Appendix A.2.2) - a one-time cost, but the method is not fully zero-shot to unseen architectures without it. Third, because pruning happens progressively inside the decoder rather than once before prefill, AudioTETRIS carries a small TFLOP overhead relative to single-shot methods (§5.3), though this is largely offset by its accuracy gains. Finally, our evaluation spans four English-source benchmarks; results on noisier, or multilingual audio remain untested, and pushing compression beyond the rates studied here would likely erode lexical accuracy first, given ASR’s sensitivity to aggressive compression.

9

Preprint

ACKNOWLEDGEMENTS Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the European Commission can be held responsible for them. This work was supported by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101214398 (ELLIOT). This publication is part of project 2026.008 of the Computing Time National Computing Facilities program which is financed by the Dutch Research Council (NWO) under the grant https://doi.org/10.61686/WBFDE48237. R EPRODUCIBILITY S TATEMENT AudioTETRIS is training-free, so no training procedure, optimizer, or learning-rate schedule needs to be specified for reproduction. All components needed to reproduce our results are given in the main text and appendix: the phoneme transition score (Eq. 1-2), the depth-adaptive attention score and its automated crossover-layer discovery procedure (Eq. 3-4, Appendix A.2.2). A one-time, perbackbone calibration pass (Appendix A.2.2) locates the crossover layer ℓ† from attention statistics; this involves no gradient computation or parameter updates. The full fused-score pruning schedule are summarized as pseudocode in Algorithms 1 and 2 (Appendix A.2.1). Pruning layers, per-stage retention rates, and the closed-form multi-stage schedule are specified in §5.1 and Appendix A.3.3 (Eq. 9, Table 7). Backbone checkpoints, exact dataset splits, and their licenses are listed in Appendix A.1.1 (Tables 2, 3); alignment sources used for the boundary-correlation analysis (§4) are described in Appendix A.2.4. We will release our full implementation, including the pruning modules described in Algorithms 1 and 2 and the evaluation scripts used to produce Tables 1-5, at https://github.com/ShreyaSajid/AudioTETRIS.

R EFERENCES Matthew Aylett and Alice Turk. The smooth signal redundancy hypothesis: A functional explanation for relationships between redundancy, prosodic prominence, and duration in spontaneous speech. Language and Speech, 47(1):31–56, 2004. doi: 10.1177/00238309040470010201. Swarup Ranjan Behera, Abhishek Dhiman, Karthik Gowda, and Aalekhya Satya Narayani. Fastast: Accelerating audio spectrogram transformer via token merging and cross-model knowledge distillation. In Interspeech 2024, pp. 4303–4307, 2024. doi: 10.21437/ Interspeech.2024-1185. URL https://www.isca-archive.org/interspeech_ 2024/behera24_interspeech.html. Saurabhchand Bhati, Samuel Thomas, Hilde Kuehne, Rogerio Feris, and James Glass. Towards audio token compression in large audio language models. arXiv preprint arXiv:2511.20973, 2025. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In International Conference on Learning Representations (ICLR), 2023. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette Chang, Sungbok Lee, and Shrikanth Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335–359, 12 2008. doi: 10.1007/s10579-008-9076-6. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large visionlanguage models. In European Conference on Computer Vision (ECCV), pp. 19–35. Springer, 2024. doi: 10.1007/978-3-031-73004-7 2. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, and Junyang Lin. Qwen2-audio technical report, 2024. URL https://arxiv. org/abs/2407.10759. 10

Preprint

CorentinJ. Librispeech alignments. https://github.com/CorentinJ/ librispeech-alignments, 2019. GitHub repository. Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. In Advances in Neural Information Processing Systems, volume 36, pp. 18090–18108, 2023. Sadaoki Furui. On the role of spectral transition for speech perception. The Journal of the Acoustical Society of America, 80(4):1016–1025, 1986a. doi: 10.1121/1.393842. Sadaoki Furui. Speaker-independent isolated word recognition using dynamic features of speech spectrum. IEEE Transactions on Acoustics, Speech, and Signal Processing, 34(1):52–59, 1986b. doi: 10.1109/TASSP.1986.1164788. Sreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang gil Lee, ChaoHan Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a. doi: 10.52202/085713-1396. Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro. Audio flamingo 2: An audio-language model with longaudio understanding and expert reasoning abilities. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267, pp. 19358– 19405, 2025b. Chao Gong, Depeng Wang, Zhipeng Wei, Ya Guo, Huijia Zhu, and Jingjing Chen. Echoingpixels: Aliasing-resistant joint token reduction for audio-visual llms. In Forty-third International Conference on Machine Learning, 2026. Peize He, Zichen Wen, Yubo Wang, Yuxuan Wang, Xiaoqian Liu, Jiajie Huang, Zehui Lei, Zhuangcheng Gu, Xiangqi Jin, Jiabing Yang, Kai Li, Zhifei Liu, Weijia Li, Cunxiang Assisting Wang, Conghui He, and Linfeng Zhang. Audiomarathon: A comprehensive benchmark for long-context audio understanding and efficiency in audio llms. arXiv preprint arXiv:2510.07293, 2025. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022. Chaeyoung Jung, Youngjoon Jang, Seungwoo Lee, and Joon Son Chung. Fastav: Efficient token pruning for audio-visual large language model inference. In ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026. doi: 10.1109/ICASSP5730.2026.11463130. Hassan Salami Kavaki and Michael I. Mandel. Audio sparse-transformer for speech classification. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025. doi: 10.1109/ICASSP63290.2025.10890475. Yuang Li, Yu Wu, Jinyu Li, and Shujie Liu. Accelerating Transducers through Adjacent Token Merging. In Interspeech 2023, pp. 1379–1383, 2023. doi: 10.21437/Interspeech.2023-599. Luca Della Libera, Cem Subakan, and Mirco Ravanelli. Beyond fixed frames: Dynamic characteraligned speech tokenization, 2026. URL https://arxiv.org/abs/2601.23174. Yueqian Lin, Yuzhe Fu, Jingyang Zhang, Yudong Liu, Jianyi Zhang, Jingwei Sun, Hai Li, and Yiran Chen. Speechprune: Context-aware token pruning for speech information retrieval. In 2025 IEEE International Conference on Multimedia and Expo (ICME), 2025. Jiale Luo, Xiaoyu Liang, and Haoji Hu. Locality matters for training-free audio token compression in audio-language models. arXiv preprint arXiv:2605.25179, 2026. 11

Preprint

Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Proceedings of the 18th Annual Conference of the International Speech Communication Association (INTERSPEECH 2017), pp. 498–502, 2017. doi: 10.21437/Interspeech.2017-1386. Microsoft, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang, Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Perez-Becker, Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang, Yelong Shen, Saksham Singhal, Subhojit Som, Xia Song, Tetyana Sych, Praneetha Vaddamanu, Shuohang Wang, Yiming Wang, Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang, Ziyi Yang, Donghan Yu, Ishmam Zabir, Jianwen Zhang, Li Lyna Zhang, Yunan Zhang, and Xiren Zhou. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras, 2025. URL https://arxiv.org/abs/2503.01743. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206–5210. IEEE, 2015. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. ICML, 2023. S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22857–22867, October 2025. Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. A survey of token compression for efficient multimodal large language models. Transactions on Machine Learning Research, 2026. URL https://openreview. net/forum?id=G2od9JVHkE. Kenneth N. Stevens. Toward a model for lexical access based on acoustic landmarks and distinctive features. The Journal of the Acoustical Society of America, 111(4):1872–1891, 2002. doi: 10. 1121/1.1458026. Changli Tang, Wenyi Yao, Guangke Zhou, Bin Chen, Siyuan Deng, Zhuo Liu, Jiahui Wang, Ziyang Zhou, Xiu Zhang, Zhou Wan, et al. SALMONN: Towards generic hearing abilities for large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Comput. Surv., 55(6), December 2022. ISSN 0360-0300. doi: 10.1145/3530811. URL https://doi.org/10.1145/3530811. Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F. Chen. AudioBench: A universal benchmark for audio large language models. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4297–4316. Association for Computational Linguistics, April 2025a. doi: 10.18653/v1/2025.naacl-long.218. URL https://aclanthology.org. Changhan Wang, Anne Conneau, Jiatao Gu, and Juan Pino. CoVoST 2 and Massively Multilingual Speech Translation. In Proc. Interspeech 2021, pp. 4234–4238, 2021. doi: 10.21437/Interspeech. 2021-1773. 12

Preprint

Jiatong Wang, Yifan Zhao, Shicong Wang, Hexin Lin, Shugao Wang, Shiyi Feng, and Dongyan Jiang. Say more with less: Variable-frame-rate speech tokenization via adaptive clustering and implicit duration coding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 25310–25318, 2025b. URL https://aaai.org. Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. Qian Yang Yang, Hongbin Wang, Zheng Li, Shunian Wang, Shifu Zhang, Meng Liu, Jun Guo, Yan Zhou, Chang Su, Zhao Zhang, Chao Li, and Wei Xu. AIR-bench: Benchmarking large audio-language models via generative comprehension. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1966–1986, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025. Xiangyu Zhang, Fuming Fang, Peng Gao, Bin Qin, Beena Ahmed, and Julien Epps. Distinctive feature codec: An adaptive efficient speech representation for depression detection, 2025a. URL https://arxiv.org/abs/2505.18516. Xiangyu Zhang, Yuxin Li, Haoyang Zhang, Shiqi Han, Hexin Liu, Qiquan Zhang, Beena Ahmed, and Julien Epps. The wer trap: Shattering the illusion of unified tokens in speech language models. arXiv preprint arXiv:2605.29209, 2026. Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference. In International Conference on Machine Learning, 2025b.

13

Preprint

A

A PPENDIX

A.1

I MPLEMENTATION D ETAILS

A.1.1

BACKBONES AND C HECKPOINTS

Table 2 lists the exact checkpoints and licenses for the three backbones evaluated in this work. All experiments use unmodified pretrained weights; the only backbone-side code changes are loadtime monkey-patches to the decoder forward pass implementing AudioTETRIS’s pruning (§3). No checkpoint weights were altered or fine-tuned. Table 2: Pretrained checkpoints used in this work. Backbone

Checkpoint

License

Qwen2-Audio (Chu et al., 2024) Phi-4-Multimodal (Microsoft et al., 2025) Audio Flamingo 3 (Ghosh et al., 2025a)

Qwen/Qwen2-Audio-7B-Instruct microsoft/Phi-4-multimodal-instruct nvidia/audio-flamingo-3-hf

Apache 2.0 MIT NVIDIA OneWay Noncommercial License*

*

Non-commercial research use only; see checkpoint card for full terms.

A.1.2

DATASETS , S PLITS , AND E VALUATION

Table 3 lists the datasets used for evaluation, the exact splits, and their licensing. We accessed all four via the AudioLLMs organization’s re-packaged mirrors on the Hugging Face Hub, used throughout the AudioBench evaluation harness (Wang et al., 2025a), rather than the original distribution channels for each corpus. For the CoVoST2 phoneme alignments generated for Appendix A.2.4, these are derivative annotations produced by us using the Montreal Forced Aligner and are not redistributed third-party material. Table 3: Evaluation datasets, splits, and licensing. Dataset

Split used

Original license

LibriSpeech (test-other) (Panayotov et al., 2015) IEMOCAP (Busso et al., 2008) MMAU-mini (Sakshi et al., 2025) CoVoST2 (En→Zh) (Wang et al., 2021)

AudioLLMs/librispeech test other, test AudioLLMs/iemocap emotion recognition, test AudioLLMs/MMAU-mini, test AudioLLMs/covost2 en zh test, test

CC BY 4.0 Custom non-commercial research license (USC SAIL) Not explicitly stated by source CC0 (per original release); Hub mirror tagged CC BY-NC 4.0

A.1.3

S PARSE VLM A DAPTATION TO F IXED -R ATE S PEECH P RUNING

SparseVLM (Zhang et al., 2025b) originally proposed an adaptive, dynamic pruning schedule based on the rank of the visual-text attention matrix. However, its public release2 replaces this dynamic controller with static token-retention schedules. Because the dynamic rank-based ratio is absent from all usable releases (main, v1.5, v2.0), we could not evaluate SparseVLM’s originally described adaptive deployment directly. Instead, we adapt its public attention-guided scoring mechanism to our fixed-rate setup for a direct, controlled comparison. Adaptation. We retain SparseVLM’s core scoring mechanism - using the text raters, and text-token attention to rate audio token importance - and apply it at layers {1, 15} (Appendix A.2.2). At each stage, we keep the top fraction r of audio tokens by score to match our evaluated compression targets (0.36, 0.51, 0.64, 0.75). This adaptation isolates the impact of SparseVLM’s attention-based importance metric under identical layer placements and compute budgets. A.1.4

ATTENTION BACKEND FOR S CORE -BASED P RUNING

AudioTETRIS and the attention-score-based baselines we compare against (SparseVLM Zhang et al. (2025b), SpeechPrune (Lin et al., 2025)) all derive token-importance scores from attention probabilities between query and audio tokens. Fused attention kernels such as FlashAttention2 (Dao, 2024) and PyTorch’s scaled-dot-product attention (SDPA) do not materialize these probabilities: both return attn weights=None, since the attention matrix is never explicitly formed. 2 https://github.com/Gumpest/SparseVLMs, accessed 87fe4319430e079a778c25d4ddf4588a3c4038ca 2025-12-22.

14

May 2026,

commit

Preprint

Eager attention is the only implementation that exposes the full [B, H, N (ℓ) , N (ℓ) ] probability tensor required for scoring. We therefore run the attention backend in eager mode only at the layers where token scores are computed, and restore the model’s configured kernel (FlashAttention-2 (Dao, 2024), in our setup) for the forward pass that produces the hidden states and KV cache used downstream. This substitution is local to the small set of scoring layers (§3) rather than applied model-wide, so the FlashAttention2 fast path is preserved for the remainder of the network’s depth. We apply this identically across AudioTETRIS and all attention-score-based baselines, so the latency and FLOPs figures reported in §5.2 reflect the same hybrid execution path for every method and remain directly comparable. We note that this eager-attention fallback is a memory-bandwidth-bound cost that is not reflected in the TFLOPs column, which counts compute operations only; it is, however, fully captured in our reported wall-clock latency, since all latency numbers are measured end-to-end on the actual hybrid execution path (eager at scoring layers, FlashAttention-2 elsewhere). Because the eager fallback is confined to exactly two decoder layers per backbone (layer 1 and that backbone’s calibrated ℓ† ; §5.1), its contribution to total latency is small in practice, and it could be eliminated entirely in a production deployment with a custom fused kernel that computes only the row/column marginal attention sums our scoring requires, without materializing the full attention matrix - a one-time kernel-engineering cost rather than a potential structural limitation of the method. A.2 A.2.1

M ETHOD D ETAILS AND A NALYSIS A LGORITHMIC S UMMARY

For reproducibility, Algorithms 1 and 2 summarize the full pipeline described in §3. Algorithm 1 is a one-time, offline calibration step run once per backbone to locate the crossover layer ℓ† (§A.2.2); Algorithm 2 is the training-free inference procedure applied per input, using ℓ† from calibration to determine which attention branch of Eq. 3 to use at each pruning layer. Algorithm 1 Crossover Layer Calibration (offline, run once per backbone) Require: Calibration set U (audio-query pairs), backbone with Ldec decoder layers Ensure: Crossover layer ℓ† 1: for each utterance u ∈ U do (ℓ) 2: Run forward pass; record attention tensors Au for all layers ℓ = 1, . . . , Ldec 3: end for 4: for ℓ = 1, . . . , Ldec do 5: aintra (ℓ) ← mean attention mass from Qa rows onto audio tokens (Eq. 3, upper case), averaged over U 6: across (ℓ) ← mean attention mass from Qq rows onto audio tokens (Eq. 3, lower case), averaged over U 7: end for 8: ℓ† ← min{ℓ : across (ℓ) ≥ aintra (ℓ)} ▷ Eq. 4 9: return ℓ†

A.2.2

P RUNING L AYER S ELECTION

AudioTETRIS prunes at two decoder layers (§5.1), and the two are chosen by different criteria because they serve different roles. The early layer is about saving compute: it is fixed by design as the earliest depth at which the attention term exists, and pruning there is viable because the acoustic prior compensates for weak early attention. The late layer ℓ† is about task relevance: it is calibrated per backbone from attention crossover statistics to locate where cross-modal grounding takes over (§A.2.2). We motivate each choice separately below. Early Pruning Layer. Because self-attention cost accrues at every subsequent decoder layer, the earlier a token is removed, the larger the cumulative FLOP reduction across the rest of the forward pass. We therefore place the first pruning stage at layer 1 - the earliest layer at which the depth(ℓ) adaptive attention score p̃t (Eq. 3) is defined, since it requires at least one decoder self-attention 15

Preprint

Algorithm 2 AudioTETRIS Inference (training-free, applied per input) Require: Audio encoder hidden states H = {h1 , . . . , hT }, pruning layers {ℓ1 , ℓ2 } with ℓ1 < ℓ† ≤ ℓ2 (in practice ℓ2 = ℓ† , the backbone’s calibrated crossover layer; §5.1), target retention fractions {kℓ1 , kℓ2 }, decoder with Ldec layers Ensure: Compressed decoder output Stage 1: Acoustic Transition Prior (computed once, before decoding) 1: d1 ← 0 2: for t = 2, . . . , T do 3: dt ← ∥ht − ht−1 ∥22 ▷ Eq. 1 4: end forP T 5: d¯ ← T1 i=1 di 6: for t = 1, . . . , T do 7: wt ← dt /d¯ ▷ Eq. 2 8: end for Stage 2: Depth-Adaptive Pruning (during decoder forward pass) 9: T ← {1, . . . , T } ▷ surviving audio-token indices 10: for ℓ = 1, . . . , Ldec do 11: Run decoder layer ℓ on current sequence (audio tokens indexed by T , plus query tokens) 12: if ℓ ∈ {ℓ1 , ℓ2 } then 13: if ℓ < ℓ† then (ℓ) 14: p̃t ← mean attention from audio-query rows Qa onto token t, ∀t ∈ T ▷ Eq. 3, intra-audio 15: else (ℓ) ▷ Eq. 3, 16: p̃t ← mean attention from text-query rows Qq onto token t, ∀t ∈ T cross-modal 17: end if (ℓ) (ℓ) 18: st ← p̃t · wt , ∀t ∈ T ▷ Eq. 5 (ℓ) 19: T ← indices of the top ⌈kℓ · |T |⌉ tokens in T ranked by st 20: Evict pruned tokens from hidden states and KV cache; update RoPE position indices for T 21: end if 22: end for 23: return final decoder output

computation to exist. Deferring pruning to a deeper early layer might yield a marginally sharper intra-audio attention signal, but this benefit is outweighed by the compute lost across every additional layer the full, unpruned token sequence has to pass through. Early pruning works well for AudioTETRIS because the layer-1 score does not rely on attention (1) (1) (1) alone. The fused score st = p̃t · wt (Eq. 5) combines the attention-based importance p̃t , computed from a single decoder layer, with the acoustic transition prior wt (Eq. 2). The prior comes directly from the audio encoder’s hidden states, so it is already fully informative before the decoder even starts (§4). The prior’s independence from decoder depth matters because attention alone is a weak signal at layer 1, where the decoder has not yet built up much context. The acoustic prior compensates for this weakness, making the fused score far more reliable at this depth than attention alone would be. This reliability is what lets us prune aggressively at layer 1 without the accuracy collapse that a purely attention-based method suffers. Our component ablation supports this conclusion (§6.2, Table 6): ‘Only Stage 2’ (attention alone) degrades much faster as compression increases than either ‘Only Stage 1’ or the full fused score. Late Pruning Layer: Crossover Calibration. Section 3.3 defines the crossover layer ℓ† (Eq. 4) as the first decoder depth at which cross-modal attention mass overtakes intra-audio attention mass, (ℓ) and uses it to set the layer partition {Learly , Llater } for the depth-adaptive attention score p̃t (Eq. 3). † Here we describe the calibration procedure used to estimate ℓ and report the resulting per-layer attention statistics for each backbone. 16

Preprint

Calibration set. We estimate ℓ† from 150 utterances drawn from LibriSpeech test-other (Panayotov et al., 2015), disjoint from the utterances used for downstream evaluation in §5.2. For each calibration utterance, we run a forward pass through the target backbone and record the full decoder attention tensors A(ℓ) at every layer, with no gradient updates or parameter changes. Calibration takes under 4 minutes on a single A100 and is amortized across all subsequent inputs for that backbone. Per-layer attention mass. For each decoder layer ℓ, we define the mean intra-audio and cross-modal attention mass, restating the two branches of Eq. 3 without the depth partition: T

H

aintra (ℓ) =

u X X X 1 X 1 (ℓ) Ah,i,t , |U| H · |Qa | · Tu t=1

u∈U

T

H

across (ℓ) =

(7)

h=1 i∈Qa

u XXX 1 X 1 (ℓ) Ah,i,t , |U| H · |Qq | · Tu t=1

u∈U

(8)

h=1 i∈Qq

Mean attention mass

where U is the calibration set, Tu is the number of audio tokens in utterance u, and Qa , Qq are the audio-query and text-query row index sets as in Eq. 3. The crossover layer is then ℓ† = min{ℓ : across (ℓ) ≥ aintra (ℓ)}, matching Eq. 4.

1.00 0.75 0.50 0.25 0.00 0

Qwen2-Audio

Phi-4-Multimodal

` =15

10

20 Layer `

Audio-Flamingo

` =13

30 0

10

aintra (`)

20 Layer `

` =18

30 0

across (`)

10

20 Layer `

30

Figure 7: Per-layer intra-audio (aintra (ℓ)) and cross-modal (across (ℓ)) attention mass on the LibriSpeech test-other calibration subset, for Qwen2-Audio (Chu et al., 2024), Phi-4-Multimodal (Microsoft et al., 2025), and Audio Flamingo 3 (Ghosh et al., 2025a) The dashed vertical line marks the identified crossover layer ℓ† for each backbone. Intra-audio attention dominates in early layers and is progressively overtaken by cross-modal attention as depth increases, consistent with Figure 2. Results. As shown in Figure 7, all three backbones exhibit the same qualitative pattern: intra-audio attention mass is highest in early layers and declines monotonically (on average) with depth, while cross-modal attention mass rises correspondingly, crossing over in the middle third of the network. The calibration procedure identifies ℓ† = 15 for Qwen2-Audio, consistent with the pruning schedule used in §5.1, ℓ† = 13 for Phi-4-Multimodal, and ℓ† = 18 for Audio Flamingo 3. The crossover layer therefore shifts across backbones with differing encoder-decoder depth and subsampling rates. A.2.3

T RANSITION P RIOR C OMPUTATION P IPELINE

Figure 8 illustrates the end-to-end computation of the phoneme transition score bt and its relationship to ground-truth phoneme transitions, using a toy example spanning the phonemes /s/-/t/-/r/-/iy/-/m/. The raw derivative dt (Eq. 1) is computed from consecutive encoder hidden states, producing sparse spikes near true phonetic transitions but with substantial token-to-token noise. Gaussian smoothing (Eq. 6) converts this into the continuous transition prior bt , which we correlate against both the discrete ground-truth transition indicator yhard and its Gaussian-smoothed counterpart ysoft to obtain rhard and rsoft respectively. A.2.4

B OUNDARY C ORRELATION ON C OVO ST2 AND IEMOCAP

Section 4 reports boundary-correlation results on LibriSpeech (Figure 3), using the MFA-based forced alignments made available through (CorentinJ, 2019). Here we repeat the same analysis on 17

Preprint

Audio signal with true phoneme segments (forced alignment) /s/

/t/

/iy/

/r/

/m/

Per-frame encoder hidden states ht

dt = dist(ht , ht − 1 )

Eq.~1: Gaussian smoothing

Smoothed Phoneme Transition Score bt (from the encoder) dt (raw derivative) bt (smoothed prior)

1.0 0.5 0.0

rhard = corr(bt , yhard )

rsoft = corr(bt , ysoft )

Ground-truth Phoneme Transitions: yhard (spikes) and ysoft (smoothed) yhard (true boundary = 1, else 0) ysoft (Gaussian-smoothed yhard )

1.0 0.5 0.0

0

5

10 Frame index t

15

20

Figure 8: Schematic of the phoneme-transition-aware prior pipeline. Top: audio waveform with forced-alignment phoneme segments. Second row: per-token encoder hidden states ht , colored by phoneme membership. Third row: raw derivative dt (bars) and the Gaussian-smoothed transition score bt (line). Bottom: ground-truth transition labels, shown as discrete spikes yhard and their Gaussian-smoothed form ysoft , used to compute rhard and rsoft .

our two other evaluation datasets, CoVoST2 (speech translation) (Wang et al., 2021) and IEMOCAP (emotion recognition) (Busso et al., 2008), to confirm that the mid-network correlation peak is not an artifact of LibriSpeech’s read-speech domain. Alignment sources. IEMOCAP ships with official word-, syllable-, and phoneme-level forced alignments as part of its dataset release, generated with Sphinx-III (Busso et al., 2008); we use 18

IEMOCAP Pearson Correlation

CoVoST2 Pearson Correlation

Preprint

0.75 0.60 0.45 0.30 0.15 0.00 0.75 0.60 0.45 0.30 0.15 0.00

Qwen2-Audio 0.67

Phi-4-Multimodal

0.36

0 4 8 12 16 20 24 28 32 0

4

Audio-Flamingo

0.58

0.61

0.32

0.37

8 12 16 20 24 0 4 8 12 16 20 24 28 32

0.60

0.56

0.58

0.28

0.33

0.34

0 4 8 12 16 20 24 28 32 0

Layer Index

4

8 12 16 20 24 0 4 8 12 16 20 24 28 32

Layer Index

rsoft

Layer Index

rhard

Figure 9: Layer-wise boundary correlation (rsoft , rhard ) across Qwen2-Audio, Phi-4-Multimodal, and Audio Flamingo 3, computed on CoVoST2 (top, MFA alignments generated in-house) and IEMOCAP (bottom, official released alignments). Shaded regions mark each backbone’s mid-network sweet spot, consistent with Figure 3. Dashed vertical lines and annotated values mark the peak layer for each curve.

these directly. No comparable off-the-shelf alignment resource exists for CoVoST2, so we generated phoneme alignments ourselves using the Montreal Forced Aligner (MFA) (McAuliffe et al., 2017) on CoVoST2’s English-source audio and transcripts, matching the alignment methodology used for the LibriSpeech release to keep the comparison consistent across datasets.3 For both datasets we recompute bt per layer and correlate against the same yhard /ysoft construction as in Section 4, holding σ = 2 fixed. As shown in Figure 9, both datasets reproduce the mid-network peak observed on LibriSpeech across all three backbones, with rsoft peaking in the same layer range identified by the sweet-spot shading (shared with Figure 3). Peak rsoft is consistently higher on CoVoST2 than on IEMOCAP for every backbone (e.g., Qwen2-Audio: 0.67 vs. 0.60; Audio Flamingo 3: 0.61 vs. 0.58), which is consistent with our finding in Section 5.2 that fine-grained lexical tasks depend more heavily on precise boundary localization than paralinguistic classification does. A.2.5

S ENSITIVITY TO THE S MOOTHING W IDTH σ

Eq. 6 defines the transition prior bt by convolving the raw distance signal dt with a Gaussian kernel of width σ, and we apply the same σ to smooth the discrete forced-alignment labels yhard into the continuous target ysoft . In the main text we fix σ = 2 (encoder tokens). Here we ablate this choice to characterize how rsoft and rhard vary as σ moves away from the selected smoothing width. We sweep σ ∈ {0.5, 1, 2, 3, 4, 5, 6, 8} encoder tokens and recompute rsoft and rhard at each value, holding the layer fixed at the mid-network sweet spot identified in Section 4 for each backbone. Because Phi-4-Multimodal’s audio encoder subsamples more aggressively than Qwen2-Audio’s or Audio Flamingo’s, the same σ in token units corresponds to roughly twice the real-time smoothing 3 IEMOCAP’s original release notes that alignment accuracy was screened but not formally evaluated by the corpus authors; we flag this as a caveat on alignment quality relative to the MFA-based LibriSpeech and CoVoST2 alignments used elsewhere in this work.

19

Pearson Correlation

Preprint

Qwen2-Audio σ (ms) 100 200 300

Phi-4-Multimodal σ (ms) 200 400 600

Audio-Flamingo σ (ms) 100 200 300

2.5

2.5

2.5

0.5 0.0

5.0

σ (tokens)

7.5

5.0

σ (tokens)

rsoft

7.5

5.0

σ (tokens)

7.5

rhard

Figure 10: Sensitivity of the transition-prior correlation to the Gaussian smoothing width σ, on all backbones. Each panel sweeps σ (encoder tokens, with milliseconds on the secondary top axis) and reports rsoft and rhard at that backbone’s mid-network sweet-spot layer. The shaded band and dashed line mark σ = 2, the value used throughout the main text.

Middle Layers

Later Layers

t-SNE dimension 2

Early Layers

t-SNE dimension 1

t-SNE dimension 1

t-SNE dimension 1

Figure 11: Qualitative analysis of Qwen2-Audio encoder token embedding trajectories for a single LibriSpeech utterance, projected via t-SNE and colored by phoneme identity, at early, middle, and late encoder depths. Colored points denote individual tokens where different colors represent different phonemes; Star markers indicate frames at phoneme transitions per forced alignment; Faint gray points show other utterances’ frames for spatial context.

window for Phi-4-Multimodal; Figure 10 therefore also reports σ in milliseconds on a secondary axis for each backbone. As shown in Figure 10, both rsoft and rhard consistently exhibit an inverted U-shaped curve across all backbones, peaking cleanly at σ = 2 tokens. Performance degrades on either side: smaller values fail to account for context distribution, while larger values (σ > 2) excessively blur distinct phonetic transitions. A.2.6

Q UALITATIVE E NCODER E MBEDDING T RAJECTORIES

Figure 11 complements the quantitative diagnostics of Section 4 with a qualitative view of the same phenomenon. We project the per-token encoder hidden states ht of a single LibriSpeech test-other utterance into two dimensions with t-SNE at an early (layer 4), middle (layer 16), and late encoder layer (layer 28) of Qwen2-Audio, and trace the utterance’s token sequence through the projected space. Frames from other utterances are shown in grey to provide a shared spatial reference. At the middle layer, which coincides with the peak of both rsoft and kurtosis (Figure 3), tokens belonging to the same phoneme form compact clusters. At the early and late layers, tokens are less consolidated by phoneme identity and boundary frames are less clearly separated from withinsegment frames, consistent with the lower correlation and kurtosis observed at those depths. Since t-SNE preserves local neighbourhoods but not global distances, we treat this figure as illustrative only. The quantitative evidence for the mid-network sweet spot is given by the correlation and kurtosis analyses in Section 4 and Appendix A.2.4. 20

Preprint

Table 4: Extended results for Phi-4-Multimodal across all compression ratios. Method No Compression

LibriSpeech WER ↓

IEMOCAP Acc. ↑

MMAU-mini Acc. ↑

CoVoST2 BLEU ↑

Latency (s)

TFLOPs

4.85±0.34

39.34±0.29

54.0±0.85

36.06±0.07

1.15

1.403

19.05±0.05 22.60±0.07 22.05±0.07 26.76±0.19 31.60±0.35

0.97 1.01 1.02 1.00 1.02

0.96 0.96 1.00 0.96 1.03

16.34±0.02 13.57±0.01 17.92±0.08 20.23±0.02 31.13±0.30

0.77 0.80 0.81 0.79 0.81

0.77 0.78 0.84 0.77 0.88

25.29±0.02 6.21±0.03 15.67±0.04 15.24±0.10 30.67±0.09

0.61 0.64 0.65 0.63 0.67

0.62 0.63 0.70 0.62 0.74

9.56±0.17 2.50±0.02 13.97±0.03 10.80±0.09 30.04±0.32

0.40 0.46 0.47 0.45 0.45

0.49 0.49 0.60 0.49 0.64

Compression Ratio 0.36 Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

5.64±1.92 5.55±0.06 5.43±0.79 6.60±0.00 4.84±1.03

39.00±0.51 38.28±1.10 39.75±0.36 40.40±0.84 40.93±0.56

Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

7.99±0.68 5.81±2.80 6.53±1.13 6.95±0.07 5.78±1.49

38.03±0.90 36.91±0.99 39.34±0.72 39.74±0.95 39.90±0.20

Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

8.49±6.06 6.38±2.01 7.53±1.09 6.80±0.46 6.04±0.87

35.84±0.25 35.38±0.40 39.04±0.56 38.97±0.77 39.44±0.52

Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

8.92±4.33 7.85±1.52 8.19±2.08 6.89±2.34 6.71±2.08

33.54±0.34 35.91±0.48 38.94±0.43 38.04±0.80 39.07±0.52

52.60±0.89 53.70±0.44 52.70±0.43 54.40±0.00 55.20±0.31

Compression Ratio 0.51 51.30±0.64 53.20±0.92 52.70±1.04 53.60±0.00 54.10±0.32

Compression Ratio 0.64 50.90±0.89 51.90±0.83 52.30±0.63 52.10±0.45 53.50±0.61

Compression Ratio 0.75

† ‡

50.60±1.04 51.30±0.39 51.80±0.48 51.60±0.91 52.50±0.38

In-LM pruning (token elimination performed inside the LLM backbone). Originally developed for the vision modality, adapted here for speech/audio.

A.3 A.3.1

A DDITIONAL E XPERIMENTS E XTENDED R ESULTS ON P HI -4-M ULTIMODAL AND AUDIO F LAMINGO 3

Section 5.2 and Figure 4 reports results on Phi-4-Multimodal and Audio Flamingo 3 at a single compression ratio (0.51) for brevity. Here we provide the extended results for both backbones across all four evaluated compression ratios (0.36, 0.51, 0.64, 0.75), reported in Table 4 for Phi4-Multimodal and Table 5 for Audio Flamingo 3. Consistent with the Qwen2-Audio results in Table 1, AudioTETRIS maintains near-lossless performance across both additional backbones at low compression and shows a widening advantage over baselines as compression increases, indicating that the method generalizes across encoder-decoder architectures rather than being tuned to a single backbone. A.3.2

E XTENDED C OMPONENT- WISE A BLATION (ℓ)

(ℓ)

Section 6.2 reports the component and stage ablation of AudioTETRIS’s fused score st = p̃t · wt at compression ratio 0.51 only. Here we provide the extended results across all four evaluated compression ratios (0.36, 0.51, 0.64, 0.75) and all four benchmarks (LibriSpeech, IEMOCAP, MMAUmini, CoVoST2), comparing the isolated single-stage variants (Only Stage 1, Only Stage 2) against the full two-stage AudioTETRIS pipeline (Stage 1 + Stage 2). Table 6 shows that combining both stages consistently outperforms either stage in isolation across every compression ratio and benchmark, with the gap widening substantially at higher compression - confirming that the phonetic transition prior and the depth-adaptive attention score contribute complementary, rather than redundant information. A.3.3

S INGLE -S TAGE AND A LTERNATIVE M ULTI -S TAGE S CHEDULES

Section 5.1 fixes AudioTETRIS’s schedule at two stages, layers {1, 15}, uniform per-stage split, with layer 1 placed for compute savings (Appendix A.2.2) and layer 15 set by crossover calibration on Qwen2Audio (Appendix A.2.2). Here we ablate two orthogonal design choices: where in the 21

Preprint

Table 5: Extended results for Audio Flamingo 3 across all compression ratios. Method

LibriSpeech WER ↓

IEMOCAP Acc. ↑

MMAU-mini Acc. ↑

CoVoST2 BLEU ↑

Latency (s)

TFLOPs

3.61±0.54

59.11±0.87

62.09±0.45

37.02±0.71

1.29

7.93

36.82±0.10 36.73±0.05 36.18±0.06 35.89±0.02 37.31±0.19

1.13 1.15 1.14 1.21 1.15

6.92 7.09 7.31 6.98 7.34

34.72±0.15 35.18±0.16 36.30±0.32 35.17±0.13 36.26±0.09

0.80 0.87 0.92 0.91 0.85

5.61 5.71 6.18 5.63 6.16

33.05±0.31 34.87±0.26 35.39±0.09 33.90±0.04 35.93±0.15

0.73 0.78 0.82 0.83 0.77

4.53 4.56 5.14 4.60 5.14

27.72±0.20 31.98±0.29 34.25±0.02 33.01±0.06 35.94±0.14

0.51 0.63 0.62 0.66 0.60

3.59 3.68 4.25 3.52 4.26

No Compression

Compression Ratio 0.36 Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

4.02±1.21 4.01±2.10 3.78±1.45 3.83±0.38 3.71±0.45

58.43±0.63 58.23±0.59 57.34±0.32 58.11±0.41 58.94±0.59

Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

6.67±0.91 5.18±1.49 4.97±1.32 5.87±0.18 4.62±0.97

56.45±0.62 57.81±0.97 57.39±0.49 58.91±0.56 58.85±0.34

Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

10.91±1.05 6.59±1.92 6.08±1.94 6.49±0.71 5.72±0.63

53.29±0.33 56.71±0.63 53.98±0.62 57.18±0.21 58.31±0.26

Random SpeechPrune SparseVLM†‡ LTBM AudioTETRIS†

15.69±1.12 11.48±2.63 9.59±1.27 8.94±0.33 6.39±1.03

51.89±1.08 54.78±0.53 52.60±0.49 57.31±0.29 58.09±0.37

60.97±0.23 61.42±0.83 58.83±0.68 61.00±0.65 61.77±0.37

Compression Ratio 0.51 59.62±0.28 61.19±0.29 58.14±0.52 60.79±0.26 61.59±0.72

Compression Ratio 0.64 58.19±0.52 60.48±0.64 56.39±0.93 59.23±0.20 59.85±0.13

Compression Ratio 0.75

† ‡

54.58±0.42 59.59±0.97 53.21±0.28 57.13±0.40 60.92±0.29

In-LM pruning (token elimination performed inside the LLM backbone). Originally developed for the vision modality, adapted here for speech/audio.

Table 6: Ablation results comparing individual stages against full AudioTETRIS across different compression ratios. The best-performing combination per compression ratio block is highlighted. Method

LibriSpeech WER ↓

IEMOCAP Acc. ↑

MMAU-mini Acc. ↑

CoVoST2 BLEU ↑

41.80±0.62 41.70±0.80 42.36±0.21

31.21±0.09 30.93±0.04 31.96±0.04

41.90±0.42 41.20±0.85 42.91±0.36

31.18±0.08 30.55±0.13 31.45±0.06

41.50±0.14 38.70±1.13 41.83±0.15

30.11±0.04 30.85±0.19 31.24±0.11

40.80±0.42 36.10±0.57 41.46±0.32

30.24±0.09 24.29±0.11 31.33±0.07

Compression Ratio 0.36 Only Stage 1 Only Stage 2 Stage 1 + Stage 2

5.50±0.25 9.21±0.36 4.52±0.21

Only Stage 1 Only Stage 2 Stage 1 + Stage 2

7.39±0.09 9.42±0.32 5.77±0.33

Only Stage 1 Only Stage 2 Stage 1 + Stage 2

8.92±0.46 9.62±0.22 6.62±0.29

Only Stage 1 Only Stage 2 Stage 1 + Stage 2

13.78±0.23 15.56±0.57 8.50±0.18

57.45±0.39 57.30±1.19 58.04±0.84 Compression Ratio 0.51 56.37±1.55 56.22±0.21 57.03±0.41 Compression Ratio 0.64 56.68±1.43 55.77±0.57 57.90±0.76 Compression Ratio 0.75 55.02±0.14 50.94±1.06 55.34±1.25

decoder the pruning stages are placed and how the compression budget is allocated across stages once placement is fixed. Schedule parameterization. For stagesPat layers (ℓ1 , . . . , ℓn ) and target keep-fraction K = 1−rate, we assign each stage a weight wi with i wi = 1 and set the per-stage retained fraction n Y

ri = K wi ,

ri = K,

(9)

i=1

so every variant reaches the same target compression rate regardless of how wi is split. Equal weights wi = 1/n give the uniform split used by the main text; w1 > w2 front-loads the budget toward the earlier stage, and the reverse back-loads it. Dense variants keep n large and the split 22

Preprint

uniform, spreading small drops across many stages instead of a few large ones. Table 7 lists all variants tested. Table 8 reports the resulting per-stage compression rate (1 − ri ) × 100% at each evaluated compression rate. For uniform-split variants this is a single number repeated n times; for front-/back-loaded it differs stage to stage, with front-loaded removing a larger share early and back-loaded the reverse. Table 7: Candidate pruning schedules. Placement variants use the uniform split; front-/back-loaded vary only wi at the default layers {1, 15}. Schedule

Layers

Single-stage @ 1 Single-stage @ 15 Two-stage {1, 15} (default) Two-stage {1, 19} Three-stage {1, 8, 15} Front-loaded Back-loaded Dense (every ∼3 layers, full depth)

1 15 1, 15 1, 19 1, 8, 15 1, 15 1, 15 1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31

n stages

Split

1 1 2 2 3 2 2 11

– – Uniform Uniform Uniform w = (0.7, 0.3) w = (0.3, 0.7) Uniform

Table 8: Per-stage token-drop rate (% of currently live tokens removed at that stage) for each schedule, at every evaluated compression rate. Equal per-stage rates are written once as “X% (×n)”; front-/back-loaded list rates stage by stage.

Schedule

Layers

rate 0.36

rate 0.51

rate 0.64

rate 0.75

Single-stage @ 1 Single-stage @ 15 Two-stage (default) Two-stage {1, 19} Three-stage Front-loaded Back-loaded Dense (11 stages)

1 15 1, 15 1, 19 1, 8, 15 1, 15 1, 15 1,4,. . . ,31

36% 36% 20% (×2) 20% (×2) 14% (×3) 27%/13% 13%/27% 4% (×11)

51% 51% 30% (×2) 30% (×2) 21% (×3) 39%/19% 19%/39% 6% (×11)

64% 64% 40% (×2) 40% (×2) 29% (×3) 51%/26% 26%/51% 9% (×11)

75% 75% 50% (×2) 50% (×2) 37% (×3) 62%/34% 34%/62% 12% (×11)

Table 9: Schedule ablation, LibriSpeech WER (%, lower is better), Qwen2-Audio, across compression rates. Schedule Single-stage @ 1 Single-stage @ 15 Two-stage {1, 15} (default) Two-stage {1, 19} Three-stage {1, 8, 15} Front-loaded Back-loaded Dense (11 stages)

rate 0.36

rate 0.51

rate 0.64

rate 0.75

±0.14

±0.17

±0.36

18.78±0.41 12.89±0.82 8.50±0.18 8.96±0.31 8.12±0.73 10.73±0.28 14.53±0.42 12.92±0.36

6.51 5.44±0.61 4.52±0.21 4.78±0.38 4.36±0.71 4.99±0.69 5.73±0.31 7.13±0.45

8.92 6.93±0.42 5.77±0.33 6.02±0.35 5.93±0.71 6.72±0.76 7.34±0.62 8.79±0.57

12.40 8.94±0.48 6.62±0.29 6.94±0.62 6.47±0.72 7.68±0.29 9.38±0.34 10.09±0.62

Results. We find that single-stage compression at layer 15 (6.93% WER at rate 0.51; Table 9) consistently outperforms layer 1 (8.92%), highlighting the role of text-conditioned attention in token ranking. Splitting the compression budget across two stages improves upon single-stage baselines across all rates, regardless of the chosen layers. Within two-stage setups, our default {1, 15} schedule achieves the strongest performance (5.77% at rate 0.51), outperforming {1, 19} (6.02%) by 0.2-0.5 percentage points across the sweep. The three-stage schedule {1, 8, 15} yields small gains over the two-stage default at three of the four evaluated rates - 4.36% vs. 4.52% WER at rate 0.36, 6.47% vs. 6.62% at rate 0.64, and 8.12% vs. 23

Preprint

Table 10: Efficiency and downstream-task trade-offs at compression rate 0.51. Latency is relative to the uncompressed baseline (1.00×). Schedule Single-stage @ 1 Single-stage @ 15 Two-stage {1, 15} (default) Two-stage {1, 19} Three-stage {1, 8, 15} Front-loaded Back-loaded Dense (11 stages)

WER

IEMOCAP

MMAU

CoVoST

Latency

TFLOPs

8.92±0.17 6.93±0.42 5.77±0.33 6.02±0.35 5.93±0.71 6.72±0.76 7.34±0.62 8.79±0.57

53.55±0.50 55.83±0.52 57.03±0.41 57.41±0.47 55.68±0.37 56.91±0.19 55.93±0.31 57.62±0.81

39.81±0.41 41.52±0.24 42.91±0.36 43.18±0.18 41.67±0.72 41.48±0.39 41.44±0.53 43.30±0.64

27.05±0.05 29.53±0.09 31.45±0.06 31.69±0.07 31.45±0.10 30.20±0.18 26.68±0.04 30.90±0.07

0.98 0.99 1.00 1.03 1.04 1.02 1.06 1.21

5.51 5.92 6.13 6.13 6.32 5.71 6.21 6.93

8.50% at rate 0.75 - but this advantage is not monotonic: at rate 0.51 the ordering reverses, with the two-stage default outperforming three-stage (5.77% vs. 5.93%). Given this inconsistency and the added latency overhead of a third pruning stage (Table 10), we retain the two-stage {1, 15} schedule as the default configuration used throughout the main text, rather than the marginally stronger but less consistent three-stage variant. Redistributing the budget within two-stage setups reveals a clear asymmetry: front-loading (larger early cut) remains competitive with the default (4.99% vs. 4.52% at rate 0.36), whereas back-loading degrades sharply under higher compression (14.53% vs. 10.73% at rate 0.75). This aligns with our analysis in Appendix A.2.2: heavy pruning in later layers, where cross-modal attention dominates, incurs a higher penalty than pruning early while representations remain largely intra-audio. Finally, the 11-stage dense schedule underperforms at all rates, showing that frequent, fine-grained pruning could accumulate more errors for the ASR task. Table 10 highlights the efficiency trade-offs at rate 0.51. We observe that the two-stage compression schedule offers one of the most favorable balance between accuracy and computational cost. In contrast, three-stage and dense schedules introduce a slight overhead (1.04× and 1.21× latency, respectively) without consistent downstream gains, confirming the 2-stage compression is an effective overall configuration. A.3.4

Q UALITATIVE E XAMPLES OF T OKEN S URVIVAL

Figure 12 shows token survival at the final pruning layer for three representative LibriSpeech utterances. The examples are consistent with our quantitative finding (§6) that the two stages contribute complementary rather than redundant information: Stage 1 supplies dense, distributed protection against losing phonetic detail, while Stage 2 contributes a smaller number of additional, task-relevant tokens that Stage 1 alone would not have prioritized.

24

Preprint

80

Mel

60 40 20 0 Only Stage 1 Only Stage 2 Stage 1 + Stage 2 0

100

200

300

400

500

600

Audio token position (post-pool encoder frames)

700

(a) Example 1: “their piety would be like their names, like their faces, like their clothes. . . a gift tenfold more acceptable than his elaborate adoration” 80

Mel

60 40 20 0 Only Stage 1 Only Stage 2 Stage 1 + Stage 2 0

100

200

300

400

500

600

Audio token position (post-pool encoder frames)

700

(b) Example 2: “now you will see in these studies that the moment the white is inclosed properly. . . the little diamonds in the round window will tell as jewels if they are gradated justly” 80

Mel

60 40 20 0 Only Stage 1 Only Stage 2 Stage 1 + Stage 2 0

100

200

300

400

500

600

Audio token position (post-pool encoder frames)

700

(c) Example 3: “well then, last, here is turner’s greek school of the highest class, and you define his art absolutely as first the displaying intensely and with the sternest intellect of natural form as it is, and then the envelopment of it with cloud and fire”

Figure 12: Token survival at the final pruning layer for three representative utterances (ground-truth transcripts shown under each panel). Rows within each panel (top to bottom): mel spectrogram, Only Stage 1 (phoneme-transitions only), Only Stage 2 (attention-only), Stage 1 + Stage 2 (attention × phoneme-transitions, our full method). Orange = kept uniquely by Only Stage 1; mauve = kept uniquely by Only Stage 2; blue = kept uniquely by Stage 1 + Stage 2; grey = pruned.

25

Related documents

• AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs#1297952
• Libra: Taming Attention Workload Skew in Long-Context LLM Training with Bounded Sequence Pool#1014774
• AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression#1202121
• Libra: Taming Attention Workload Skew in Long-Context LLM Training with Bounded Sequence Pool#405615
• Data-Free Pruning of Self-Attention Layers in LLMs#1141466