An Exact Instrument for State Usage in Selective State-Space Models, and the Input-Driven Migration It Reveals
arXiv:2607.11796v1 [cs.LG] 13 Jul 2026
Raktim Bhattacharya Aerospace Engineering, Texas A&M University Abstract Selective state-space models such as Mamba route information through a bank of first-order modes whose input coupling is set by a learned selection mechanism. We give an exact instrument for measuring how a trained model uses these modes. Because the state matrix is diagonal, each channel’s output decomposes exactly into per-mode contributions, and a per-(layer, channel, window) Gram tensor yields the exact output error of dropping any subset of modes, offline, at any budget. Validated against the reference implementation to a relative error of 2.3 × 10−7 on the Mamba-1 family where it is exact, the instrument predicts a layer’s deployed pruning error to a median relative deviation of 5 × 10−7 over 4,464 configurations, its floor set by the reconstruction. Applying the instrument across the Mamba-1 family (130M–2.8B), the deployed 7B Falcon-Mamba, and Mamba-2, we find that trained models re-allocate their state space with the input: which modes carry the signal migrates across contexts, and at the most affected layers a per-input oracle roughly halves the output error of a fixed mode set. Frozen-signal counterfactuals attribute the migration primarily to the input-dependent write map Bt ; the timestep usually identified with selectivity carries almost none of it. Input-scheduled mode pruning on this measurement outperforms static, Hankel-based, and layer-adaptive rankings at every scale from 130M to the deployed 7B Falcon-Mamba, and at half the state budget it matches the unpruned model. Because the scheduler reads each window’s mode usage from a first pass, this demonstrates realizable headroom; we claim no deployed compute or memory saving.
1
Introduction
Selective state-space models (SSMs) have made linear recurrences competitive with attention for long-sequence modeling (7, 3). A Mamba layer maintains, in each channel, a small linear state driven by a first-order recurrence with a diagonal state matrix; the layer is selective in that the input, output, and timestep matrices are functions of the current token. The state matrix itself is fixed after training. A layer is therefore a bank of fixed first-order modes whose coupling to the signal is scheduled, token by token, by the selection mechanism. How a trained model uses that bank is not known. The question matters in two ways. For efficiency, the state is the memory and compute bottleneck of an SSM, and a line of work prunes or reduces it (10, 1); every such method rests on an assumption about which modes matter. For interpretability, the modes are the layer’s internal degrees of freedom, and their usage is a direct measure of what the layer stores. Both uses require an exact measurement of state usage. Existing mode-importance criteria are static and approximate. Output-aware pruning of Mamba-2 scores modes once, from calibration statistics, and prunes a fixed set (10); activity-based criteria rank modes by the timestep (1); the classical reduction they approximate, balanced truncation, is defined for time-invariant systems and bounds a balanced surrogate; the bound does not cover the deployed pruned layer (11, 6). None measures, per input, which modes a trained layer is using, and none is exact. An exact measurement is possible because the state matrix is diagonal: the modes are decoupled, and a channel’s scalar output is an exact sum of per-mode contributions. Accumulating the outer products of these contributions over a window yields a small Gram tensor, per (layer, channel, window), from which the exact output error of pruning any subset of modes follows in closed form, offline, at any budget. The decomposition is exact, and we validate it in two ways: against the reference implementation to a relative error of 2 × 10−7 on the diagonal case where it is exact, and we verify that it predicts the output error a pruned layer actually produces in the running model to a median relative deviation of 10−6 over 4,464 configurations spanning five models, every layer, and three budgets. Across every model we test, the set of modes carrying a layer’s output migrates with the input: which modes matter changes from one context to the next. We quantify the migration by the gap between input-scheduled and static 1
mode selection at a fixed budget, and the gap is large: at the most affected layers, static selection incurs roughly twice the error of an input-scheduled oracle. The migration is present in the released Mamba-1 family from 130M to 2.8B parameters, in the deployed 7B Falcon-Mamba, and in Mamba-2, so it is a property of the selective-SSM design. We then ask what causes the migration and whether it can be exploited. Freezing each selective signal at its corpus mean, one at a time, isolates the cause: the input-dependent write map Bt carries the migration, and freezing it removes most of the effect; the readout Ct carries part of it; and the timestep ∆, the signal usually identified with selectivity, carries almost none. A two-pass scheduler that re-selects modes per window recovers most of the loss that static pruning incurs, and at half the state budget it matches or exceeds the unpruned model; a static timestep-activity criterion, consistent with the mechanism, collapses. Contributions. • An exact state-usage instrument. The diagonal state matrix gives an exact per-mode output decomposition and a Gram tensor that returns the exact pruning error of any mode subset offline, validated to 2×10−7 and matching a pruned layer’s deployed output error to a median deviation of 10−6 across 4,464 configurations (Section 3). • Input-driven mode migration. Using the instrument, we show that trained selective SSMs re-allocate their state across inputs, quantify it by the scheduled-vs-static gap, and show it holds across the Mamba-1 family, the 7B Falcon-Mamba, and Mamba-2 (Section 4.1). • Mechanism. Frozen-signal counterfactuals attribute the migration primarily to the input-dependent write map Bt on both Mamba-1 and Mamba-2; the timestep carries almost none of it (Section 4.2). • Consequence. Two-pass input-scheduled mode pruning outperforms static, modal-HSV, and LAST rankings at equal budget at every scale to 7B and, at half the state budget, matches the unpruned model on perplexity; because it reads the mask from a first pass, this is realizable headroom, and we claim no deployed saving. On downstream tasks that use long-range state it recovers most of static pruning’s loss (Section 4.3).
2
Related Work
Pruning and reducing selective SSMs. The state is an SSM’s compute and memory bottleneck, and several methods shrink it. GHOST (10) prunes Mamba-2 states by an output-aware score computed once from forward-pass statistics, removing half the state at a cost of about one point of perplexity; PerfMamba (1) profiles and removes low-activity states; unstructured methods prune SSM weights by magnitude or gradient (14). Each of these methods chooses a single fixed set of states to keep. Our static baseline is exactly this choice, and the migration gap measures what it forgoes: because the important set moves with the input, a fixed set incurs up to twice the oracle’s error at the most affected layers. The mechanism result also bears on activity-based criteria: the timestep does not carry the allocation signal, so a criterion scored on timestep activity cannot capture the migrating allocation that an output-aware score reads. Reducing neural SSMs by system-theoretic tools. A recent line of work reduces deep SSMs with tools from linear-systems theory. LAST (8) prunes states by a layer-adaptive global Hankel-energy threshold; Schwerdtner et al. (13) regularize the Hankel singular values during training to make the model compressible; Ezoe and Sato (4) apply balanced truncation to the diagonal state-space layers of S4; Forgione et al. (5) give a system-theoretic order reduction of deep SSMs. These methods build one reduced model, time-invariant per channel, from training-time or calibration statistics, and fix the mode set in advance. We find that the trained model’s allocation moves from one input to another, so a single reduced set leaves error unrecovered. Our instrument is complementary: these methods could use it to check whether a fixed reduction is safe for a given input. Model reduction for linear systems. The output-aware scores above approximate balanced truncation (11), whose H∞ error bound (6) is defined for a time-invariant system and controls the error of a balanced surrogate; it does not bound a directly pruned layer. A selective SSM is time-varying, and its per-mode importance is a function of the operating point; the exact per-mode error (4) we use is a measurement on the realized trajectory. The pruning we deploy drops modes in the layer’s native diagonal basis, modal truncation rather than a balancing transform,
2
so the near-optimality of balanced truncation is not invoked; consistent with this, a modal-HSV ranking built from empirical per-mode Gramians underperforms realized-energy ranking in our experiments (Section 4.3). The timevarying reduction literature (15, 12) provides the Gramian and Hankel-singular-value language for the phenomenon, which we use in the analysis; it does not certify the deployed pruning. Interpreting Mamba. Two lines of work characterize what a Mamba layer computes. Hidden-attention methods reformulate the selective recurrence as a data-controlled linear operator and extract implicit token-to-token attention (2); auto-encoder probes measure which token contents a layer forgets, and tie forgetting to pretraining frequency (9). These give an attention view and a token-content view; we read the state directly, asking which of a layer’s internal modes carry the signal for each input. All three describe the same layer; only the state view makes the reduction consequence exact, because the modes are the coordinates on which the reduction acts. To our knowledge, input-resolved per-mode importance (the effective order of a selective layer as a function of the input, and the gap between input-scheduled and static mode selection) has not been measured before. We state this as the gap our instrument fills; we make no priority claim about the phenomenon.
3
The Instrument
3.1
Selective SSM layer
We use the Mamba-1 layer (7); the Mamba-2 case is treated at the end of the section. Write H for the number of channels (the expanded inner width) and N for the state size. In channel h, at step t, the layer holds a state xh,t ∈ RN driven by a scalar input uh,t (the post-convolution, post-activation hidden signal). The recurrence is xh,t,i = Āh,t,i xh,t−1,i + ∆h,t Bt,i uh,t , Āh,t,i = exp ∆h,t Ah,i , (1) for each state index i = 1, . . . , N , where Ah ∈ RN is the fixed diagonal state matrix (Ah,i < 0, learned as Ah,i = − exp(·)), Bt , Ct ∈ RN and the timestep ∆h,t > 0 are the selective signals produced from the current token, and Dh is a skip term. The channel output, after the layer’s output gate gh,t = SiLU(zh,t ), is P N yh,t = gh,t C x + D u (2) h h,t . i=1 t,i h,t,i The state matrix is diagonal and fixed; the token dependence enters only through Bt , Ct , ∆h,t , and the gate. Each channel is therefore a diagonal realization of order N with fixed continuous-time dynamics (Ah ; the discrete decay Ā = exp(∆A) still moves with the token) whose input coupling, output coupling, and timestep are functions of the input: an input-parameterized (quasi-LPV) system, linear time-varying along each realized trajectory. The channel is a bank of N first-order modes, each a real pole Ah,i < 0, written and read on a per-token schedule.
3.2
Exact per-mode decomposition
Because Ah is diagonal, the states xh,t,i do not interact across i, so the output (2) is an exact sum of per-mode contributions. Define the post-gate output of mode i in channel h, mh,t,i = gh,t Ct,i xh,t,i ,
fh,t = gh,t Dh uh,t ,
(3)
P
so that yh,t = i mh,t,i + fh,t exactly. Pruning P a mode means omitting its contribution from the output. For a kept S set S ⊆ {1, . . . , N }, the pruned output is ŷh,t = i∈S mh,t,i + fh,t , and the squared output error over a window of length L is a quadratic form in the dropped modes: L X t=1
S yh,t − ŷh,t
2
=
X X
Gh,ij ,
Gh,ij =
L X
mh,t,i mh,t,j .
(4)
t=1
i∈S / j ∈S /
The Gram matrix Gh ∈ RN ×N is symmetric positive semidefinite; we store its upper triangle, N (N +1)/2 numbers per (layer, channel, window). Equation (4) gives the exact output error of pruning any subset S, in a single pass. For the nested family of top-r sets under a mode ordering π, the error at every budget r is a single reverse cumulative sum over Gh . This is the object GHOST (10) and activity criteria (1) approximate: they score modes once and keep a fixed set, whereas (4) returns the exact cost of any set, resolved per input. 3
3.3
Validation and deployment cross-check
The instrument recomputes each mixer’s internal signals from its captured input and reconstructs the output through the layer’s own weights. We verify it in two ways. First, reconstruction: on three windows by three layers per model, the reconstructed mixer output matches the reference implementation to a relative error of at most 2.3 × 10−7 on the Mamba-1 family, 2.6×10−7 on Falcon-Mamba, and 1.0×10−6 on Mamba-2 (whose pre-norm decomposition, below, carries one further gated normalization). Second, deployment: pruning a set of modes inside the running model and measuring that layer’s actual output error against the reference forward’s own SSM output agrees with the value (4) predicts offline to a median relative deviation of 1 × 10−6 over 4,464 configurations (five models, every layer, six windows, r ∈ {4, 8, 12}), maximum 3.6 × 10−5 ; the same sweep bounds the reconstruction error at the pre-projection output by 6.9 × 10−6 . The pruned side necessarily runs through the per-mode decomposition (the reference kernel has no per-channel mode mask), so the deviation floor is set by that reconstruction error; the Gram identity itself is exact.
3.4
Measures of state usage
From the Gram tensor and the per-mode energies eh,i = Gh,ii we read off, per layer: Migration gap. Fix a budget r and a per-window keep-set rule that selects a kept set Sw for each evaluation window w. Write the layer’s energy-weighted relative output error over the evaluation set as v uP P P (w) u w h i,j ∈S Gh,ij / , (5) E(r) = t P P P w(w) 2 w h t (yh,t ) the total dropped energy (4) over the total output energy, both summed over channels and windows. Let Estatic (r) use one fixed set S ranked by mean per-mode energy on disjoint calibration windows, and Ediag (r) use the per-window top-r set Sw by that window’s own per-mode energies ei = Gii . Their ratio ρ(r) = Ediag (r)/Estatic (r) measures the excess error of a static set: ρ ≈ 1 means the important set is fixed, and small ρ means that it migrates with the input. The static set is ranked on disjoint calibration windows while the per-window set reads the evaluation window’s own energies, so ρ is an in-sample oracle ratio and measures headroom; the churn metric, the frozen-Bt result, and the switched-system experiment below establish that the gap reflects genuine cross-context migration. The gap has a closed form that identifies the source of a static set’s excess error. Fix a channel h and write (w) w ei = Gh,ii for its per-mode diagonal energy in window w; neglecting the off-diagonal Gram terms, dropping the set w b 2 (D) = P b2 b2 D ⊆ {1, . . . , N } costs E i∈D ei , and Estatic (w), Ediag (w) denote this cost for D the complement of S and of Sw respectively. Proposition 1. For each channel and window, with S the static top-r set and Sw the per-window top-r set by ew , X X b2 b2 E ew ew static (w) − Ediag (w) = i − i ≥ 0, i∈Sw \S
i∈S\Sw
the net energy of the modes that migrate into the top-r set. Proof. The two dropped sets share the complement of S ∪ Sw , so their costs differ only on the symmetric difference, giving the identity. For the inequality, both sets keep r modes, so |Sw \ S| = |S \ Sw |; and since Sw holds the r largest w energies, ew i ≥ ej for every i ∈ Sw \ S and j ∈ S \ Sw . Pairing the two equal-size sets and summing makes the difference non-negative. This establishes the diagonal ρ ≤ 1: the top-r churn |Sw \ S| sets the support of the gap, and the energy of the churning modes sets its size. The gap is governed by an energy-weighted churn. Two measurements connect the theorem to the full-Gram quantity we report (the inequality itself holds in every channel-window case, so it serves only as a consistency check on the pipeline). Across layers, the diagonal gap matches the full-Gram ρ at correlation 0.96, and the energy-weighted churn predicts 1 − ρ at Spearman 0.80 against 0.74 for raw set churn. The full-Gram gap adds only the off-diagonal cross-term difference; it is small except in the static tail, where it can push the measured ρ slightly above one (layers 21–22 of 130M reach 1.003); the theorem covers only the diagonal quantity. Churn. The mean Jaccard distance between the top-r mode sets of two windows, split into within-domain and cross-domain pairs, with a half-window null (the churn between the two halves of the same window; median 0.22 at 4
130M); migration appears as cross-domain churn above the null (median 0.29). The margin is modest: cross-domain churn exceeds within-domain in every layer, but the pilot ratio was 1.28×, below the 1.5× we pre-registered. Churn is corroborating evidence; the migration case rests on the gap ρ, the frozen-signal counterfactuals, and the switchedsystem experiment. P P Effective order. The participation ratio PRh = ( i eh,i )2 / i e2h,i , the number of modes that carry the channel’s energy: the effective order of the realization at the operating point.
3.5
Mamba-2
In Mamba-2 (3) the state decay is scalar per head, so the per-mode decomposition (3) holds at the pre-norm SSM output; the gated RMSNorm that follows couples modes within a position, so the exact offline-Gram identity (4) does not carry past the norm. We therefore compute the same per-mode outputs and the migration gap online, per window, and measure end-to-end pruning error empirically (Section 4.3). The state size N = 128 also makes storing Gh impractical, which the online computation avoids. The instrument’s exactness is thus specific to the per-channel diagonal case (Mamba-1, Falcon-Mamba); the state-usage measures transfer.
4
Migration, Mechanism, and Consequence
Setup. We audit the released Mamba-1 checkpoints (130M, 370M, 790M, 1.4B, 2.8B), the 7B Falcon-Mamba (16), and two Mamba-2 checkpoints (130M, 780M). Windows are 1024 tokens, 32 per domain from three domains (English prose, code, technical text), with a disjoint evaluation set of 24 windows per domain; all seeds are fixed. The evaluation windows are later windows of the same domain streams, disjoint at the window level, though a document may span both sets; any residual leakage helps the static baseline, whose ranking comes from the calibration windows, so the reported gaps are conservative. Every reported number carries the reference-vs-code validation of Section 3. Unless stated, the budget is r = 8 of N = 16 for Mamba-1 and Falcon-Mamba, and r = 64 of N = 128 for Mamba-2 (one-half of the state).
4.1
Input-driven mode migration
The set of modes carrying a layer’s output changes with the input. At the most affected layer of each model the migration gap reaches ρ ∈ [0.44, 0.57]: an input-scheduled oracle halves the output error a static top-r set incurs at the same budget. The effect is present at every scale of the Mamba-1 family (minimum ρ(8) between 0.51 and 0.54), in the deployed 7B Falcon-Mamba (0.48), and in Mamba-2 (0.44–0.57). Cross-domain churn exceeds within-domain churn in every layer of every model (368 of 368 layers across the eight models), at a modest median ratio of 1.15–1.32 (Section 3). The migration is localized in depth, and its geometry depends on the architecture. In the small Mamba-1 models (130M–790M) the migrating layers form a contiguous mid-network band at relative depth 0.6–0.75, while the final layers are static (ρ → 1); at 1.4B and above and in Falcon-Mamba the strongly migrating layers are punctuated, located early-to-mid network; Mamba-2 migrates throughout depth, with architecture-dependent locations and no static tail (Figure 1). The effective order shows the same pattern: the participation ratio rises from about two modes at the ends of the network to five–eight in the migrating band. We report the contiguous-band geometry as a Mamba-1 finding; the phenomenon itself is general.
4.2
Mechanism
Three token-dependent signals could account for the migration, each with a distinct role in the realization. The write map Bt sets the instantaneous input coupling, which modes the current token writes into; the readout Ct sets the instantaneous output coupling, which modes the output reads; and the timestep ∆ enters twice in (1), as the effective poles Ā = exp(∆A) and as a gain on the write term. Within a channel, however, ∆h,t is common to all N modes, so its write-gain role is mode-uniform: of the three signals, only Bt (and Ct at readout) can differentially select among modes at a token, while ∆ differentiates them only through the exponent exp(∆Ah,i ). We isolate each by rerunning a layer’s per-mode decomposition with one signal frozen at its corpus mean and measuring the migration gap under the counterfactual (Table 1, Figure 2). Freezing Bt removes the migration: the band-median gap rises from 0.70 to 0.96
5
Figure 1: Migration gap ρ(8) = Eoracle /Estatic (left) and effective order (right) versus relative depth, over six models (Mamba-1 130M–2.8B and Falcon-Mamba-7B). Coloured lines are individual models (shared legend); the bold black line is the across-model median with an interquartile ribbon; the shaded region is the migrating band (relative depth 0.4–0.75). Left: ρ < 1 (dashed) means a per-input oracle reconstructs the layer output with less error than any fixed mode set; the median dips through the band and each model’s strongest layer, marked by a dot in the model’s colour, reaches ρ ≈ 0.5: the static set’s error is twice the oracle’s at the same budget. Right: the participation ratio rises from about two modes at the ends of the network to five–eight in the same band, so migration concentrates in the layers that spread their output over more modes. Table 1: Migration gap under frozen-signal counterfactuals: Mamba-1 median over the migrating band, Mamba2 median over all layers (it migrates throughout depth, with no static tail to exclude). Freezing the write map Bt removes most of the migration (ρ → 0.96); freezing the timestep ∆ does not. On Mamba-1, where we verified it, each frozen variant’s total output energy stays within an order of magnitude of the real layer’s, and the energy-matched control (Section 4.2) rules out a renormalization artifact. Mamba-1 130M Mamba-2 130M Mamba-2 780M real freeze Bt freeze Ct freeze ∆
0.70 0.96 0.84 0.72
0.77 0.96 0.90 0.74
0.77 0.97 0.90 0.77
on Mamba-1 and from 0.77 to 0.96–0.97 on Mamba-2, and cross-domain churn collapses. Freezing Ct removes part of it; freezing ∆ (both roles at once) leaves the gap and the churn essentially unchanged. The migration is therefore an input-coupling effect: the write map selects, token by token, which modes the signal drives, and freezing it pins that selection to a single average pattern on both architectures. The readout coupling Ct supplies the remainder; the timestep ∆, the basis of activity pruning criteria, does not carry the migration, consistent with its mode-uniform write gain, which leaves the poles as its only mode-differential action. Two controls confirm the attribution. Keeping only Bt live (freezing both Ct and ∆) reproduces much of the migration on its own (ρ = 0.87 against the real 0.70 and the frozen 0.96), so Bt is both necessary and largely sufficient. And matching the frozen-Bt layer’s total output energy to the real layer’s leaves the gap unchanged (0.964 → 0.968), which rules out an energy-renormalization artifact: the ρ → 1 under freeze-Bt is a loss of context-dependence. Consistent with the mechanism, the migrating modes are the slow modes. Ordering each mode by its trained ¯ h ), the instability of top-r membership across contexts rises with the time constant from its time constant 1/(|Ah,i | ∆ minimum near one token toward the slow modes, reaching 0.78 in the slowest decile against a baseline near 0.3, at both 130M and 790M (Figure 3). Slow modes integrate the input-gated writes over long horizons, so the direction a slow mode holds, and hence which slow modes the input reaches, is whatever the context has written.
6
Figure 2: Frozen-signal mechanism at Mamba-1 130M: migration gap ρ(8) (left) and cross-domain churn (right) per layer, each selective signal frozen at its corpus mean. Real (black); freeze Bt (blue); freeze Ct (rust); freeze ∆ (green dashed). The shaded region is the migrating band (layers 14–17). Freezing the write map Bt lifts the band-median gap from 0.70 to 0.96; freezing the readout Ct lifts it partway (0.84); freezing the timestep ∆ overlays the real curve (0.72). Freezing Bt or Ct collapses the cross-domain churn, while freezing ∆ leaves it unchanged, so the timestep carries neither the migration nor the churn.
4.3
Consequence
If the important set migrates, then re-selecting modes per input should outperform a static choice at equal budget. The per-layer error (4) is exact; the end-to-end perplexity below is measured empirically, since pruning shifts the inputs of downstream layers and the per-layer identity does not compose. We compare, at each budget, three static rules and the scheduler: a calibration energy ranking (static); a modal Hankel ranking that scores each mode by the Hankel singular value built from its empirical per-mode Gramians (realized write and readout second moments accumulated over the calibration windows), instantiating the output-aware / Hankel-energy scoring principle of GHOST and LAST on our per-mode outputs (modal-HSV); a layer-adaptive allocation of the same scores with the layer-wise cumulative normalization of Gwak et al. (8), which keeps the same total budget but distributes the kept modes non-uniformly across layers (LAST; we normalize the realized-Gramian score rather than the LTI H∞ norm of 8, which is not defined for a token-varying realization; that method was designed for time-invariant S4/S5 stacks); and a two-pass input-scheduled set (scheduled, a two-pass oracle: a first pass measures the scored window’s mode energies, a second prunes to that window’s top-r, so this column quantifies headroom). Table 2 and Figure 4 report held-out perplexity at half the state budget across the Mamba-1 family (130M–2.8B), the deployed 7B Falcon-Mamba, and Mamba-2, with 95% bootstrap confidence intervals over the evaluation windows. At the quarter budget r = 4 the same ordering holds with larger gaps: at 130M, static reaches perplexity 19.8 and modal-HSV 20.8, LAST collapses to 234 (its layer-adaptive allocation strips some small-model layers below viability), while scheduled stays at 14.2 (gap CI [+1.54, +2.16] above the unpruned 12.4). A timestep-activity ranking and random mode sets, the two rules the mechanism predicts should fail, collapse: at 130M, r = 8 they reach perplexity 52.5 and 127. Input-scheduled selection outperforms every static rule it is compared against, at every scale: at 7B it is 0.38 perplexity below the static energy ranking and further below modal-HSV and LAST, and this gap stays substantial as scale grows (Table 2); no static rule follows a set that moves with the input. Separately, the scheduled model also falls below the unpruned model at half budget, with a paired bootstrap CI excluding zero at every scale we test; the margin shrinks with scale, from 0.54 perplexity at 130M to 0.04 at 7B. The window bootstrap treats contiguous same-stream windows as exchangeable; a circular block bootstrap (length-4 blocks within domain), which respects within-stream autocorrelation, widens the 7B gap CI to [−0.073, −0.020] and still excludes zero. A control identifies the source of this margin: selecting the mask on the first half of each window and scoring the second half does not outperform the unpruned model, while the same-window mask still does under the same continuation scoring. The sub-unpruned margin is therefore same-window adaptation: the mask must be read from the tokens being scored. The control rules out a transferable denoising effect. The comparisons against static rules, which share the protocol, are unaffected. The same ordering holds on Mamba-2 (Table 2).
7
Figure 3: Timescale structure at Mamba-1 130M (blue) and 790M (green): median mode energy (left) and top¯ h ), binned into deciles. r membership instability 4p(1−p) (right) versus per-mode time constant τ = 1/(|Ah,i | ∆ Shaded: the slow modes (τ > 10 tokens). Both quantities rise toward the slow modes: median energy grows by over an order of magnitude, and the instability of top-r membership rises from a fast-mode baseline near 0.28 to 0.78 in the slowest decile, in both models. We also ran the released GHOST code natively on Mamba-2 at the same 50% budget (Table 4). Under this table’s protocol its perplexity exceeds the realized-energy static ranking’s at both scales and lies 2.98 (130M) and 1.44 (780M) above the scheduler. Because GHOST’s importance score is an output-aware realized-energy quantity, the comparison could turn on calibration data; we therefore recomputed it with GHOST calibrated on our own 96 calibration windows, the same data the static ranking uses. The result is nearly unchanged (13.98 → 13.86, 8.77 → 8.71) and remains above static, so the gap is a property of the static-vs-input-scheduled distinction. On GHOST’s own WikiText-2 protocol the pruned models cost +3.2 and +1.16 word-perplexity; the 780M figure matches its reported ∼ 1-point operating point, while 130M is more sensitive to 50% pruning at small scale. The released code’s calibration hooks attach to the reference forward, which transformers bypasses whenever the fused Mamba kernels are installed, so on such machines the released pipeline silently prunes nothing; we verified the no-op and forced the reference path in a wrapper, leaving the method’s code unchanged and confirming that the resulting masks zero 50.0% of the state at both scales and both calibrations. The scheduling advantage carries to downstream tasks where the state is used: on LAMBADA, static pruning to r = 8 collapses 790M accuracy from 0.63 to 0.41 and the two-pass oracle mask (read from the scored passage) recovers 86% of the drop; on tasks that do not stress long-range state (PIQA, ARC-easy) pruning to half budget has almost no effect and the two rankings are within noise. The scheduled result uses a full first pass to read the window’s energies, and the headroom it reaches is not cheaply captured. Under continuation scoring at half budget, the fraction of the two-pass gain captured grows monotonically with how much of the scored window the mask has seen: a regime-conditioned mask (one static set per domain, applied by the window’s domain) captures 2–6% (130M–7B), despite the domain being almost perfectly readable from the same energies (the probe at the end of this section); a 256-token prefix estimator captures 2–6%, and 5–9% after a permode ramp-up correction learned on calibration windows; selecting on the first half of the window captures 18–26% on the second. The capture fractions are protocol-sensitive: scoring the continuation from position 128 rather than from the prefix end credits the same prefix estimator with 14–18%; we report the stricter accounting. The important set moves within a regime and within a window, decorrelating over a few hundred tokens, so the headroom is reached only by measuring the tokens being scored. A scheduler that predicts the mask ahead of the tokens it prunes remains future work; we do not claim one here. Table 3 isolates the strongest out-of-window scheduler under a protocol shared by all conditions: it recovers a quarter of the oracle’s gain over static without seeing the scored tokens, and it does not match the unpruned model. Finally, the phenomenon is not an artifact of text. A two-layer Mamba trained on a synthetic switched linear system, with two known dynamical regimes, allocates its modes by regime: a linear probe reads the active regime from the mode energies at accuracy 1.00, and cross-regime churn exceeds within-regime churn. On the released language models the same probe reads the input domain from a layer’s mode energies at 99.7% mean accuracy across all layers, against a shuffled-label control near chance; the three domains are well separated, so this shows the regime signature exists and is linearly readable; it does not show that the discrimination is hard.
8
Table 2: Held-out perplexity at half the state budget (r = 8 of 16; Mamba-2 at r = 64 of 128), with the paired scheduled-vs-unpruned gap’s 95% bootstrap CI (10,000 resamples over windows). Input-scheduled selection outperforms every static rule on every model where that rule is run (modal-HSV and LAST are implemented for the per-channel diagonal case, hence absent on Mamba-2), and outperforms the unpruned model with a CI that excludes zero from 130M to the deployed 7B; 370M was audited but not included in this grid. Scheduled is a two-pass oracle (its mask is read from the window being scored), so its columns quantify headroom. Baselines instantiate the published scoring principles on our per-mode outputs with no per-method tuning; scheduled uses none either. The margin over the unpruned model shrinks with scale but stays significant. A native comparison against the released GHOST code on Mamba-2 is reported separately in Table 4. model 130M 790M 1.4B 2.8B Falcon-7B Mamba-2 130M Mamba-2 780M
unpruned
static
modal-HSV
LAST
sched. (oracle)
sched. gap CI
12.38 8.05 7.52 6.82 4.48 12.30 8.07
14.16 8.78 8.25 7.43 4.82 13.58 8.53
14.52 9.29 8.49 7.64 4.96 – –
25.78 12.23 8.62 8.41 5.27 – –
11.84 7.75 7.31 6.66 4.44 11.00 7.33
[−0.66, −0.44] [−0.36, −0.24] [−0.27, −0.17] [−0.21, −0.13] [−0.06, −0.03] – –
Table 3: Out-of-window scheduling at half budget (r = 8), continuation perplexity on tokens ≥ 512 (all conditions share the scoring range). The half-window mask is selected on each window’s first 512 tokens and scored on its second half: a deployable selection rule. It recovers 25% (130M) and 18% (790M) of the two-pass oracle’s gain over static and does not outperform the unpruned model. model
unpruned
static
half-window
two-pass (oracle)
130M 790M
10.37 6.78
12.19 7.49
11.69 7.33
10.14 6.60
Reproducibility. All models are public released checkpoints, run in the reference fp32 path with the fused kernels disabled so the instrument’s reconstruction gate applies; the only trained-by-us model is the disclosed two-layer switched-system Mamba, fully specified with fixed seeds. Windows, calibration/evaluation splits, and all random draws use seed 0. All experiments ran on a single GB10 node; each phase takes minutes to about an hour, and the full campaign is roughly one GPU-day. Code (https://github.com/isrlab/selective-layer-audit) reproduces every table, figure, and validation gate from the released checkpoints.
5
Limitations
The instrument is exact only for the per-channel diagonal case. Mamba-1 and Falcon-Mamba have a distinct diagonal state matrix per channel and no mode coupling after the SSM, so the Gram identity (4) gives the exact error of any mode subset offline. Mamba-2 shares a scalar decay per head and applies a gated normalization that couples modes within a position; there the decomposition holds only at the pre-norm output and the end-to-end pruning error is measured empirically. The depth geometry is architecture-specific. The contiguous mid-network band with a static tail is a Mamba-1 finding; the larger Mamba-1 models, Falcon-Mamba, and Mamba-2 migrate mid-network but in punctuated layers. The phenomenon, the mechanism, and the compression consequence hold across all of them; only the band shape is specific. The input-scheduled result is a measurement of headroom. Two-pass scheduling reads a window’s mode energies from a full first pass, and therefore does not by itself constitute a compute saving. Every inexpensive scheduler we tried falls short: regime-conditioned masks recover 2–6% of the gain, a 256-token prefix estimator 2–9% even with a calibration-learned correction, and half-window selection 18–26%. The important set decorrelates within a few hundred tokens, so the mask must be measured on the scored tokens themselves. The sub-unpruned margin likewise
9
Figure 4: Held-out perplexity versus mode budget r for Mamba-1 130M and 790M. Input-scheduled selection (orange, a two-pass oracle) is below static (blue) at every budget, and at or below the unpruned baseline (dashed) for r ≥ 8; the gap between static and scheduled widens as r decreases (at r = 4, static 19.8 versus scheduled 14.2 on 130M). The mechanism-predicted rules are off scale and omitted: timestep-activity (∆-activity) and random reach perplexity 52.5 and 127 at r = 8 on 130M (5.0 × 103 and 9.1 × 104 at r = 4). Table 4: Native comparison against the released GHOST code on Mamba-2 at 50% state budget, evaluated under the Table 2 protocol (72 held-out windows). GHOST is run as released, with its own default calibration (home) and re-run with calibration drawn from our 96 calibration windows (ours), the same data the static ranking uses. Its perplexity exceeds that of plain realized-energy static selection at both scales under either calibration, and the input-scheduled oracle lies well below it. Home-protocol WikiText-2 costs: +3.2 (130M), +1.16 (780M) word-perplexity. model Mamba-2 130M Mamba-2 780M
unpruned
static
GHOST (home)
GHOST (ours)
sched. (oracle)
12.30 8.07
13.58 8.53
13.98 8.77
13.86 8.71
11.00 7.33
does not survive out-of-window selection. Predicting the mask before the tokens it prunes remains an open problem. The downstream evaluation covers standard tasks up to 1024 tokens; the migration is strongest in the slow, longhorizon modes, so its effect on much longer contexts is untested.
6
Conclusion
A selective SSM layer is a bank of fixed diagonal modes, and the diagonal structure makes state usage exactly measurable: one pass yields the exact output error of pruning any mode subset, at any budget. The measurement shows that trained selective SSMs re-allocate their state across inputs, that the input-dependent write map Bt produces the re-allocation, and that acting on it outperforms every static mode ranking at every scale and, at half the state budget, matches the unpruned model. The scheduler reads its mask from a first pass, so this is realizable headroom; it is not a deployed saving. The phenomenon and mechanism hold across the Mamba-1 family, the deployed 7B Falcon-Mamba, and Mamba-2. The exact instrument replaces the heuristic choice of which states to keep with a direct measurement whose answer varies with the input. The realization-theoretic account of why the migration takes the form it does, and a learned scheduler that exploits it cheaply, are the natural next steps.
References [1] Abdullah Al Asif, Mobina Kashaniyan, Sixing Yu, Juan Pablo Muñoz, and Ali Jannesari. PerfMamba: Performance analysis and pruning of selective state space models. In Benchmarking, Measuring, and Optimizing (Bench), volume 16471 of Lecture Notes in Computer Science. Springer, 2026. 10
[2] Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of Mamba models. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 1516–1534, 2025. [3] Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024. [4] Haruka Ezoe and Kazuhiro Sato. Model compression method for S4 with diagonal state space layers using balanced truncation. arXiv preprint arXiv:2402.15993, 2024. [5] Marco Forgione, Manas Mejari, and Dario Piga. Model order reduction of deep structured state-space models: A system-theoretic approach. arXiv preprint arXiv:2403.14833, 2024. [6] Keith Glover. All optimal Hankel-norm approximations of linear multivariable systems and their l∞ -error bounds. International Journal of Control, 39(6):1115–1193, 1984. [7] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. [8] Minseon Gwak, Seongrok Moon, Joohwan Ko, and PooGyeon Park. Layer-adaptive state pruning for deep state space models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [9] Tamanna Hossain, Robert L. Logan IV, Chandrasekhara Ganesh Jagadeesan, Sameer Singh, Joel R. Tetreault, and Alejandro Jaimes. Characterizing Mamba’s selective memory using auto-encoders. In IJCNLP-AACL, 2025. [10] Michael Menezes and Anastasios Kyrillidis. GHOST: Unmasking phantom states in Mamba2 via grouped hidden-state output-aware selection and truncation. arXiv preprint arXiv:2602.11408, 2026. [11] Bruce C. Moore. Principal component analysis in linear systems: Controllability, observability, and model reduction. IEEE Transactions on Automatic Control, 26(1):17–32, 1981. [12] Henrik Sandberg and Anders Rantzer. Balanced truncation of linear time-varying systems. IEEE Transactions on Automatic Control, 49(2):217–229, 2004. [13] Paul Schwerdtner, Jules Berman, and Benjamin Peherstorfer. Hankel singular value regularization for highly compressible state space models. In Advances in Neural Information Processing Systems (NeurIPS), 2025. [14] Ibne Farabi Shihab, Sanjeda Akter, and Anuj Sharma. Efficient unstructured pruning of Mamba state-space models for resource-constrained environments. In Empirical Methods in Natural Language Processing (EMNLP), 2025. [15] S. Shokoohi, L. M. Silverman, and P. Van Dooren. Linear time-variable systems: Balancing and model reduction. IEEE Transactions on Automatic Control, 28(8):810–822, 1983. [16] Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon Mamba: The first competitive attention-free 7B language model. arXiv preprint arXiv:2410.05355, 2024.
11