C2 R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders
Haoran Jin 1 2 Xiting Wang 3 Shijie Ren 3 Hong Xie 1 2 Defu Lian 1 2
arXiv:2606.30609v1 [cs.LG] 29 Jun 2026
Abstract
SAE Constraints ℓ1 TopK Batch TopK Matryoshka Ort Ours
Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features, but scaling to large dictionaries exposes fundamental challenges. Systematic studies reveal pervasive feature splitting that fragments coherent concepts into non-atomic latents and widespread feature absorption that creates arbitrary exceptions in general features, severely compromising latent reliability. These issues stem from inconsistent latent assignment across samples: without cross-sample constraints, per-sample optimization often allows a single underlying concept to be inconsistently distributed across multiple redundant or interfering latents. To address this, we introduce C2 R (Cross-sample Consistency Regularization). C2 R explicitly encourages that each semantic feature is consistently represented by a unified latent across the batch by penalizing the co-activation of directionally similar latents. Comprehensive evaluation demonstrates that C2 R effectively mitigates both splitting and absorption while, crucially, preserving reconstruction fidelity, providing a principled solution that enhances latent interpretability without degrading model performance. Source code is available* .
Theoretical Guarantee Splitting Absorption × × × × × × × × × × ✓∗ ✓∗
Intuitively Solved Splitting Absorption × × × × × × ✓ ✓ × ✓ ✓ ✓
Reconstruction Preservation ✓ ✓ ✓ × ✓ ✓
Table 1. Comparison of different SAE constraints in terms of theoretical guarantees, intuitive solutions, and reconstruction fidelity. Our cross-sample consistency regularization uniquely offers a theoretical guarantee against splitting and absorption and preserves reconstruction fidelity. ∗ Theoretical guarantee holds under the condition in Eq. 13, which is empirically satisfied in 88.1% of absorption pairs (see Appendix K).
interpretable features (Huben et al., 2023; Bricken et al., 2023). These features provide valuable insights into reasoning, alignment (Zhao et al., 2025; Yeo et al., 2025; Wang et al., 2025), knowledge awareness, hallucinations (Ferrando et al.), and cross-model feature spaces (Lan et al., 2025) of LLMs. This approach is grounded in the hypothesis that the semantic features in a model’s representation are more effectively captured by an overcomplete sparse basis (Olshausen & Field, 1997) than by dense neuron activations, which tend to be polysemantic. Ideally, each latent in an SAE corresponds to a single, human-interpretable concept. While SAEs are effective, they face significant challenges, specifically feature splitting (Bricken et al., 2023; Leask et al.) and feature absorption (Chanin et al., 2024), which undermine the reliability of learned latents. Feature splitting fragments coherent, high-level concepts into overly specific pieces. For example, a single “Mathematics” feature might break down into separate latents for “Algebra,” “Geometry,” and others (Chanin et al., 2024). Although these granular features are interpretable, this fragmentation is problematic because it obscures the true high-level concept the model functionally uses. Moreover, this is inefficient: the dictionary wastes capacity on redundant variations of known concepts instead of finding new ones. Feature absorption, on the other hand, creates “holes” in general features when specific latents capture their activations. A “starts with S” latent, for instance, might fail to activate on “short” or “small” because token-specific latents absorb the signal. This effectively changes the latent to “starts with S (except for short/small),” distorting the intended pattern. Leask et al.
1. Introduction Sparse Autoencoders (SAEs) have emerged as a powerful tool for interpreting large language models (LLMs), breaking down complex internal representations into sparse, 1
University of Science and Technology of China 2 State Key Laboratory of Cognitive Intelligence 3 Gaoling School of Artificial Intelligence, Renmin University of China. Correspondence to: Xiting Wang <[email protected]>, Hong Xie <[email protected]>, Defu Lian <[email protected]>. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s). * https://github.com/hr-jin/ Cross-sample-Consistency-Regularization
1
Cross-sample Consistency Regularization for Sparse Autoencoders
2. Related Work
show that splitting scales with model size, while Chanin et al. (2024) find that absorption affects hundreds of LLM SAEs. These systematic failures undermine the utility of SAEs for critical tasks like causal analysis and circuit discovery.
2.1. Sparse Autoencoders Sparse Autoencoders (SAEs) are grounded in the linear representation hypothesis, which posits that the dense activation space of a language model is constructed from the superposition of sparse, discernible concepts, referred to as features. The goal of an SAE is to recover these groundtruth features by learning a dictionary of latents. Ideally, there exists a one-to-one mapping where each learned latent corresponds precisely to a single meaningful feature.
We argue that these failures arise from a mismatch between the hierarchical nature of language model features and the local scope of standard sparsity constraints. Real-world concepts are inherently hierarchical (Bussmann et al.), yet common objectives like ℓ1 (Bricken et al., 2023) or TopK (Gao et al.) enforce sparsity on a per-sample basis, which potentially penalizes the hierarchical structure. activating both uses more of the sparsity budget than activating the child feature alone. Consequently, the optimizer suppresses the parent latent and forces the child latent to take over its role to keep the active count low. Similarly, regarding feature splitting, the objective does not distinguish between activating a general latent or a specific one. The SAE allows disjoint latents to handle different contexts of a single concept, as nothing ensures consistent latent assignment across samples. Solving these issues, therefore, requires looking beyond per-sample optimization to enforce cross-sample consistency in how latents are selected.
Formally, given an input activation vector x ∈ Rdmodel (e.g., from a Transformer’s residual stream), an SAE projects x into a higher-dimensional sparse latent code f ∈ Rddict , where ddict ≫ dmodel . The encoding process is parameterized by an encoder weight matrix We ∈ Rddict ×dmodel and a bias be : f = ϕ(We x + be ), (1) where ϕ is a non-linear activation function, typically ReLU, TopK, or JumpReLU, designed to induce sparsity. The input is then reconstructed via a linear decoder Wd ∈ Rdmodel ×ddict : x̂ = Wd f + bd . (2)
To address this issue, we propose C2 R (Cross-sample Consistency Regularization). This objective builds on the geometry of the Minkowski inequality (Gruber, 1979) and the strict convexity of the ℓ2 norm. It exploits the fact that the sum of the norms of separate vectors strictly exceeds the norm of their sum: ∥u∥2 + ∥v∥2 > ∥u + v∥2 for nonaligned vectors. By applying this constraint across the batch dimension, C2 R makes it expensive to split a concept into multiple disjoint latents. This formulation penalizes spreading semantic information across redundant latents, driving the SAEs to consolidate activations into a single, consistent latent without supervision.
The training objective minimizes a combination of reconstruction error and a sparsity penalty: LSAE (x) = ∥x − x̂∥22 + λR(f ).
(3)
Common choices for the regularizer R(f ) include the ℓ1 norm (Bricken et al., 2023) or the auxiliary loss associated with TopK constraints (Gao et al.). Various architectural improvements have been proposed to enhance SAE quality. Gated SAEs (Rajamanoharan et al., 2024a) and JumpReLU SAEs (Rajamanoharan et al., 2024b) introduce learnable thresholds to improve the fidelity-sparsity frontier. However, these methods focus on the per-sample activation sparsity rather than enforcing the hierarchical structure of the SAE.
Our contributions are threefold: • Theoretical diagnosis: We identify the lack of crosssample consistency in per-sample sparsity objectives as the root cause of feature splitting and absorption, providing a unified formal analysis of these phenomena.
2.2. Structural SAEs More recently, approaches attempting to structure the latent space have emerged. Batch TopK SAEs (Leask et al.) relax the rigid per-sample TopK constraint to a batch-level aggregate, allowing for variable sparsity across samples. While this improves reconstruction, it lacks any mechanism to enforce hierarchical structure among the latents and still faces feature absorption and splitting challenges.
• Principled objective: We propose C2 R, a novel regularization objective that utilizes decoder geometry and batch-level statistics to enforce consistent latent selection, effectively distinguishing between true polysemanticity and harmful redundancy. • Empirical validation: We demonstrate that C2 R significantly mitigates splitting and absorption, achieving better feature hierarchy without compromising reconstruction fidelity compared to state-of-the-art baselines.
Matryoshka SAEs (Bussmann et al.) enforce a nested structure where subsets of latents are trained to approximate the input at different sparsity levels. While this creates a hierarchy, it compromises reconstruction fidelity and lacks a clear theoretical explanation for why it would fix the optimization issues of standard sparsity objectives. OrtSAE (Korznikov 2
Cross-sample Consistency Regularization for Sparse Autoencoders (i)
et al., 2025) addresses feature splitting and composition by penalizing the cosine similarity between decoder weights. However, this approach tries to handle absorption indirectly via the decoder geometry, rather than addressing the encoder activation patterns where absorption is formally defined. In contrast, our method targets the latent activations directly to penalize redundancy, offering a theoretically guaranteed solution derived from the formal definitions of splitting and absorption.
Here, z1 corresponds to the feature-aligned activation (i) along L1 , while z2 encodes the orthogonal residual components required for accurate reconstruction. Samples 1, . . . , m exclusively activate L1 , whereas samples m + 1, . . . , m + n possess a non-zero component from L2 . In practice, sparse autoencoders often learn perturbed latent directions, denoted L′1 and L′2 , that deviate from the ideal basis. We assume L′2 contains a fraction α ∈ [0, 1] of the feature direction L1 , forming a new latent that partially overlaps with it:
2.3. Minkowski Inequality (Gruber, 1979) L′1 = L1 ,
For any real number p ≥ 1 and any two real sequences a = (a1 , a2 , . . . , an ) and b = (b1 , b2 , . . . , bn ), their pnorms satisfy ∥a + b∥p ≤ ∥a∥p + ∥b∥p . (4)
L′2 =
The corresponding activation pattern becomes:
3. Unified Problem Formulation In this section, we propose a geometric framework that unifies feature splitting and absorption. Rather than viewing them as separate pathologies, we model them as instances of latent redundancy arising from perturbed basis directions. We introduce a redundancy parameter α to quantify the extent to which a semantic feature “leaks” into varying latents, allowing us to derive a single consistency condition that prevents both failure modes.
Sample 1 .. .
L′1 (1) z1 .. .
m
z1
m+1 .. .
(m+1) (1 − α)z1
m+n
(1 − α)z1
(m)
.. . (m+n)
(i)
(m+1)
(m+n)
z1
.. . q (m+n) 2 (m+n) 2 (αz1 ) + (z2 )
(i)
4. Theoretical Analysis on Two SAE Latents 4.1. Limitation of Per-sample Sparsity Objectives
L2 0 .. . 0
We analyze the behavior of ℓ1 and TopK objectives under the activation patterns defined in the unified problem formulation. Lemma 4.1. Per-sample sparsity constraints, specifically ℓ1 regularization and TopK, strictly favor feature splitting and absorption (α → 1) over the ideal orthogonal decomposition (α = 0) given equivalent reconstruction fidelity.
(m+1)
z2
.. .
(m+1) 2 (m+1) 2 (αz1 ) + (z2 )
in Eq. 6. The (αz1 )2 + (z2 )2 term ensures the total reconstructed model activation remains the same through vector addition of L′1 and L′2 . This formalization unifies feature splitting and absorption: splitting and full absorption correspond to α=1, where feature-aligned energy is duplicated across latents, and partial absorption corresponds to α<1, where the L′2 inherits part of the feature along with its orthogonal component.
Let z1 and z2 be the corresponding activations of L1 and L2 for sample i. In the ideal case (no splitting nor absorption, α = 0), all information related to F is represented solely by L1 , and L2 contributes only to the orthogonal reconstruction components. The activation pattern is:
z1
0 q
This activation pattern illustrates that for samples m + 1 through m + n, part of the original feature direction L1 is reconstructedqvia L′2 due to the shared component αL1 (i)
L1 (1) z1 .. . (m) z1
L′2 0 .. .
Table 2. Activation pattern when feature splitting or absorption occurs.
Let L1 denote the ideal latent direction that captures the complete semantic feature F , and let L2 denote an orthogonal direction that captures residual, non-feature components. Both L1 and L2 are unit vectors and orthogonal to each other: ∥L1 ∥ = ∥L2 ∥ = 1, L1 ⊥ L2 . (5)
Sample 1 .. . m m+1 .. . m+n
(6)
Here α quantifies the degree of cross-latent feature sharing. When α = 0, L′2 is perfectly orthogonal to L′1 (no dispersion). When α = 1, L′2 fully aligns with L′1 , corresponding to a complete feature split into two disjoint latents.
When the activations of a single semantic feature are distributed across multiple redundant SAE latents, the combined p-norm of their activations exceeds that of a single latent capturing the same feature. This inequality motivates our regularization term that penalizes redundant feature allocation across latents, thereby constraining feature splitting and absorption.
(i)
(1 − α)L1 + αL2 . ∥(1 − α)L1 + αL2 ∥
.. . (m+n)
z2
3
Cross-sample Consistency Regularization for Sparse Autoencoders
Feature Absorption 𝐿! : Tokens starting with ‘S’ or ‘s’
Latent #
0 𝑏" … …
Decoder Vector
…
Sample
𝐿’$ : Base64 ASCII
… 𝑎! 0 0 … 0 𝑏" 0 … 0 0 𝑐% … … … … 𝐿’!
Base64 letters Base64 digits Base64 ASCII
𝐿’%
𝐿’"
𝑎!
𝑏!
𝑐!
𝑎"
𝑏"
𝑐"
.. .
.. .
.. .
#
#
! 𝑎!"
! 𝑏!"
!$%
!$%
𝐿’# : Token “short”
𝐿’!
Decoder Vector
Latent! Latent " Latent %
𝐿! : Base64 texts 𝐿’# : Base64 digits
𝐿! : Tokens starting with ‘S’ or ‘s’ 𝐿’! : Tokens starting with ‘S’ or ‘s’, except “short”
Other Tokens starting with ‘S’ or ‘s’
𝑤!( 𝑤# & 𝜌!,# ∗ (!) = 𝑚𝑎𝑥#'! ReLU( )& ||𝑤! ||& ||𝑤# ||&
Feature Splitting
𝐿’! : Base64 letters
Prevent Absorption
…
… … …
𝐿’#
𝑎! 0
Other Tokens starting with ‘S’ or ‘s’
…
Latent "
Latent #
𝐿’# : Token “short”
𝐿’!
Token “short”
Latent!
…
𝐿’! : Tokens starting with ‘S’ or ‘s’, except “short”
Latent! Latent "
Token “short”
…
… … … …
𝐿’#
𝑎! 0 𝑎" 0 … …
… … … …
Merge The Split Features 𝐿! : Base64 texts 𝐿’! : Base64 letters
𝐿’# : Base64 digits
Base64 letters Base64 digits
…… …
Base64 ASCII
𝐿’$ : Base64 ASCII
𝐿’! 𝐿’" 𝐿’% … 𝑎! 0 0 … 𝑎" 0 0 … 𝑎% 0 0 … … … …
𝐂 𝟐 𝐑 𝐋𝐨𝐬𝐬
Figure 1. Overview of C2 R (Cross-sample Consistency Regularization). Each mini-batch contains activations of multiple samples encoded by a sparse autoencoder. C2 R enforces consistency of latent usage across samples by constraining activation patterns along the batch dimension. This encourages each latent to represent a complete semantic feature rather than fragmented or absorbed subfeatures, mitigating feature splitting and absorption.
Proof. Case 1: The ℓ1 Penalty. We compare the cumulative ℓ1 penalty for the ideal configuration versus the perturbed configuration, assuming equivalent SAE reconstruction.
driving the solution toward splitting or absorption to reduce the total norm while maintaining reconstruction fidelity. Case 2: The TopK Constraint. The TopK objective enforces a hard constraint on the number of active latents, effectively minimizing the ℓ0 norm of the activation vector for a fixed reconstruction error tolerance. We evaluate the cardinality of the active set for the intersection samples i ∈ {m + 1, . . . , m + n}.
(1) Ideal Decomposition (α = 0): In this state, the feature L1 and the residual L2 are orthogonal. The total ℓ1 norm over the batch is: ℓ1 (α = 0) =
m+n X
m+n X
(i)
z1 +
i=1
(i)
z2 .
(7)
(1) Ideal Decomposition (α = 0): The signal consists of two orthogonal non-zero components: the feature activation (i) (i) z1 and the residual z2 . Since the basis vectors L1 and L2 are orthogonal, exact representation requires both to be active. Thus, the sparsity consumption is:
i=m+1
(2) Perturbed Decomposition (α > 0): Under splitting or absorption, the activation energy is redistributed. The total ℓ1 norm becomes: m X
ℓ1 (α > 0) =
(i)
z1 +
i=1 m+n X
+
m+n X
(i)
(i)
(αz1 )2 + (z2 )2 .
(i)
i=m+1
m+n X i=m+1
(i) z2 >
m+n X
(i)
(z1 )2 + (z2 )2 . As a result, the SAE represents the same vector space using a single active latent:
We apply the Minkowski inequality to the terms summing over indices i ∈ {m + 1, . . . , m + n}. For any α > 0: (i) αz1 +
(10)
(2) Full Splitting or Absorption (α = 1): Substituting α = 1 into the activation pattern defined in Table 2, the (i) coefficient for the first latent becomes zero: (1 − α)z1 = 0. ′ The q second latent, L2 , captures the entire vector magnitude
(8)
i=m+1
m+n X
for i ∈ {m + 1, . . . , m + n}.
(i)
(1 − α)z1
i=m+1
q
∥z (i) ∥0 = 2
∥z (i) ∥0 = 1
q (i) (i) (αz1 )2 + (z2 )2 . (9)
for i ∈ {m + 1, . . . , m + n}.
(11)
i=m+1
By reducing the active set from 2 latents to 1, the split configuration (α = 1) saves the sparsity budget. This creates a strong pressure on the optimization: the TopK constraint pushes the SAEs to use latents to represent mixed feature directions rather than maintaining atomic features, as this
Subtracting the shared terms from Eq. 7 and Eq. 8, it follows that ℓ1 (α > 0) < ℓ1 (α = 0). The strict inequality holds for all α ∈ (0, 1]. Consequently, the optimization process minimizes the global objective by maximizing α, thereby 4
Cross-sample Consistency Regularization for Sparse Autoencoders
saves capacity within the k-latent budget to reconstruct other features and lower the global reconstruction loss.
We empirically verify the condition in Eq. 13 on our trained 65,536-latent SAEs using a 4M-token test set from SAEBench. As shown in Appendix K, the condition is satisfied in 88.1% of absorption pairs (N =4,555), and the median ratio of the left-hand side to the right-hand side of Eq. 14 is 81.56. The remaining 11.9% of violated pairs correspond to marginal cases with low absorption coefficients, where neither the parent nor the child feature carries strong signal. This confirms that the theoretical guarantee holds in the vast majority of practically relevant cases, though we note its conditional nature.
4.2. Mitigating Splitting via Cross-Sample Consistency Previous analysis shows that per-sample objectives (ℓ1 and TopK) cannot distinguish between atomic and split features. In contrast, the Minkowski inequality (Gruber, 1979) (Eq.4) for p = 2 offers a strict convexity condition to reverse this preference. Since the sum of norms for separate vectors is always greater than the norm of their sum, minimizing the sum of ℓ2 norms across the batch dimension, i.e. ∥Z:,1 ∥2 + ∥Z:,2 ∥2 , encourages SAEs to consolidate shared semantic feature into a single latent.
5. Cross-Sample Consistency Regularization Following the analysis in Section 4, we introduce C2 R (Cross-sample Consistency Regularization). Minkowski inequality helps merge redundant features, but applying it to a large dictionary requires a careful approach. Here, we extend the pairwise analysis to the full SAE dictionary and examine the gradient dynamics that enable C2 R to enforce both cross-sample consistency and latent orthogonality.
We analyze this mechanism using the unified formulation in Table 2. We define a regularization term Lpair as the sum of the batch-norms for the two split latents: Lpair (α) = ∥Z:,1 ∥2 + ∥Z:,2 ∥2 v um m+n X uX (i) (i) = t (z )2 + ((1 − α)z )2 1
1
i=1
(12)
i=m+1
v u m+n m+n X u X (i) (i) +t (αz )2 + (z )2 . 1
5.1. Generalizing from Pairwise to Multiple Latents
2
i=m+1
i=m+1
The derivation in Section 4 used a simple system with two latents for one parent feature and a child feature. However, standard SAEs have thousands of latents, and most represent distinct concepts. If we minimize the sum of norms across random pairs without selection, we would force independent features to merge, which causes feature collapse and reduces the SAEs’ ability to resolve semantic differences.
To check if minimizing this term suppresses splitting, we calculate its gradient with respect to the splitting factor α. ∂L The derivative ∂αpair shows that the loss increases monotonically with α (meaning the penalty reduces splitting) as long as the following condition holds (derivation in Appendix A): α≥ r P
1
(i) 2 m i=1 (z1 ) Pm+n (i) 2 i=m+1 (z2 )
.
(13)
We therefore need the regularization to be selective. It should only penalize latent pairs that look like split fragments or absorbed variations, while leaving independent features alone. Previous work on feature splitting (Chanin et al., 2024) and orthogonal constraints (Korznikov et al., 2025) shows that redundant latents usually have similar decoder weight directions. In contrast, distinct features tend to be nearly orthogonal in the high-dimensional space.
+1
Empirical results from recent literature support this condition for practical SAE training. Leask et al. find that split latents retain high cosine similarity with their parent latents, implying the feature component magnitude far ex(i) (i) ceeds the residual (∥z1 ∥ ≫ ∥z2 ∥). Additionally, Chanin et al. (2024) note large frequency gaps between parent and child features (e.g., P (f0 ) = 0.25 vs P (f1 ) = 0.05), which means the cumulative energy of the primary feature dominates the residual: m X
(i) (z1 )2 ≫
i=1
m+n X
(i) (z2 )2 . i=m+1
Based on this, we use the cosine similarity of decoder weights to detect redundancy. For each latent i, we find its nearest neighbor j ∗ (i) in the decoder space: j ∗ (i) = arg max ⟨ŵi , ŵj ⟩,
where ŵ =
j̸=i
(14)
w . ∥w∥2
(15)
We then define the C2 R loss by weighting the pairwise norm penalty with the squared rectified cosine similarity ρ2i,j ∗ (i) = ReLU(⟨ŵi , ŵj ∗ (i) ⟩)2 :
Under these settings, the right-hand side of Eq. 13 approaches zero. As a result, Lpair increases monotonically with α in the relevant domain. This confirms that a crosssample ℓ2 penalty on redundant pairs theoretically ensures the consolidation of semantically related activations into a single consistent latent, mitigating both splitting and absorption.
LC2 R (X) =
k 1X 2 ρi,j ∗ (i) · ∥Z:,i ∥2 + ∥Z:,j ∗ (i) ∥2 . (16) k i=1 | {z } Si,j ∗ (i)
This weight acts as a gate. For distinct features where ŵi ⊥ 5
Cross-sample Consistency Regularization for Sparse Autoencoders
KL Divergence Score
1.000 0.996
TopK Batch TopK Ort Matryoshka C 2R
0.992 60
80
0.930
0.960
0.956
0.900
TopK Batch TopK Ort Matryoshka C 2R
0.885
0.952 100
L0 (Sparsity)
120
(a) Loss Recovered
60
80
100
120
L0 (Sparsity)
60
(b) KL Divergence Score
TopK Batch TopK Ort Matryoshka C 2R
0.705
0.915
Autointerp Score
0.964
1.004
Loss Recovered
TopK Batch TopK Ort Matryoshka C 2R
Disentanglement Score
1.008
80
100
L0 (Sparsity)
(c) Autointerp
120
0.690
0.675
0.660
60
80
100
L0 (Sparsity)
120
(d) Disentanglement Score
Figure 2. Quantitative comparison of SAE performance across different sparsity levels. (a) and (b) evaluate reconstruction fidelity using Cross-Entropy Loss and KL Divergence at the LLM output layer, respectively, (c) assesses SAE latent interpretability using the Autointerp score, and (d) measures the extent to which real-world features are disentangled into steerable latents. Notably, C2 R-enhanced Batch TopK SAE does not exhibit performance degradation compared to the vanilla Batch TopK baseline. Instead, it maintains competitive or superior results, particularly in KL Divergence and Autointerp scores, demonstrating that our proposed C2 R constraint preserves these important capabilities.
ŵj , ρ2 is zero, so the consistency constraint does not apply. For redundant features with high alignment, ρ2 is large, which fully applies the regularization.
pairs, our method scales the penalty by Si,j ∗ (i) (the sum of feature activations). This allows C2 R to adjust regularization based on feature frequency and magnitude. Strong, high-frequency features (large S) are subject to stricter orthogonality pressure to prevent redundancy. Conversely, for newly initialized or rare latents (small S), aggressive orthogonality enforcement can be counterproductive, potentially pushing them away from valid directions before they stabilize. By scaling the gradient with activation magnitude, C2 R prevents such disruption, allowing developing latents to converge naturally. This mechanism promotes orthogonality while adapting the regularization strength to each feature’s convergence state. We further clarify the implementationlevel relationship between C2 R and OrtSAE in Appendix L.
The final training objective adds this regularization to the standard SAE loss, which includes reconstruction and sparsity terms: L(X) = LSAE (X) + λC2 R LC2 R (X),
(17)
where λC2 R controls the weight of the cross-sample consistency term. 5.2. Gradient Analysis and Implicit Orthogonality We can better understand C2 R by looking at its gradient dynamics. The loss function is the product of a geometric alignment term (ρ2 ) and an activation magnitude term (Si,j ∗ (i) ). The gradient with respect to the model parameters θ follows the product rule ∇(AB) = B∇A + A∇B: ∇θ LC2 R ∝ ρ2 · ∇θ Si,j ∗ (i) + Si,j ∗ (i) · ∇θ (ρ2 ) . | {z } | {z } Consistency Gradient
6. EXPERIMENTS 6.1. Baselines Baselines include four different SAE architectures: TopK SAEs (Gao et al.), Batch TopK SAEs (Leask et al.), Matryoshka SAEs (Bussmann et al.), and OrtSAEs (Korznikov et al., 2025), as introduced in 2. For the main results in Figures 2 and 3, C2 R uses Batch TopK as its base architecture to ensure a fair comparison with OrtSAE and Matryoshka SAE, both of which also build on Batch TopK. We evaluate performance by iterating over sparsity levels of k ∈ {60, 80, 100, 120}, ensuring a fair comparison between baselines and C2 R-enhanced Batch TopK SAE under equivalent sparsity conditions.. We run baselines and implement C2 R on top of a public codebase (Karvonen, 2024), and the trained SAEs are evaluated with SAEBench (Karvonen et al.). Matryoshka SAEs use 5 layers, with the sizes of its 4 sub-SAEs set to {1/32, 1/8, 1/4, 1/2} of the total latents.
(18)
Orthogonality Gradient
This shows that C2 R creates two simultaneous forces during optimization. 1. Consistency Pressure (ρ2 ∇S). The first term minimizes the sum of norms, scaled by the similarity weight ρ2 . As shown in Section 4, this uses the Minkowski inequality to push the splitting factor α toward 0. It merges the activation energy of redundant latents into a single one, which helps fix feature splitting and absorption. 2. Implicit Orthogonality Pressure (S∇ρ2 ). The second term minimizes cosine similarity between decoder weights to encourage feature orthogonality, similar to the goals of OrtSAE (Korznikov et al., 2025). Unlike OrtSAE that typically applies a uniform penalty to all selected max-cosine 6
Cross-sample Consistency Regularization for Sparse Autoencoders
0.2 0.1 0.0
1.50 1.35 1.20
80
100
L0 (Sparsity)
120
(a) Feature absorption
0.30
TopK Batch TopK Ort Matryoshka C 2R
0.24 0.18 0.12
1.05
60
0.36
Cosine Similarity
Mean Absorption Fraction
0.3
TopK Batch TopK Ort Matryoshka C 2R
1.65
Split Feature Number
TopK Batch TopK Ort Matryoshka C 2R
0.4
60
80
100
L0 (Sparsity)
120
(b) Feature splitting
60
80
100
L0 (Sparsity)
120
(c) Feature composition
Figure 3. Analysis of feature structural metrics. We compare feature absorption, splitting, and composition across different sparsity levels. Among methods that maintain high reconstruction fidelity (i.e., excluding Matryoshka SAEs), the proposed C2 R constraint achieves the lowest rates of feature absorption and splitting. Furthermore, it achieves the optimal performance in feature decomposition, consistently yielding the lowest cosine similarity to ensure more atomic features.
6.2. Experiment Settings
6.3. Metrics
We conduct systematic experiments on Gemma-2-2B (Team, 2024)† to validate the effectiveness of C2 R. Specifically, we sample a 500M-token subset from the OpenWebText dataset (Gokaslan et al., 2019)‡ and use it to train a series of SAEs on the residual stream activations of the 12th layer of Gemma-2-2B. Each SAE has 65,536 latents, which is approximately 28 times the model’s residual dimension, making it easier to observe feature absorption and feature splitting (Karvonen et al.). We performed a hyperparameter sweep for λC2 R over the set {0.1, 0.5, 1, 5, 10}. We selected λC2 R = 5, as it represents the maximal regularization strength that does not degrade reconstruction fidelity. All SAEs are optimized with Adam using a learning rate of 2 × 10−4 , batch size of 2,048, and context length of 1,024.
To holistically assess the relative performance of SAEs when integrating C2 R, our evaluation utilizes a comprehensive set of seven key metrics. These metrics, implemented using the code framework from SAEBench (Karvonen et al.), cover four areas: reconstruction fidelity, feature hierarchy, interpretability, and disentanglement. Specifically, our evaluation consists of six key metrics: Loss Recovered, KL Div. Score, AutoInterp, Split Num, Absorption Rate, Composition, and Disentanglement. Detailed description of these metrics is in E. 6.4. Reconstruction Fidelity As shown in Figure 2, integrating C2 R preserves reconstruction fidelity of the backbone SAEs. Figure 2a reports the Loss Recovery metric (details in Appendix E), where a value ≥ 1 indicates that the SAE-reconstructed LLM activations yield a cross-entropy loss lower than or equal to the original LLM. We observe that across the tested sparsity levels, only Matryoshka SAEs exhibit performance loss, while other methods maintain full recovery. Figure 2b displays the KL Divergence Score, which measures the shift in the LLM’s output logit distribution. Higher scores correspond to smaller deviations from the original distribution. The results demonstrate that SAEs trained with the C2 R constraint maintain high reconstruction fidelity, strictly adhering to the original model’s behavior.
Computational Efficiency. Computing the pairwise cosine similarity for a large dictionary (e.g., k = 65, 536) imposes a quadratic O(k 2 ) computational complexity and substantial memory overhead. To ensure computational feasibility and maintain a fair comparison with the state-ofthe-art baseline, we adopt the efficient optimization strategy used in OrtSAEs (Korznikov et al., 2025). Specifically, we employ a block-wise computation strategy with a chunk size of 8,192 and compute the consistency regularization term every 5 training steps, scaling the coefficient λC2 R accordingly. This approach reduces the overhead to negligible levels while preserving performance (detailed in Appendix B). † ‡
Gemma Terms of Use Creative Commons Zero v1.0 Universal
6.5. Interpretability Following the evaluation framework of (Paulo et al.), we assess the interpretability of all SAEs using the AutoInterp metric. The results in Figure 2c show that adding C2 R does 7
Cross-sample Consistency Regularization for Sparse Autoencoders
6.8. Feature Composition
not significantly affect interpretability. For each SAE, we sampled 128 latents and constructed prompts based on their activations over a 2M-token input. We prompt an LLM explainer to generate concise and comprehensive textual descriptions for each latent from 15 observed samples, and prompt an independent LLM judge predicted whether each latent would activate on 15 unseen test samples. The detailed prompts used for both the LLM explainer and LLM judger, as well as the instructions provided to human annotators, are included in Appendix C.
Following Bussmann et al., we quantify feature composition by measuring the average maximum cosine similarity between latent vectors. High similarity implies that multiple latents represent overlapping information, indicating a lack of atomicity. As shown in Figure 3c, our approach achieves the optimal performance on this metric. The C2 R constraint yields significantly lower cosine similarity compared to vanilla Batch TopK and Matryoshka SAEs, and marginally outperforms OrtSAE. This result confirms that our method effectively minimizes redundancy, producing a set of highly distinct and atomic features.
We employ GPT-5-mini (OpenAI, 2025) as the LLM judge and validate its reliability via a user study. GPT-5-mini achieved a 97.3% match rate with humans and a Pearson r of 0.74, confirming its suitability as a proxy evaluator for automated interpretability assessment. Details are in Appendix D. To further verify that the results are not sensitive to the choice of judge model, we re-evaluate all AutoInterp scores using GPT-5. The per-method Pearson correlations between GPT-5-mini and GPT-5 scores range from 0.81 to 0.94 (all p < 0.001), confirming that the interpretability rankings are consistent across judge models. Full results are in Table 7.
6.9. Robustness and Generalization We conduct a series of additional experiments to verify the robustness and generalization of C2 R across random seeds, model scales, layers, and training data. Detailed results are provided in Appendix G. Statistical validation. To assess the stability of our main results, we perform 5 independent training runs with different random seeds at sparsity level k=100 on Gemma-2-2B layer 12. As shown in Table 8, C2 R consistently achieves the best feature composition and absorption with narrow confidence intervals, confirming that the reported improvements are statistically robust.
6.6. Disentanglement We evaluate disentanglement using the RAVEL benchmark (Huang et al., 2024), which employs interchange interventions to determine if specific SAE latents causally control individual attributes such as continent or gender. The metric averages cause and isolation scores to measure how well the SAE governs a target concept without affecting unrelated ones. As illustrated in Figure 2d, Batch TopK SAEs trained with the C2 R constraint achieve performance comparable to the vanilla baseline. This demonstrates that our regularization effectively preserves a disentangled and manipulable latent space.
Scaling to larger models. We evaluate C2 R on two current-generation models, Qwen3-8B and Llama-3-8B, both at layer 20 with sparsity level k=100. Tables 9 and 10 show that C2 R maintains its effectiveness at the 8B scale: it achieves the lowest composition and absorption while preserving reconstruction fidelity, demonstrating that our method generalizes beyond the Gemma-2-2B setting. Cross-layer consistency. Beyond the layer 12 experiments in the main evaluation, we additionally train SAEs on Gemma-2-2B layer 20 (Table 11). C2 R continues to achieve optimal or near-optimal structural metrics at a deeper layer, confirming cross-layer consistency.
6.7. Feature Splitting and Feature Absorption Figures 3a and 3b illustrate the evaluation of feature absorption and splitting. Our proposed C2 R constraint effectively mitigates both issues compared to the vanilla Batch TopK baseline and outperforms the orthogonal constraint in OrtSAE. Although Matryoshka SAEs achieve lower absorption and splitting scores, this improvement comes at the expense of reconstruction fidelity, as evidenced in Figure 2a and 2b. Therefore, among methods that maintain the original model’s performance, the C2 R constraint strikes the best balance, achieving the lowest levels of absorption and splitting while preserving latent quality and reconstruction capabilities.
Sensitivity to training data. We investigate the effect of training data scale and composition by (1) extending the training corpus from 500M to 1B tokens on OpenWebText, and (2) training on The Pile (Gao et al., 2020), a structurally diverse dataset spanning academic, internet, prose, and dialogue domains. As shown in Tables 12 and 13, C2 R consistently achieves optimal or near-optimal structural feature metrics in both settings, confirming that our method is not sensitive to the scale or composition of the training data. 6.10. Ablation Study We ablate the key design choices of C2 R and study the sensitivity to the regularization strength λC2 R . All experiments 8
Cross-sample Consistency Regularization for Sparse Autoencoders
are conducted on Gemma-2-2B layer 12 at sparsity level k=100. Detailed results are provided in Appendix H.
pathological negative correlations, ensuring training stability in signed encoder settings.
Component ablation. We evaluate two variants: (1) NoNNR, which removes the nearest-neighbor restriction and applies the consistency penalty to all latent pairs; and (2) NoRCG, which removes the ReLU cosine gate and applies the penalty without directional selectivity. As shown in Table 14, removing either component imposes an overly aggressive merging constraint. While this severely penalizes feature absorption, it does so at the unacceptable cost of significantly degrading reconstruction fidelity and inflating composition scores. This demonstrates that the cautious, gated constraints of C2 R are necessary to mitigate feature pathologies without destroying dictionary utility.
7.1. Downstream Causal Intervention Tasks We further evaluate the practical utility of learned SAE features on two downstream causal intervention tasks from SAEBench. Spurious Correlation Removal (SCR) measures the ability to remove spurious correlations by ablating relevant SAE features, and Targeted Probe Perturbation (TPP) measures the precision of causal interventions via targeted feature perturbation. As shown in Table 17 in Appendix J, C2 R outperforms TopK, Batch TopK, and Ort on both tasks and achieves comparable performance to Matryoshka, indicating that the improved feature structure transfers to downstream causal applications.
Hyperparameter sensitivity. We sweep λC2 R ∈ {0.1, 0.5, 1, 5, 10} to understand its effect on the trade-off between reconstruction and structural metrics (Table 15). As λC2 R increases, composition and absorption decrease monotonically, while reconstruction fidelity remains stable up to λC2 R =5 and begins to degrade at λC2 R =10. We select λC2 R =5 as the default, as it maximizes the reduction in feature composition and absorption without sacrificing reconstruction quality.
8. Conclusion In this work, we introduced C2 R, a theoretically grounded constraint aimed at improving feature hierarchy in sparse autoencoders. By leveraging the Minkowski inequality, our approach provides a rigorous guarantee against feature splitting and absorption. Extensive empirical evaluations show that incorporating the C2 R constraint significantly enhances the structural quality of the learned dictionary, reducing feature composition and minimizing redundancy. Importantly, our approach preserves reconstruction fidelity, latent interpretability, and disentanglement capabilities on par with the baselines. These results highlight the effectiveness of the cross-sample consistency regularization in addressing the trade-off between feature atomicity and model fidelity, offering a robust solution for large language model analysis.
7. Compatibility with Other SAE Architectures A key design goal of C2 R is to serve as a general-purpose regularizer that can be applied on top of any SAE architecture. To verify this, we evaluate C2 R when integrated with different backbone architectures beyond the Batch TopK used in our main experiments. All experiments are conducted on Gemma-2-2B layer 12 at sparsity level k=100. Detailed results are provided in Appendix I.
9. Limitations
2
Integration with TopK and OrtSAE. We apply C R to both the standard TopK and OrtSAE backbones. As shown in Table 16, C2 R consistently and substantially improves feature structural metrics regardless of the base architecture, without degrading reconstruction fidelity. Notably, TopK + C2 R achieves a significant reduction in Composition and Absorption, demonstrating that C2 R’s benefits are not tied to the Batch TopK backbone but arise from the cross-sample consistency mechanism itself.
The theoretical guarantee in Eq. 13 is conditional and holds in 88.1% of absorption pairs in our verification (Appendix K). A fully unconditional guarantee remains an open problem. Although we extend our evaluation to Qwen38B and Llama-3-8B, we have not verified scalability to larger models or diverse architectures such as Mixture-ofExperts. Our AutoInterp evaluation uses the proprietary GPT-5-mini as the judge model. We verify high consistency with GPT-5 and human evaluations, but this still introduces a reproducibility constraint. In addition, C2 R may suppress legitimate polysemanticity when distinct features have moderately aligned decoder directions. We find that C2 R maintains over 99% dictionary utilization (Appendix N), but the interaction with polysemantic features needs further study.
Integration with AbsTopK. To evaluate whether C2 R adapts to bi-directional encoder architectures, we apply it on top of AbsTopK, which uses signed activations. As shown in Table 16, C2 R effectively improves feature hierarchy in this signed setting while preserving reconstruction fidelity. The ReLU cosine gate remains appropriate because feature absorption and splitting imply cosine similarity > 0 between the involved decoder directions. By targeting only positive similarities, C2 R avoids collateral damage to non9
Cross-sample Consistency Regularization for Sparse Autoencoders
Acknowledgements
Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations.
This work was supported by the National Natural Science Foundation of China (NSFC) (No. U24A20253, NO. 62476279, NO. 92470205, NO. U2436209), Scientific Research Innovation Capability Support Project for Young Faculty, Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, Renmin University of China, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China No. 24XNKJ18. Supported by fund for building world-class universities (disciplines) of Renmin University of China and Public Computing Cloud, Renmin University of China.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. Gokaslan, A., Cohen, V., Pavlick, E., and Tellex, S. Openwebtext corpus. http://Skylion007.github. io/OpenWebTextCorpus, 2019. Gruber, P. M. Geometry of numbers, pp. 186–225. Birkhäuser Basel, Basel, 1979. ISBN 978-3-0348-5765-9. doi: 10.1007/978-3-0348-5765-9_7. URL https:// doi.org/10.1007/978-3-0348-5765-9_7.
Impact Statement This work advances the methodology of sparse dictionary learning for neural networks. By improving the fidelity of feature extraction, we aim to enable a more granular understanding of large language model internals. Such interpretability is essential for auditing model behavior, identifying latent failure modes, and verifying safety properties prior to deployment. However, we acknowledge that deeper insights into model representations can also be leveraged to improve model efficiency or steerability, potentially accelerating the development of powerful systems and amplifying the societal risks associated with their deployment.
Huang, J., Wu, Z., Potts, C., Geva, M., and Geiger, A. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8669–8687, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long. 470. URL https://aclanthology.org/2024. acl-long.470/.
References
Huben, R., Cunningham, H., Smith, L. R., Ewart, A., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2023.
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformercircuits.pub/2023/monosemantic-features/index.html.
Karvonen, A. dictionary_learning_demo. https: //github.com/adamkarvonen/dictionary_ learning_demo, 2024.
Bussmann, B., Nabeshima, N., Karvonen, A., and Nanda, N. Learning multi-level features with matryoshka sparse autoencoders. In Forty-second International Conference on Machine Learning.
Karvonen, A., Rager, C., Lin, J., Tigges, C., Bloom, J. I., Chanin, D., Lau, Y.-T., Farrell, E., McDougall, C. S., Ayonrinde, K., et al. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability. In Forty-second International Conference on Machine Learning.
Chanin, D., Wilken-Smith, J., Dulka, T., Bhatnagar, H., Golechha, S., and Bloom, J. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409.14507, 2024.
Korznikov, A., Galichin, A., Dontsov, A., Rogov, O., Tutubalina, E., and Oseledets, I. Ortsae: Orthogonal sparse autoencoders uncover atomic features, 2025. URL https://arxiv.org/abs/2509.22033.
Ferrando, J., Obeso, O. B., Rajamanoharan, S., and Nanda, N. Do i know this entity? knowledge awareness and hallucinations in language models. In The Thirteenth International Conference on Learning Representations.
Lan, M., Torr, P., Meek, A., Khakzar, A., Krueger, D., and Barez, F. Quantifying feature space universality across large language models via sparse autoencoders, 2025. URL https://arxiv.org/abs/2410.06981. 10
Cross-sample Consistency Regularization for Sparse Autoencoders
Leask, P., Bussmann, B., Pearce, M. T., Bloom, J. I., Tigges, C., Al Moubayed, N., Sharkey, L., and Nanda, N. Sparse autoencoders do not find canonical units of analysis. In The Thirteenth International Conference on Learning Representations. Olshausen, B. A. and Field, D. J. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311–3325, 1997. OpenAI. Gpt-5 system card, 2025. URL https:// openai.com/index/gpt-5-system-card/. Paulo, G. S., Mallen, A. T., Juang, C., and Belrose, N. Automatically interpreting millions of features in large language models. In Forty-second International Conference on Machine Learning. Pearson, K. Vii. note on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352):240–242, 1895. Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., and Nanda, N. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404.16014, 2024a. Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024b. Team, G. Gemma. 2024. doi: 10.34740/KAGGLE/M/3301. URL https://www.kaggle.com/m/3301. Wang, S., Asilis, J., Akgül, Ö. F., Bilgin, E. B., Liu, O., Fu, D., and Neiswanger, W. Resa: Transparent reasoning models via saes. arXiv preprint arXiv:2506.09967, 2025. Yeo, W. J., Prakash, N., Neo, C., Lee, R. K.-W., Cambria, E., and Satapathy, R. Understanding refusal in language models with sparse autoencoders. arXiv preprint arXiv:2505.23556, 2025. Zhao, Y., Devoto, A., Hong, G., Du, X., Gema, A. P., Wang, H., He, X., Wong, K.-F., and Minervini, P. Steering knowledge selection behaviours in LLMs via SAE-based representation engineering. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5117–5136, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long. 264. URL https://aclanthology.org/2025. naacl-long.264/.
11
Cross-sample Consistency Regularization for Sparse Autoencoders
A. C2 R Promotes Cross-sample Consistency When feature absorption occurs, the C2 R Loss is: v um m+n X uX (i) (i) LC2 R (X) = t (z1 )2 + ((1 − α)z1 )2 i=m+1
i=1
v u m+n m+n X u X (i) (i) (αz )2 + (z )2 . +t 1
(19)
2
i=m+1
i=m+1
Taking the partial derivative with respect to α yields: P (i) 2 (α − 1) m+n ∂LC2 R (X) i=m+1 (z1 ) = qP Pm+n ∂α (i) 2 (i) 2 m i=m+1 ((1 − α)z1 ) i=1 (z1 ) + P (i) 2 α m+n i=m+1 (z1 ) . + qP Pm+n (i) 2 (i) 2 m+n (z ) (αz ) + 2 1 i=m+1 i=m+1
(20)
The correct loss should be proportional to α ∈ [0, 1]: ∂LC2 R (X) ≥ 0. ∂α
(21)
Introducing the shorthand notation: A=
B=
m X
(i)
(z1 )2
i=1 m+n X
(i)
(z2 )2
(22)
i=m+1
C=
m+n X
(i)
(z1 )2
i=m+1
The inequality simplifies to: √
αC α2 C + B
(1 − α)C . ≥p A + (1 − α)2 C
The final form is:
1
α≥ r P
(i) 2 m i=1 (z1 ) Pm+n (i) 2 (z i=m+1 2 )
(23)
(24) +1
B. Efficient Implementation Details Applying the C2 R constraint naively requires computing the cosine similarity between all pairs of decoder weight vectors, resulting in a k × k similarity matrix. For a dictionary size of k = 65, 536, this operation requires O(k 2 ) memory and computation, which significantly slows down training and increases VRAM usage. To address this, we implement two engineering optimizations following the methodology of OrtSAE (Korznikov et al., 2025): Chunk-wise Approximation. Instead of searching for the nearest neighbor j ∗ (i) across the entire dictionary, we randomly permute the feature indices at each step and partition the dictionary into smaller blocks (chunks). For a dictionary size k and chunk size C, we partition the latents into N = k/C chunks. The nearest neighbor search and loss computation are then restricted to within each chunk. j ∗ (i) ≈ arg max ⟨ŵi , ŵj ⟩. (25) j∈Chunk(i),j̸=i
12
Cross-sample Consistency Regularization for Sparse Autoencoders
In our experiments, we use a chunk size of C = 8, 192. This reduces the complexity from O(k 2 ) to O(C · C). Since high-cosine similarity features (redundant pairs) are rare and the permutation is randomized at every step, the probability of a redundant pair falling into the same chunk accumulates rapidly over training steps, ensuring the regularization remains effective. Periodic Updates. To further reduce the computational overhead, we compute the C2 R loss and its gradients only every T training steps rather than at every iteration. To maintain the same effective regularization strength over time, we scale the regularization coefficient λC2 R by the period T : Lstep t (X) = LSAE (X) + (1t mod T =0 · T · λC2 R )LC2 R (X).
(26)
We set T = 5 for all experiments of OrtSAEs and our approach. This configuration aligns our training cost with other SAEs’ training times, adding only negligible overhead (slightly more than the findings in OrtSAE, where overhead was reduced to < 10%).
C. Prompts for LLMs and Instructions for Human Annotators This appendix presents the detailed prompts used in the interpretability evaluation. The prompts were designed to elicit consistent reasoning from both LLM-based and human evaluators. Two types of LLMs were employed: an Explainer to describe latent semantics and a Predictor (or Judge) to estimate latent activation likelihood. For comparison, human annotators followed analogous instructions. C.1. LLM Explainer Prompt An example of the prompts used to generate textual descriptions for each latent activation is shown in Table 3. C.2. LLM Predictor Prompt An example of the prompts used by the LLM judge (the predictor) to decide whether the described latent would activate for each unseen test sample is shown in Table 4. C.3. Human Annotator Instruction Human annotators were provided with the latent’s same activating samples and unseen samples as the LLM explainer and the LLM predictor. An example of the instructions is shown in Table 5.
D. User Study Details We recruited three human annotators with high-school-level English proficiency, who replicated the explainer–judge process on 30 latents sampled from Batch TopK SAE and its C2 R-enhanced variant at layer 12 of Gemma-2-2B, covering five L0 settings. Each annotator produced 450 activation predictions. The probability of agreement between human annotators and GPT-5-mini, as well as the Pearson correlation coefficient (Pearson, 1895) between human- and LLM-derived scores, are summarized in Table 6. GPT-5-mini achieved a 97.3% match rate with humans and a Pearson r of 0.74, confirming its suitability as a proxy evaluator for automated interpretability assessment. The annotators are recruited from the university, and the compensation was set according to the standard payment guidelines for on-campus research participation. To verify that our results are not sensitive to the specific judge model, we re-evaluate the AutoInterp scores of all methods using GPT-5 and compute the per-method Pearson correlation with the original GPT-5-mini scores. As shown in Table 7, all correlations exceed 0.81 with p < 0.001, confirming high consistency.
E. Detailed Metrics Description The following six key metrics are used to evaluate the performance of SAEs integrated with C2 R: • Loss Recovered This metric is the primary measure of reconstruction fidelity. It quantifies the degree to which an 13
Cross-sample Consistency Regularization for Sparse Autoencoders
Prompt example for the LLM explainer We’re studying neurons in a neural network. Each neuron activates on some particular word/words/substring/concept in a short document. The activating words in each document are indicated with «token[act:activation]». We will give you a list of ACTIVATE documents, where the neuron fires, ordered by strength. Look at the marked parts of the ACTIVATE documents and summarize in a single sentence what the neuron is activating on. Try not to be overly specific or overly broad. Your explanation should cover most or all activating words. Pay attention to things like capitalization and punctuation if relevant. Keep the explanation as short and simple as possible, limited to 30 words or less. Omit punctuation and formatting. Some examples: "This neuron activates on the word ’knows’ in rhetorical questions", and "This neuron activates on verbs related to decision-making and preferences", and "This neuron activates on the substring ’Ent’ at the start of words", and "This neuron activates on text about government economic policy". The relevant documents are given below: ACTIVATE (1). see he was enjoying the other shapes too – the« round[act:55.5]» bowl and basket and the books underneath them, the ACTIVATE (2). The tube may be« cylindrical[act:16.75]» (or conical) with« circular[act:55.0]», rectangular or any desired cross section.←-By ACTIVATE (3). the factors affecting the appearance of impact craters←-The« circular[act:53.0]» features so obvious on the Moon’s surface are ACTIVATE (4). example, the simple cylindrical case which cylinder has a« circular[act:52.5]» cross section, will be considered in detail. If ACTIVATE (5). when Picasaweb closed. They consist of a« circular[act:50.0]» emitter Psurrounded by a ring shaped base N ACTIVATE (6). home.←-A mosquito bite appears as an itchy« round[act:50.0]», red, or pink skin bump. It’ ACTIVATE (7). of the large square sew-on. Take the« round[act:48.25]» sew-on and glue it on the left side ACTIVATE (8). multiple locations to accommodate various connection sizes and elevations.« Round[act:47.75]» or rectangular shapes available per design specifications. KnockACTIVATE (9). . Tie and suspend with gold thread from either our« round[act:44.25]» hoop or a stick of your choice.←-Hang ACTIVATE (10). bles has been played over the centuries with everything from« rounded[act:41.75]» sea pebbles to fruit pits, today the game is ACTIVATE (11). discount on Flashflight.com’s most popular« circular[act:41.25]» and spherical objects. From now until March 1 ACTIVATE (12). front extension, stone corbelling under eaves,« circular[act:41.25]» light in gable peak, slender turret with Christian cross ACTIVATE (13). piles of rock (called ejecta) around the« circular[act:41.25]» hole as well as←-bright streaks of target material ACTIVATE (14). waterproof back and an outer back with 16« round[act:41.0]» openings. Manufactured in 1967, ACTIVATE (15). A showcases her gorgeous slender body with swollen breasts,« round[act:40.75]» butt, and slender toes on the veranda.←Table 3. Prompt example for LLM explainer to explain a latent based on its activations.
14
Cross-sample Consistency Regularization for Sparse Autoencoders
Prompt example for the LLM predictor We’re studying neurons in a neural network. Each neuron activates on some particular word/words/substring/concept in a short document. You will be given a short explanation of what this neuron activates for, and then be shown 15 example sequences in random order. You will have to return a comma-separated list of the examples where you think the neuron should activate at least once, on ANY of the words or substrings in the document. For example, your response might look like "1, 2, 6, 9, 12". Try not to be overly specific in your interpretation of the explanation. If you think there are no examples where the neuron will activate, you should just respond with "None". You should include nothing else in your response other than comma-separated numbers or the word "None" - this is important. Here is the explanation: this neuron fires on words describing round or circular shapes including round circular rounded and cylindrical. Here are the examples: 1. in South Africa • Uganda) · Asia (in China • India • Myanmar • Pakistan • Taiwan • Japan 2. ized was either beheaded or shot at point blank range." more »←-A Syrian mother and widow was tortured 3. method’ anyone can do that but getting the right mindset to succeed. This is something most traders simply cannot 4. Facebook Be Fixed?←-CMS Wire (May 24, 2012) - Facebook 5. . Finished in a weathered brown and accented with a circular polished silver bezel. The metal dial has polished silver 6. when a customer has changed his or her mind about a transaction, or when an error has occurred, the 7. . Several of you have reached out to us and to our colleagues across the Administration. You’ve warned 8. crater is that←-you cannot see it. Its circular structure is nearly a kilometer below the←-surface and 9. sound crazy? Okay?←-DW:I’m just going to tell you the truth.←-THE 10. am I missing some key information here?<eos>The eleventh round of 2020 Monster Energy Super 11. awesome Flashflight Light-Up Flying Discs are circular, and our equally saucy Meteorlight LED Light 12. rather than doubling Defense on Dodge←-Strength ••, Brawl •←-Add Brawl rather than doubling Defense on Dodge 13. achieve a perfectly snug fit. Lastly the grain is circular-grained, after which the stone will not move 14. . Instead he drew a Dalek with two big round holes in it, and a guy catching a baseball 15. abb, Sean McDermott, Kevin Kolb). If the Eagles are waiting for a Packers assistant, the best Table 4. Prompt example for LLM judger to predict latent activations based on its explanation generated by the LLM explainer.
15
Cross-sample Consistency Regularization for Sparse Autoencoders
Instruction example for human annotators We’re studying neurons in a neural network. Each neuron activates on some particular word/words/substring/concept in a short document. The activating words in each document are highlighted. We will give you a list of ACTIVATE documents (where the neuron fires, ordered by strength), please look at the marked parts of the ACTIVATE documents. Summarize in a single sentence what the neuron is activating on. Try not to be overly specific or overly broad. Your explanation should cover all activating words. Pay attention to things like capitalization and punctuation if relevant. Keep the explanation as short and simple as possible, limited to 30 words or less. Omit punctuation and formatting. Some examples: "This neuron activates on the word ’knows’ in rhetorical questions", and "This neuron activates on verbs related to decision-making and preferences", and "This neuron activates on the substring ’Ent’ at the start of words", and "This neuron activates on text about government economic policy". The relevant documents are given below: ACTIVATE (1). TR UDA passes into your breast milk . Continue to take prednis olone regularly until your doctor tells you to ACTIVATE (2). 0.4% vs. You may take this medicine with or without meals . Please once you are cured ACTIVATE (3).
←- How the interaction
occurs : ←- When
these
two
medicines
are
taken
together ,
very
harmful
effects .
cime tidine may cause your body ACTIVATE (4). c.com.<eos>Serious. These medicines
may
interact
and
cause
Contact your healthcare professional ... ACTIVATE (15). bruising, or dark stools , contact your doctor right away . Your healthcare professionals may already be aware of this Based on your explanation of what this neuron activates for, please review the following 15 examples and indicate if you believe the neuron should activate at least once on ANY of the words or substrings within the document. Provide the corresponding text IDs. For instance, your response might look like "2, 3, 5, 6, 13". Avoid being overly specific in your interpretation of the explanation. Here are the examples: 1. off-the-wall in this first directorial effort from the 49-year-old Belgian 2. your organization.<eos>The Academy of Motion Picture Arts and Sciences which is best known for organizing the Oscars has 3. Realtors and the Mortgage Bankers Association.←-But this time, lobbyists are worried. That’s because 4. considered a natural antihistamine. Valtrex is used to treat herpes zoster and herpes simplex and, ... 15. of the stomach and intestines.←-Be sure to tell your doctor if you experience any of these side effects Table 5. Instruction example for human annotators to predict latent activation.
16
Cross-sample Consistency Regularization for Sparse Autoencoders
Annotator 1 Annotator 2 Annotator 3 Average
Prediction Match 97.8% 96.7% 97.6% 97.3%
Pearson r 0.74 0.63 0.86 0.74
Table 6. User study comparing GPT-5-mini (OpenAI, 2025) and human annotators in the automated interpretability task. The table reports the match rate and Pearson correlation (Pearson, 1895) between human- and LLM-derived AutoInterp scores.
Method C2 R Batch TopK Matryoshka Ort TopK
GPT-5-mini 0.9239 0.9208 0.9120 0.9196 0.8762
GPT-5 0.9218 0.9161 0.9031 0.9153 0.8735
Pearson r 0.94 0.89 0.89 0.81 0.94
Table 7. Consistency between GPT-5-mini and GPT-5 as AutoInterp judges. All Pearson correlations are significant (p < 0.001).
SAE can preserve the original language model’s Next-Token Prediction performance after its internal activations are (H ∗ −H0 ) reconstructed. It is defined as: Loss Recovered = (H where Horig is the original cross-entropy loss, H ∗ is the orig −H0 ) loss after replacement with SAE-reconstructed activations, and H0 is the loss after zero-ablating the original activations. A higher value indicates better reconstruction fidelity. • KL Div. Score (Gao et al.) As a complementary measure of reconstruction quality, this metric assesses how effectively the SAE’s reconstruction recovers the model’s output distribution from a zero-ablated baseline. It is a normalized score that quantifies the reduction in Kullback-Leibler (KL) divergence between the output logits and the original DKL (PSAE ∥Porig ) model’s logits distribution (Porig ). The score is calculated as: KL Div. Score = 1 − DKL (Pablated ∥Porig ) where DKL (Pablated ∥ Porig ) is the KL divergence when the activation is zero-ablated, and DKL (PSAE ∥ Porig ) is the KL divergence when the activation is replaced by the SAE reconstruction. This score is bounded between 0 and 1, where a higher score signifies superior reconstruction performance relative to the zero-ablated state. • AutoInterp (Paulo et al.) This metric evaluates the human-understandability of learned latents using LLMs. It operates in two stages: an LLM generates a feature description based on activating inputs, and another LLM judge uses this description to predict latent activation on new sequences. The prediction accuracy serves as the AutoInterp score. • Split Num (Chanin et al., 2024) A diagnostic metric focusing on feature splitting. This metric serves as a proxy for the granularity and non-redundancy of the learned latents. It is measured by identifying a single high-level concept (e.g., all tokens starting with a specific letter) and counting the minimum number of distinct SAE latents required to significantly improve classification performance on a probe for that concept. A higher count can indicate that more latent are fragmented into distinct components. • Absorption (Chanin et al., 2024) This metric focuses on feature absorption. It measures the tendency of an SAE to learn two coupled hierarchical features (e.g., A and “B excluding A”) instead of two independent features (A and B). The metric is calculated by diagnosing the activation patterns of the general SAE latents: measuring the frequency at which they fail to activate when a token-aligned child latent is present in the input. A lower score is desirable, indicating better feature isolation and reduced feature absorption. • Max CosSim This metric quantifies the maximum cosine similarity between the decoder weight vectors of all learned latents, reflecting the feature composition level of SAEs (Bussmann et al.). High similarity suggests significant directional overlap or redundancy among SAE latents. • Disentanglement (Huang et al., 2024) This benchmark evaluates the ability of interpretability methods to disentangle independent attributes within language model representations. It utilizes interchange interventions to test whether a targeted feature (e.g., city country) can be modified without affecting other related attributes (e.g., city language). The performance is summarized by the disentangle score: Disentangle Score = 12 (Cause + Iso) where Cause measures the 17
Cross-sample Consistency Regularization for Sparse Autoencoders
success rate of changing the target attribute’s value through intervention, and Iso (Isolation) measures the frequency with which non-target attributes remain unchanged. A high score indicates that the SAE has successfully localized individual concepts into independent, causal units.
F. GPU Budget We ran all SAE training experiments utilizing an NVIDIA H800 GPU, consuming a total of 300 GPU hours and achieving a peak memory utilization of 65 GB. Evaluating a trained SAE across all metrics required approximately 1 GPU hour. The breakdown of evaluation time per SAE is as follows: • Reconstruction Fidelity Metrics: 30 minutes • Interpretability Analysis: 10 minutes • Feature Hierarchy Metrics: 10 minutes • Disentanglement Metrics: 15 minutes In terms of memory footprint, the reconstruction fidelity evaluation was the most demanding, requiring up to 50 GB of VRAM. The interpretability and feature hierarchy analyses were less memory-intensive, each requiring approximately 15 GB.
G. Robustness and Generalization Method Batch TopK Matryoshka Ort C2 R
KL Score ↑
Interp ↑
Composition ↓
Absorption ↓
Split ↓
0.9621 ± 0.0006 0.9597 ± 0.0008 0.9615 ± 0.0006 0.9628 ± 0.0004
0.9311 ± 0.0098 0.9311 ± 0.0120 0.9297 ± 0.0082 0.9309 ± 0.0052
0.3325 ± 0.0003 0.1870 ± 0.0003 0.1038 ± 0.0004 0.0991 ± 0.0003
0.1908 ± 0.0240 0.0360 ± 0.0066 0.0819 ± 0.0203 0.0656 ± 0.0185
1.0962 ± 0.0722 1.0673 ± 0.0361 1.0577 ± 0.0218 1.0385 ± 0.0238
Table 8. Statistical validation with 95% confidence intervals over 5 independent runs (Gemma-2-2B, layer 12, k=100). Bold and underline indicate the best and second-best values, respectively.
Method Batch TopK Matryoshka Ort C2 R
KL Score ↑
Interp ↑
Composition ↓
Absorption ↓
Split ↓
0.9889 0.9870 0.9886 0.9874
0.9220 0.9277 0.9214 0.9333
0.2677 0.1422 0.0945 0.0675
0.0074 0.0032 0.0034 0.0018
1.1154 1.1154 1.1154 1.1154
Table 9. Results on Qwen3-8B (layer 20, k=100).
Method Batch TopK Matryoshka Ort C2 R
KL Score ↑
Interp ↑
Composition ↓
Absorption ↓
Split ↓
0.9860 0.9853 0.9859 0.9858
0.9405 0.9552 0.9265 0.9443
0.3334 0.1803 0.1200 0.0816
0.1470 0.0349 0.0979 0.0410
1.1923 1.1154 1.0385 1.0385
Table 10. Results on Llama-3-8B (layer 20, k=100).
18
Cross-sample Consistency Regularization for Sparse Autoencoders
Method TopK Batch TopK Matryoshka Ort C2 R
KL Score ↑
Interp ↑
Composition ↓
Absorption ↓
Split ↓
0.9720 0.9706 0.9727 0.9722 0.9727
0.9307 0.9323 0.9410 0.9543 0.9570
0.3270 0.3329 0.1781 0.1000 0.0781
0.1325 0.1087 0.0205 0.0352 0.0260
1.5769 1.3462 1.2308 1.2692 1.2692
Table 11. Results on Gemma-2-2B layer 20 (k=100).
Method TopK Batch TopK Matryoshka Ort C2 R
KL Score ↑
Interp ↑
Composition ↓
Absorption ↓
Split ↓
0.9631 0.9621 0.9598 0.9629 0.9631
0.9196 0.9313 0.8870 0.9072 0.9302
0.3416 0.3430 0.1892 0.1053 0.0870
0.2987 0.2149 0.0241 0.1058 0.0654
1.2692 1.1538 1.0769 1.0385 1.0385
Table 12. Results with 1B training tokens on OpenWebText (Gemma-2-2B, layer 12, k=100).
H. Ablation Study Table 14 reports the component ablation results, where removing the nearest-neighbor restriction (NoNNR) or the ReLU cosine gate (NoRCG) leads to severe degradation in reconstruction fidelity or inflated composition scores. Table 15 reports the sensitivity of C2 R to the regularization strength λC2 R , showing that λC2 R =5 achieves the best trade-off.
I. Compatibility with Other SAE Architectures Table 16 reports the full results of applying C2 R to three different SAE backbones: TopK, OrtSAE, and AbsTopK. In each pair, adding C2 R improves all structural metrics without degrading reconstruction fidelity, confirming that C2 R is a backbone-agnostic regularizer.
J. Downstream Causal Intervention Tasks Table 17 reports the results on two causal intervention tasks from SAEBench. C2 R achieves the second-best performance on both SCR and TPP, outperforming TopK, Batch TopK, and Ort, and achieving comparable results to Matryoshka.
K. Empirical Verification of Eq. 13 We verify the condition in Eq. 13 using our trained SAEs on Gemma-2-2B layer 12 with a 4M-token test set from SAEBench. Pm
i=1 Figure 4 shows the distribution of the log A/B ratio, i.e., log Pm+n
(i)
(z1 )2 (i)
2 i=m+1 (z2 )
, across all absorption pairs. The median A/B
ratio is 81.56, confirming that the cumulative energy of the primary feature strongly dominates the residual in practice. Figure 5 shows the scatter plot of α versus the right-hand side of Eq. 13 for all absorption pairs. Points above the diagonal satisfy the condition. 88.1% of pairs (N =4,555) fall in the satisfied region, and the violated pairs are concentrated in a low-α regime where absorption is minimal.
L. Relationship to OrtSAE The chunk-wise nearest-neighbor calculation in Appendix B is an engineering optimization to reduce complexity and control variables, not a methodological dependence. Fundamentally, OrtSAE constrains decoder weights, whereas C2 R operates in activation space using the Minkowski inequality. In C2 R, decoder cosine similarity acts as a gating weight (ρ2 ) to target redundant features, creating a gradient dynamically scaled by activation magnitude S. This adaptive approach distinguishes C2 R from OrtSAE’s uniform penalty, protecting low-frequency features from premature destruction. As shown in Table 16, 19
Cross-sample Consistency Regularization for Sparse Autoencoders
Method Batch TopK Matryoshka Ort C2 R
KL Score ↑
Interp ↑
Composition ↓
Absorption ↓
Split ↓
0.9606 0.9602 0.9611 0.9618
0.9211 0.9117 0.9202 0.9424
0.3227 0.1781 0.1053 0.0766
0.1920 0.0158 0.0627 0.0396
1.1538 1.0385 1.0769 1.0385
Table 13. Results on The Pile (Gemma-2-2B, layer 12, k=100).
Method Batch TopK Ort C2 R (NoNNR) C2 R (NoRCG) C2 R
KL Score ↑
Interp ↑
Composition ↓
Absorption ↓
Split ↓
0.9598 0.9617 0.9439 0.8439 0.9629
0.9208 0.9208 0.9488 0.9873 0.9239
0.3321 0.1046 0.5655 0.4326 0.0990
0.1985 0.0606 0.0255 0.0520 0.0590
1.1154 1.1154 1.1538 1.1923 1.0769
Table 14. Component ablation. NoNNR: without the nearest-neighbor restriction. NoRCG: without the ReLU cosine gate. Removing either component causes a severe degradation in reconstruction fidelity or feature composition.
C2 R can be applied on top of OrtSAE to achieve further improvements, confirming that the two methods are complementary.
M. Sensitivity to Feature Frequency Since C2 R aggregates batchwise ℓ2 norms, the effective regularization strength on a given latent pair scales with how frequently those features appear in the batch. To investigate whether rare features receive insufficient regularization, we analyze the relationship between feature frequency and absorption rate using 26 letter-specific features from SAEBench on a Gemma-2-2B Batch TopK SAE. As shown in Figure 6, the Pearson correlation between feature frequency and absorption rate is r = +0.301 (p = 0.135), which is not statistically significant. The Spearman correlation is ρ = +0.403 (p = 0.041), suggesting a slight positive trend where absorption may marginally increase with frequency rather than decrease. Rare features such as Q, J, and X exhibit absorption rates below 0.1, comparable to or lower than high-frequency features such as C and S. This indicates that the consistency pressure remains effective across the frequency spectrum and that rare features do not suffer from reduced reliability due to batch statistics.
N. Dead Feature Rates Table 18 reports the dead feature rates for all methods on Gemma-2-2B layer 12. C2 R maintains high dictionary utilization with a dead feature rate under 1%.
20
Cross-sample Consistency Regularization for Sparse Autoencoders
KL Score ↑
Interp ↑
Composition ↓
Absorption ↓
Split ↓
– (Batch TopK) – (Ort)
0.9598 0.9617
0.9196 0.9196
0.3321 0.1046
0.1985 0.0606
1.1154 1.1154
0.1 0.5 1 5 10
0.9629 0.9638 0.9633 0.9629 0.9521
0.9291 0.9496 0.9407 0.9239 0.9168
0.2944 0.2138 0.1676 0.0990 0.0777
0.1926 0.1726 0.1349 0.0590 0.0425
1.0385 1.0385 1.1154 1.0769 1.0385
λC2 R
Table 15. Sensitivity to λC2 R . Baselines (Batch TopK and Ort) are shown above the rule for reference. λC2 R =5 achieves the best trade-off between reconstruction fidelity and structural metrics.
KL Score ↑
Interp ↑
Composition ↓
Absorption ↓
Split ↓
TopK TopK + C2 R
0.9620 0.9627
0.8762 0.9227
0.3311 0.1004
0.2680 0.0752
1.3462 1.0769
Ort Ort + C2 R
0.9617 0.9623
0.9196 0.9217
0.1046 0.0662
0.0606 0.0524
1.1154 1.0385
AbsTopK AbsTopK + C2 R
0.9576 0.9580
0.6759 0.7579
0.2357 0.0810
0.1407 0.0846
1.3462 1.0769
Method
Table 16. C2 R applied to different SAE backbones (Gemma-2-2B, layer 12, k=100). Within each pair, the better value is in bold. C2 R consistently improves structural metrics across all architectures.
Method
SCR ↑
TPP ↑
TopK Batch TopK Matryoshka Ort C2 R
0.0481 0.0353 0.1172 0.0859 0.1047
0.0038 0.0045 0.0370 0.0061 0.0226
Table 17. Downstream causal intervention tasks (Gemma-2-2B, layer 12, k=100). SCR: Spurious Correlation Removal. TPP: Targeted Probe Perturbation. Bold and underline indicate the best and second-best values, respectively.
Method
Dead Feature Rate ↓
TopK Batch TopK Matryoshka Ort C2 R
0.11% 0.06% 0.10% 4.61% 0.90%
Table 18. Dead feature rates on Gemma-2-2B layer 12. C2 R maintains over 99% dictionary utilization.
21
Cross-sample Consistency Regularization for Sparse Autoencoders
Distribution of log10 (A/B) N = 4,947 | median A/B = 81.56 A/B: mean=2980.53 med=81.56 std=21211.32 log (A/B): mean=2.03 med=1.91 std=0.96
median log (A/B)=1.91 mean log (A/B)=2.03
0.5
Density
0.4 0.3 0.2 0.1 0.0
2
1
0
1
2
log10 (A/B)
3
4
Figure 4. Distribution of the log A/B ratio across absorption pairs (N =4,947, median = 81.56).
22
5
6
Cross-sample Consistency Regularization for Sparse Autoencoders
1.0
Verification of Eq. 13: RHS Satisfied: 88.1% | Violated: 11.9% | N = 4,555 RHS (4,011) < RHS (544) = RHS
Estimated
0.8
0.6
0.4
0.2
0.0 0.0
Satisfied: 88.1% 0.2
0.4
RHS
0.6
1 A/B + 1
0.8
Figure 5. Scatter plot of α vs. RHS of Eq. 13 (N =4,555). 88.1% of pairs satisfy the condition.
23
1.0
Cross-sample Consistency Regularization for Sparse Autoencoders
Feature Frequency vs. Absorption Rate I
0.6
Linear fit (slope=5.74e-05)
O
Absorption Rate
0.5 0.4 U
0.3
W
0.2
F
Y
H N V
X Q
0.0 0
J
L
C
M
S
T
A
Pearson r = +0.301 (p = 0.135) Spearman = +0.403 (p = 0.041)
K
500
P
ER
G
Z
0.1
D B
1000
1500
Feature Frequency
Batch TopK SAE
2000
2500
gemma-2-2b, layer 12
Figure 6. Absorption rate vs. feature frequency for 26 letter-specific features (Gemma-2-2B, Batch TopK SAE). No significant negative correlation is observed, indicating that rare features do not suffer from higher absorption.
24