Beyond a Bag of Features: Set-Level Instability in Sparse Autoencoders

Beyond a Bag of Features: Set-Level Instability in Sparse Autoencoders Nikolai Bolik and Lennart Stöpler and Artur Andrzejak Heidelberg University {nikolai.bolik,lennart.stoepler,artur.andrzejak}@uni-heidelberg.de

arXiv:2608.11197v1 [cs.LG] 11 Aug 2026

Abstract Shani et al. (2026) show that LLM representations broadly recover human category boundaries, while failing to reflect fine-grained typicality structure. Their analysis uses cosine similarity over dense model representations. We revisit their approach using overlap over active sparse autoencoder (SAE) latent sets as a more interpretable similarity measure. We first verify that this set-level measure is meaningful: SAE latent sets can recover union-like compositional structure in controlled toy models and induce semantically coherent neighborhoods in natural text. Extending the human-concepts analysis to SAE set similarities, we find that SAE activation sets do not recover human category boundaries or within-category typicality more faithfully than dense embeddings or residual-stream states, but instead track modelinternal similarity structure. To probe this gap further, we study active latent sets under wellcontrolled semantic modifications, revealing a substantial mismatch between human judgements of conceptual change and change in the SAE active set. We interpret this as evidence that, outside idealised settings, SAE features do not compose via simple bag-of-features semantics.

1

Introduction

In a recent study Shani et al. (2026) probe how human concepts are encoded in Transformer models. Using cosine similarity of token embeddings as a measure of semantic closeness, they find that LLM-derived similarity clusters broadly align with human conceptual structure, while failing to reflect the fine-grained internal organisation of those clusters. A natural follow-up question is whether a more interpretable similarity measure changes this picture. Active feature sets from sparse autoencoders (SAEs) (Cunningham et al. (2024); Templeton et al. (2024); inter alia) offer such a candidate: They decompose Transformer activations

into sparse features (Elhage et al., 2022; Park et al., 2024; Bricken et al., 2023), with individual latents shown to admit meaningful semantic interpretations, scaling up to state-of-the-art model sizes (Bricken et al., 2023; Cunningham et al., 2024; Templeton et al., 2024; Lieberum et al., 2024). We therefore revisit the experiments of Shani et al. (2026), replacing cosine similarity of embeddings with overlap over active SAE latent sets1 . While active SAE latent sets are increasingly used as objects of analysis (Park et al., 2025b; Olson et al., 2025; Wattenberg and Viégas, 2024; Olah, 2024), most SAE work focuses on the interpretability of individual latents (Gao et al., 2025; Bussmann et al., 2024; Rajamanoharan et al., 2024a,b). We therefore first test whether set-level SAE similarity is meaningful in our setting. In preliminary studies, we show that SAE latent-set overlap forms semantically coherent clusters on large text corpora, and that SAEs can compose in a bag-offeatures manner in a controlled synthetic toy model. We then replicate the conceptual-similarity experiments of Shani et al. (2026) using SAE latent-set overlap. The qualitative picture is preserved: SAEfeature similarity broadly tracks human conceptual groupings. However, SAE-based similarity performs slightly worse than raw cosine similarity, despite the interpretability of individual latents. To understand this gap, we study active latent sets under controlled semantic modifications of the input and find a substantial mismatch between human judgements of conceptual change and changes in SAE active latent sets. We interpret this as evidence that, outside idealised settings, SAE features do not compose via simple bag-of-features semantics. We summarise our main contributions as follows: 1. We provide preliminary evidence that active 1

In this work, we use feature to refer to a ground-truth concept the model operates on, and latent to refer to a learned SAE direction that may encode such a feature.

SAE latent sets are a meaningful unit of analysis, using clustering experiments on naturalistic text sequences and a toy-model study demonstrating bag-of-features compositionality on synthetic inputs. 2. We replicate the conceptual-similarity experiments of Shani et al. (2026) with SAE latentset overlap as the similarity measure, finding qualitatively similar but quantitatively weaker alignment with human conceptual structure than cosine similarity. 3. We study active SAE latent sets under controlled semantic modifications, revealing a mismatch between human conceptual change and latent-set change, suggesting that bag-offeatures compositionality does not hold in naturalistic settings.

2

Terms and Definitions

Linear Representation Hypothesis A central conjecture in mechanistic interpretability is the Linear Representation Hypothesis (LRH) (Elhage et al., 2022; Park et al., 2024). It states that the residual stream of a Transformer encodes many human-interpretable concepts as directions in activation space, and that model computations can be understood as non-linear operations over these linearly embedded concepts. In the form commonly assumed in the SAE literature, a representation vector x is decomposed as X x=b+ fi di , (1) i

with non-negative coefficients fi , directions di ∈ Rd , and bias b ∈ Rd (Bricken et al., 2023).2 Since the number of encoded concepts is assumed to vastly exceed the dimensionality of the representation space, the directions di cannot all be orthogonal, and the model must be robust to the resulting interference. Sparse Autoencoders If the LRH holds, recovering the directions di and coefficients fi from observed activations is an instance of dictionary learning, i.e. finding an overcomplete dictionary together with sparse coordinates. This is a wellstudied problem in compressed sensing (Donoho, 2006; Candes et al., 2006), where sparsity regularizes an otherwise underdetermined problem. 2

More complex formulations of the LRH exist, which we discuss in Section 6.

Sparse autoencoders operationalize this idea as one-hidden-layer MLPs trained to reconstruct their input while enforcing sparse hidden activations. Given x ∈ Rd , encoder weights We ∈ Rm×d , decoder weights Wd ∈ Rd×m , and biases be ∈ Rm , bd ∈ Rd , an SAE computes3 F (x) = τ (We (x − bd ) + be ) , (G ◦ F )(x) = Wd F (x) + bd ,

(2) (3)

where τ is a sparsity-inducing nonlinearity such as ReLU (Bricken et al., 2023; Templeton et al., 2024) or JumpReLU (Rajamanoharan et al., 2024b), and m ≫ d makes the representation overcomplete. The columns of Wd are the learned latents, conventionally constrained to unit norm (Olshausen and Field, 1996; Bricken et al., 2023); the hidden activations F (x) provide the sparse coordinate representation of x. Active Latents and Signatures. We call a latent active if F (x)i > 0. The signature of an activation is the set of active latent indices. For a collection of activations, such as all activations induced by an input, we define its signature as the union of the individual signatures.

3

Preliminary Studies

Our analysis treats sets of active SAE latents, i.e. SAE signatures as the unit of analysis. While prior work motivates such set-level perspectives (Park et al., 2025b; Olson et al., 2025; Wattenberg and Viégas, 2024; Olah, 2024), it is not obvious that signatures are meaningful objects in their own right. We therefore first test whether they carry interpretable structure. This section summarizes the main observations; detailed results are provided in Appendix B. In a toy-model study, we use a controlled substitute for feature composition. Synthetic residualstream states are generated from known directions, representing a sparse set of ground-truth features. SAEs are trained to reconstruct these synthetic states. We then compare the SAE signature for a composed input, in which two ground-truth features are active, with the union of the signatures obtained when each feature is presented alone. We 3 We adopt the pre-encoder bias parameterization used by Anthropic (Bricken et al., 2023), which treats the decoder bias as a constant shift in the data and most closely matches Equation 1. SAEs trained without a pre-encoder bias can be converted to this form after training.

a) b)

Figure 1: Excerpts from the preliminary studies in Appendix B. Panel (a) shows that sufficiently wide SAEs recover set-consistent behaviour with respect to active ground-truth latents. Jaccard, recall, and precision compare the composed-input signature with the union of the individual-feature signatures; the red dotted line marks N = 70, the number of generating ground-truth features. Panel (b) shows the first ten entries of an example cluster from The Pile (Gao et al., 2020), extracted using SAE latent-set overlap. The cluster groups contexts related to political conferences, with highest-overlap tokens often corresponding to the hosting city or organisation.

call this union consistency. Under a bag-of-features view, adding an independent feature should add corresponding latents without removing those already active. Figure 1(a) shows that sufficiently wide SAEs can satisfy this property, with high Jaccard, recall, and precision between the composed signature and the expected union. Details and further experiments are given in Appendix B.1. We then test coherence of the set-level view on natural text. We sample 30,000 snippets from The Pile, compute pairwise distances using a SAE latent set overlap based metric, and construct local neighborhoods around seed snippets. The resulting neighborhoods are often semantically interpretable. An example is shown in Figure 1 (b), with details and further examples in Appendix B.2. These two checks suggest that SAE signatures can support compositional structure in controlled settings and meaningful semantic neighborhoods in real data.

4

Experiment 1 - Typicality Judgements

Shani et al. (2026) show that LLM representations, extracted from static embeddings and hidden states, broadly recover human-like category boundaries, but only weakly reproduce the internal associative geometry of human concepts. In particular, model states reflect the grouping of items into categories similarly to humans, while failing to reliably capture graded typicality judgments within categories, such as the intuition that a robin is a more typical bird than a penguin. We closely follow their experimental setup and evaluation procedure, but extend the analysis to sparse feature representations induced by SAEs. Since SAEs are commonly motivated as decomposing model activations into more monosemantic and human-interpretable latent features (Templeton et al., 2024; Cunningham

et al., 2024; Bricken et al., 2023), one might expect their active feature sets to expose conceptual structure more directly than dense residual-stream states. We therefore ask two related questions; whether SAE-induced similarities recover human category boundaries, and whether they better capture finegrained within-category typicality than dense representations. Design For this analysis, we use the humanconcepts dataset introduced by Shani et al. (2026), based on classic psychology studies of category typicality (Rosch, 1973, 1975; McCloskey and Glucksberg, 1978). The dataset consists of semantic categories, such as bird, paired with category instances, such as hawk or penguin, together with human typicality scores shum indicating how representative each instance is of its category. To evaluate modelinduced category structure, we first define similarity scores between arbitrary word pairs; instance– instance pairs for category-boundary clustering, and category–instance pairs for within-category typicality. Since words may consist of multiple tokens, we first aggregate token-level representations into a single word-level representation. Following the setup from Shani et al. (2026), we use static input embeddings for the raw word tokens. For contextual hidden states, we use the neutral prompt template "This is a {word}._" and evaluate only the token positions corresponding to the inserted word, thereby controlling the context without involving the surrounding prompt in the direct evaluation. Let T (w) denote the token positions of the target word w. For contextual representations, xℓt (w) is the residual-stream state of token t at layer ℓ.4 We 4

For static embeddings, xt (w) is taken directly from the embedding matrix and the layer index is omitted.

Figure 2: Category-boundary and typicality analyses. Panel (a) reports AMI-based category-boundary recovery across layers, comparing dense residual-stream cosine similarity sℓcos from Equation (5) with SAE set similarity sℓJac from Equation (8). Panel (b) shows per-category Spearman correlations between human typicality rankings and SAE-Jaccard rankings. Panel (c) compares residual-stream cosine rankings with SAE-Jaccard rankings.

compute the word-level representation by averaging over target-word tokens, X 1 x̄ ℓ (w) = xℓt (w), (4) |T (w)| t∈T (w)

and use cosine similarity as the model-induced similarity score for two words w and w′ :   sℓcos (w, w′ ) = cossim x̄ ℓ (w), x̄ ℓ (w′ ) . (5) For the SAE-based comparison, we use the same prompted inputs and target-token positions. We encode the corresponding residual states with the SAE, binarize the latent activations, and aggregate all active latents across target-word tokens into a single word-level SAE signature. [ S ℓ (w) = {k | zkℓ (w, t) > 0}, (6) t∈T (w) ℓ

z (w, t) = Fℓ (xℓt (w)),

(7)

where F corresponds to the SAE encoder from Equation 2. Similarity for SAE signatures is then measured by Jaccard similarity between the corresponding word-level SAE signatures: sℓJac (w, w′ ) =

|S ℓ (w) ∩ S ℓ (w′ )| . |S ℓ (w) ∪ S ℓ (w′ )|

(8)

With these similarity scores in place, we evaluate category structure at two levels. First, following Shani et al. (2026), we ask whether the induced geometry recovers human category boundaries. For each representation, we compute pairwise similarities between category instances, convert them to distances d = 1 − s, cluster instances with Kmedoids using K equal to the number of groundtruth categories, and compare the resulting partition to human category labels using Adjusted Mutual Information (AMI). AMI is chance-corrected:

AMI = 1 indicates perfect agreement with human labels, whereas AMI ≈ 0 indicates chance-level agreement. Further information theoretical motivation on AMI is given by Shani et al. (2026) and a brief introduction is provided in Appendix C. Second, we evaluate within-category typicality. For each semantic category c, the category–instance similarities sℓcos (c, i) or sℓJac (c, i) induce a ranking over instances i. We compare this ranking to the human typicality ranking using Spearman rank correlation. This tests whether representations recover not only coarse category boundaries, but also fine-grained human judgments about which instances are central category members. If SAE latents capture semantically meaningful and human-interpretable factors more directly than dense model states, one might expect sℓJac to align more closely with human typicality than cosine similarity over embeddings or residual-stream states. Unless stated otherwise, main-text results use Gemma 3 270M with the 16k big SAE; further models and SAE variants are in the appendix. Discussion Shani et al. (2026) found that LLM representations can recover human category boundaries while remaining largely misaligned with human within-category typicality rankings. We recover the latter result for embedding state similarities and extend the analysis to SAE-induced set similarities. The category-level clustering AMI results show that SAE activation sets do not recover human category boundaries more reliably than dense residual-stream representations, as can be read from Figure 2 (a). Interestingly, the highest SAE-based AMI remains below the highest embedding-based AMI, a pattern observed in many investigated settings. Additional AMI results are provided in Appendix C.

Similarly, the within-category typicality analysis shows that human typicality rankings and SAE set-based similarity scores are essentially uncorrelated. As shown in Figure 2 (b), the per-category Spearman rank coefficients fluctuate around zero for most categories, indicating that Jaccard overlap between SAE signatures does not systematically recover human typicality structure. Additional models and SAE variants are reported in Appendix D. Taken together, SAE activation sets do not recover human category understanding more faithfully than dense embedding states, either for coarse category boundaries or for graded within-category typicality. This is surprising given their role as interpretability tools (Bricken et al., 2023; Cunningham et al., 2024; Templeton et al., 2024) and the qualitative cluster structure observed in the preliminary studies Section 3 and Appendix B.2. Instead, SAE-based Jaccard similarities more closely track model-internal similarity structure, as can be seen in Figure 2 (c). Since these correlations are not one, however, SAE activations appear to capture a related but partially distinct structure rather than simply reproducing dense-state similarity.

5

Experiment 2 - Set Consistency under Semantically Related Inputs

Having observed that SAE activation sets align more closely with model-internal similarity structure than with human conceptual intuition, we next examine their compositional consistency under near-identical prompts that differ only by the addition of a single adjective. Motivated by prior work on feature-based representations (Park et al., 2025b; Olson et al., 2025; Wattenberg and Viégas, 2024; Olah, 2024) and by our LRH-aligned toymodel preliminary investigation in Appendix B.1, we ask whether SAE signatures behave compositionally under such refinements. If SAE latents faithfully decompose an activation into its underlying active concepts, adding an adjective should refine rather than overwrite the representation. A shovel should not become less of a shovel because it is yellow; rather, yellow shovel should preserve latents associated with shovel while additionally activating latents associated with yellow. This motivates a union-of-properties hypothesis: Composing an object description with a compatible property should approximately expand the active latent signature, so that the composed signature is close to the union of the base-object and adjective signa-

tures. Our preliminary results in Appendix B.1 show that SAEs can satisfy such set-consistent behaviour in controlled settings.5 Compositional Consistency We construct sequences of increasingly specific prompts by prefixing adjectives to a noun, yielding chains of the form t0 = noun, t1 = adj1 noun, ...

(9)

tk = adj1 · · · adjk noun. We then compare the SAE signatures induced by these related prompts. Similar to the typicality analysis above, we use the fixed prompt template "This is a {ti }._", where ti denotes the phrase obtained at construction step i. This fixes the surrounding context while varying only the composed noun phrase. Analogously to Equation (7), we compute SAE signatures over the token positions corresponding to the "noun" word. We construct 200 such sequences from a hand-curated dataset, typically involving adjectives such as color, condition, size, and other descriptive modifiers. For each noun, five adjectives are available, so that kmax = 5. 5.1

Overview

Our analysis proceeds in four steps; Figure 3 outlines steps (i)–(iii). Fixed prompt template tokens are excluded from the evaluation. (i) Measuring lost latents We identify SAE latents active for the base prompt t0 but inactive for the more specific prompt tk , measuring how much of the target-noun signature is removed by adding compatible adjectives. (ii) Upstream occurrence We ask whether these lost latents appeared earlier in the computation, separating previously active latents from those not detected upstream. (iii) Pre-activation analysis For both groups, we inspect SAE pre-activations before the nonlinearity, αpre = We (x − bd ) + be .

(10)

5 This behavior is structurally unattainable for SAEs with fixed sparsity constraints, such as top-k SAEs (Gao et al., 2025), where activating new latents necessarily forces others to deactivate in order to maintain constant support size.

Figure 3: Schematic of steps (i)–(iii). Lost latents are grouped by upstream presence and pre-activation regime. For previously active latents, pre-activations are evaluated at layer l; for never-previously-active latents, across the upstream trajectory.

We distinguish near-zero latents, indicating weak alignment (Rajamanoharan et al., 2024a), from strongly negative pre-activations, which we interpret as stronger suppression (Mayne et al., 2024; Elhage et al., 2021). (iv) Qualitative interpretation Finally, we qualitatively discuss which types of latents are lost. 5.1.1 Set-Consistency Degradation We quantify degradation from compositional consistency by comparing the SAE signature of the noun in the base prompt t0 with the corresponding noun signature in progressively more specific prompts tk , obtained by prefixing adjectives. Thus, the comparison is always made on the noun token positions. This measures how much of the original noun-related signature is lost as the surrounding description becomes more specific. We define the (k) lost-latent rate Rℓ as the fraction of latents active for the base prompt that are no longer active for the more specific prompt: (k)

Rℓ

=

F ℓ (t0 ) \ F ℓ (tk ) F ℓ (t0 )

.

(11)

Here, F ℓ (tk ) denotes the signature layer ℓ over the noun token positions under the prompt containing tk . Under the compositionality assumption from Section 5, this set of latents should largely be preserved when compatible adjectives are added. As shown in Figure 4 (a), lost-latent rates are substantial, typically between 20% and 60%, increasing with k and often also with model depth, as indicated by the inset. A more detailed breakdown by model, layer and k is provided in Appendix E.2. The central observation is that adding

a semantically compatible adjective can deactivate a large fraction of latents that were active for the less specific noun prompt. Although the resulting signatures are not entirely disjoint, this directly contradicts the simple union-of-properties expectation, under which the more specific prompt should preserve object-related noun latents while adding adjective-related ones. It also suggests a possible intuition for why human semantic coherence correlate poorly with SAE activation based distance metrics, as discussed in Section 4. This raises the question of why the anticipated behavior fails and what mechanism drives the observed loss of latents. 5.2

Upstream Presence of Lost Latents

Having established that SAE set compositional consistency is violated, we relax this strict local criterion and ask whether lost latents were nevertheless recoverable earlier in the computation before disappearing at the inspected noun position. For latents active on the base prompt t0 but absent from the noun signature of tk at layer ℓ, we use the layer-ℓ SAE as a cross-layer detector, since this SAE is known to resolve the lost latent. Following related uses of SAEs as transferable probes (Lindsey et al., 2024; Balagansky et al., 2025), we apply this detector to residual-stream states from layers 0, . . . , ℓ. This separates latents that were used earlier and later disappear from the local noun signature from latents not detected along the probed trajectory. Figure 4 (b) shows the fraction of lost latents that were active upstream when probing the full sequence. This fraction varies with the layer at which the latent is lost, as it is low in early layers, rises to roughly 60% in middle layers, and decreases again in later layers. Thus, a substantial subset of lost latents was previously recoverable from the computation, but disappears at layer ℓ at the noun position where loss is measured. The inset restricts the upstream check to noun signatures in previous layers. Its close agreement with the main curve suggests that upstream-active lost latents were usually already active on the noun itself, rather than only elsewhere in the prompt. We interpret this to mean that–despite violating compositional consistency in the stricter sense–a large fraction of lost latents remain in use by the model. Their disappearance could as such be a result of SAEs only probing a single point in the residual stream, rather than a failure to resolve composition. However, the results also show that this explanation is insufficient to account for all lost latents. Additional results are

Figure 4: Set-consistency analysis for Gemma 3 270M with the 16k big SAE. Panels show dataset means with one-standard-deviation bands. Panel (a) reports the lost-latent rate R from Equation (11) as adjectives are added; the inset shows the layerwise trend for k = 1. Panel (b) shows, among latents lost on the noun signature at layer ℓ, the fraction previously active upstream when probing all tokens of the constructed phrase; the inset restricts this check to noun signatures. Panel (c) decomposes this previously active subset at the layer ℓ where loss is measured, separating near-zero from negative pre-activations over the phrase tokens; the inset shows the noun-only variant. Panel (d) applies the same near-zero/negative diagnostic to never-previously-active lost latents across the upstream trajectory of the constructed phrase; the inset again restricts to noun signatures. Panels (b)–(d) use k = 1, representative across investigated models as shown in Appendix E.2.1. Additional models and SAE variants are shown in Appendix E.2.

provided in Appendix E.2. 5.3

Pre-activation regimes of lost latents

To characterize lost latents beyond binary activation, we inspect their SAE pre-activations from Equation (10). We attempt to distinguish cases where the latent is inhibited by a strong negative alignment with the residual-stream from cases where it is simply below its activation threshold. To this end we define a near-zero regime, i αpre ∈ [−θ, θ],

(12)

as well as a negative regime, i αpre < −θ.

(13)

For JumpReLU SAEs, the learned offset provides a natural threshold θ; for other SAE types, we use the pragmatic choice θ = 0.1, motivated in Appendix E.1. For lost latents that were previously active, Figure 4 (c) evaluates these regimes at layer ℓ, where the loss is observed. Across the full sequence at this layer, the negative fraction is relatively small, typically 10%–20%, while the near-zero fraction is larger and tends to increase in later layers. Thus, previously active lost latents are not categorically suppressed across the layer-ℓ sequence, nor are they consistently near-zero. On the noun signature, however, the picture is much clearer. Around 80% of latents fall into the near-zero regime, with only few negative latents. This suggests that, at the position where loss is measured, active latents are removed, rather than suppressed by a negative component. For lost latents that were never previously active,

Figure 4 (d) applies the same diagnostic across the upstream trajectory, i.e. over the full content sequence and all layers up to ℓ. These categories test whether a latent is consistently near-zero or consistently negative before the loss point. Both fractions are generally low, indicating that most such latents do not follow a simple pattern of being persistently near zero or persistently in the negative regime. The noun-restricted inset is broadly similar, although the extended sweeps in Appendix E.2 show more pronounced near-zero behavior on the noun for several models and SAE types, especially in middle layers. Overall, binary loss masks a more graded picture: many lost latents are recoverable upstream or show weak rather than negative alignment. Thus, latent disappearance often reflects a local change in the SAE signature rather than a single uniform mechanism of latent removal. 5.4

Qualitative View on Lost Latents

This raises the question of what could be the driver behind these fluctuations in the latent signature. One possibility is that no information is lost in a semantic sense, and that the apparent loss instead reflects geometric interference. Since SAE latents are not exactly orthogonal, adding an adjective may perturb projections onto nearby latent directions, as suggested in Section 2 and by Templeton et al. (2024). However, as discussed in Appendix F, interference alone is unlikely to explain the observed magnitude and structure of lost latents. A second possibility is feature absorption, where the more specific prompt may activate a more specific latent while deactivating a more general one (Chanin et al., 2024). This explanation also appears incom-

plete, as if absorption were the dominant mechanism, we would expect qualitatively different lostlatent rates for Gemma Matryoshka SAEs, which are designed to mitigate feature absorption (Bussmann et al., 2025), compared to other SAE families. Instead, the qualitative trends are broadly consistent across SAE types, as shown in Appendix E.2. We additionally characterize lost latents using the SAEBench auto-interpretability pipeline (Karvonen et al., 2025). As shown in Appendix F.2, lost latents typically receive slightly lower autointerpretability scores than latents active in both t0 and t1 . We interpret this as weak evidence that lost latents are less well captured by current automatic explanations. One possible reason is that they participate in more distributed or context-dependent computations, or encode properties whose relevance is tied more to downstream model behavior than to directly verbalizable input patterns.

6

Related Work

Linear Representation Hypothesis Section 2 introduced a deliberately minimal formulation of the strong LRH. At the same time, a growing body of work suggests that this picture is incomplete (Smith, 2024). In particular, the features recovered by SAEs appear to exhibit geometric structure that is not captured by a simple linear superposition view (Mendel, 2024; Park et al., 2025a). A related line of work has shown that transformer representations can also organize computation along circular (Engels et al., 2025a) and helical (Kantamneni and Tegmark, 2025) structures, motivating the broader perspective that some features may inhabit higher-dimensional manifolds rather than isolated linear directions (Olah and Jermyn, 2024). In addition, Engels et al. (2025b) provide evidence that transformer embeddings may contain a pathologically non-linear component. Feature Composition A cornerstone assumption of mechanistic interpretability is decomposability: Independent features are represented independently within the model (Elhage et al., 2022). This assumption makes internal analysis tractable (Olah, 2022), but how features are organized in practice remains open. Several theoretical frameworks have been proposed for the representational strategies models might use (Olah, 2024), alongside mechanistic accounts of how transformers could implement feature composition through architectural operations (Wattenberg and Viégas, 2024).

Empirical findings on SAEs complicate this picture. SAEs exhibit forms of distributed coding that deviate from the idealized decomposable regime, either as side-effects of the training objective (Chanin et al., 2024) or because recovered features are not atomic representational units (Leask et al., 2025; Bricken et al., 2023). Whether these effects reflect genuine model structure or SAE artifacts remains unsettled. More broadly, it is unclear to what extent a model’s hypothesized “true” features compose, whether SAEs faithfully recover this structure, or whether they introduce spurious feature combinations absent from the model itself (Smith, 2024; Olah and Jermyn, 2024). Sparse Autoencoder Variants SAE architecture design remains an active research area. TopK (Gao et al., 2025) and BatchTopK (Bussmann et al., 2024) enforce sparsity directly through the activation mechanism, avoiding an explicit sparsity penalty. Gated (Rajamanoharan et al., 2024a), ProReLU (Taggart, 2024), and JumpReLU (Rajamanoharan et al., 2024b) SAEs separate interference filtering from latent-magnitude estimation. Matryoshka SAEs (Bussmann et al., 2025) instead use grouped training objectives to capture features at multiple levels of conceptual granularity.

7

Conclusion

We investigated whether the interpretability of individual SAE latents extends to active latent signatures. Preliminary studies show that these sets can be meaningful: SAEs recover union-like compositional structure in controlled toy models, and latent-set overlap induces semantically coherent neighborhoods in natural text. However, in the setup of Shani et al. (2026), SAE set similarities do not align with human category boundaries or typicality judgments better than dense model representations. Moreover, semantically compatible refinements, such as adding an adjective to a noun, can deactivate many previously active latents, violating a simple union-ofproperties expectation. Overall, our results constrain a naive bag-offeatures interpretation of SAE signatures. Active latent sets reveal meaningful structure, but their dynamics are context-sensitive and model-internal, rather than straightforwardly equivalent to human concepts or symbolic semantic feature sets.

8

Limitations

Our analysis covers multiple model scales and SAE variants, from 270M to 27B parameters, with broadly consistent qualitative trends. Still, the conclusions are tied to the datasets, semantic constructions, and SAE families considered here. The typicality experiments rely on human psychology datasets and are therefore limited to their concepts, items, and judgment structure. The lost-latent experiments study one controlled form of semantic refinement, adjective modification of nouns, and may not directly generalize to paraphrases, taskrelevant abstractions, or other forms of semantic consistency. Finally, we focus primarily on residual-stream SAEs and do not exhaustively evaluate other hook locations or SAE families, such as BatchTopK or gated SAEs. Whether the same set-level instabilities hold in these settings remains open. See Appendix G for resources and AI-use disclosure.

References Nikita Balagansky, Ian Maksimov, and Daniil Gavrilov. 2025. Mechanistic permutability: Match features across layers. In The Thirteenth International Conference on Learning Representations. Joseph Bloom, Curt Tigges, Anthony Duong, and David Chanin. 2024. SAELens. https://github.com/ decoderesearch/SAELens. Trenton Bricken, Adina Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Cheng Cheng, Sasha Kravec, Nicholas Schiefer, Tristan Hume, Logan Engstrom, Chris Olah, and 1 others. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread. Bart Bussmann, Patrick Leask, and Neel Nanda. 2024. BatchTopK sparse autoencoders. In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning. Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. 2025. Learning multi-level features with matryoshka sparse autoencoders. In Proceedings of the 42nd International Conference on Machine Learning, ICML’25. E. J. Candes, J. Romberg, and T. Tao. 2006. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theor., 52(2):489–509. David Chanin, Tomáš Dulka, and Adrià GarrigaAlonso. 2025. Feature hedging: Correlated features break narrow sparse autoencoders. Preprint, arXiv:2505.11756.

David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom. 2024. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. In Advances in Neural Information Processing Systems, volume 38. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2024. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations. D. L. Donoho. 2006. Compressed sensing. IEEE Trans. Inf. Theor., 52(4):1289–1306. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Transformer Circuits Thread. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. https://transformer-circuits.pub/ 2021/framework/index.html. Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. 2025a. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations. Joshua Engels, Logan Riggs Smith, and Max Tegmark. 2025b. Decomposing the dark matter of sparse autoencoders. Transactions on Machine Learning Research. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027. https://arxiv.org/ abs/2101.00027. Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2025. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. Gemma 3 technical report. Technical report, Google DeepMind.

Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 278–300, Miami, Florida, US. Association for Computational Linguistics.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. 2024. Sparse crosscoders for cross-layer features and model diffing - october 2024. Transformer Circuits Thread.

Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. 2024. Llama scope: Extracting millions of features from llama-3.1-8b with sparse autoencoders. Preprint, arXiv:2410.20526. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. Subhash Kantamneni and Max Tegmark. 2025. Language models use trigonometry to do addition. In ICLR 2025 Workshop on Building Trust in Language Models and Applications. Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Isaac Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum Stuart McDougall, Kola Ayonrinde, Demian Till, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. 2025. SAEBench: A comprehensive benchmark for sparse autoencoders in language model interpretability. In Forty-second International Conference on Machine Learning. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. 2022. Matryoshka representation learning. In Advances in Neural Information Processing Systems, volume 35, pages 30233–30249. Curran Associates, Inc. Patrick Leask, Bart Bussmann, Michael T Pearce, Joseph Isaac Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. 2025. Sparse autoencoders do not find canonical units of analysis. In The Thirteenth International Conference on Learning Representations. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant

Harry Mayne, Yushi Yang, and Adam Mahdi. 2024. Can sparse autoencoders be used to decompose and interpret steering vectors? In Interpretable AI: Past, Present and Future. Michael E. McCloskey and Sam Glucksberg. 1978. Natural categories: Well defined or fuzzy sets? Memory & Cognition, 6(4):462–472. Callum McDougall, Arthur Conmy, János Kramár, Tom Lieberum, Senthooran Rajamanoharan, and Neel Nanda. 2025. Gemma scope 2 - technical paper. Technical report, Google DeepMind. Jake Mendel. 2024. Sae feature geometry is outside the superposition hypothesis. AI Alignment Forum. Neel Nanda and Joseph Bloom. 2022. TransformerLens. https://github.com/TransformerLensOrg/ TransformerLens. Chris Olah. 2022. Mechanistic interpretability, variables, and the importance of interpretable bases. Transformer Circuits Thread. Chris Olah. 2024. Distributed representations: Composition & superposition. Transformer Circuits Thread. Chris Olah and Adam Jermyn. 2024. Circuits updates july 2024. Transformer Circuits Thread. Bruno A Olshausen and David J Field. 1996. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609. Matthew L. Olson, Musashi Hinck, Neale Ratzlaff, Changbai Li, Phillip Howard, Vasudev Lal, and ShaoYen Tseng. 2025. Probing the representational power of sparse autoencoders in vision models. In 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 6226–6236. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. 2025a. The geometry of categorical and hierarchical concepts in large language models. In The Thirteenth International Conference on Learning Representations. Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org.

Seongwan Park, Taeklim Kim, and Youngjoong Ko. 2025b. Decoding dense embeddings: Sparse autoencoders for interpreting and discretizing dense retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 26468–26485, Suzhou, China. Association for Computational Linguistics. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical Report 8, OpenAI. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah, and Neel Nanda. 2024a. Improving sparse decomposition of language model activations with gated sparse autoencoders. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. 2024b. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. Preprint, arXiv:2407.14435. Fabien Roger. 2023. Some ml-related math i now understand better. LessWrong. Eleanor Rosch. 1975. Cognitive representations of semantic categories. Journal of Experimental Psychology: General, 104(3):192–233. Eleanor H. Rosch. 1973. On the internal structure of perceptual and semantic categories. In Timothy E. Moore, editor, Cognitive Development and the Acquisition of Language, pages 111–144. Academic Press, New York. Chen Shani, Liron Soffer, Dan Jurafsky, Yann LeCun, and Ravid Shwartz-Ziv. 2026. From tokens to thoughts: How LLMs and humans trade compression for meaning. In The Fourteenth International Conference on Learning Representations. Lewis Smith. 2024. The ‘strong’ feature hypothesis could be wrong. LessWrong. Glen Taggart. 2024. Prolu: A nonlinearity for sparse autoencoders. LessWrong. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, and 7 others. 2024. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread. Martin Wattenberg and Fernanda Viégas. 2024. Relational composition in neural networks: A survey and call to action. In ICML 2024 Workshop on Mechanistic Interpretability.

A. D. Wyner. 1967. Random packings and coverings of the unit n-sphere. The Bell System Technical Journal, 46(9):2111–2118.

A

Models and Sparse Autoencoders Used

We evaluate residual-stream SAEs across a set of pretrained language models spanning several model families and parameter scales. Table 1 summarizes the backbone models, SAE suites, dictionary widths, and SAE architectures used in this work. For Gemma 3, we use Gemma Scope 2 SAEs, which are JumpReLU SAEs trained with an additional Matryoshka loss. We report results for all the available sparsity regimes (small, medium, big) of Gemma Scope 2 variants. The public model cards specify target average-L0 ranges of approximately 10–20 for small, 30–60 for medium, and 60–150 for big. Note that medium sparsity models are only available for a limited amount of layers, while small and big sparsity SAEs are available for each layer. For GPT-2 Small and Llama 3.1 8B, we use TopK-style SAE suites, while the Mistral7B SAEs use a ReLU nonlinearity. Table 1: Language models and residual-stream SAEs used in this work. Widths are given in the common shorthand. For Gemma Scope 2, S/M/L denote the small, medium, and large target average-L0 variants. Backbone model Gemma 3 270M, 1B, 4B, 12B, 27B (Gemma Team et al., 2025) GPT-2 Small (Radford et al., 2019) Llama 3.1 8B (Grattafiori et al., 2024) Mistral-7B (Jiang et al., 2023)

SAE suite Gemma Scope 2 (McDougall et al., 2025) OpenAI v5 (Gao et al., 2025) Llama Scope (He et al., 2024) Engels et al. (Engels et al., 2025a)

Width SAE type 16k, 262k Matryoshka BatchTopK; JumpReLU; target-L0 : S/M/L 32k, 128k TopK

32k, 128k JumpReLU

65k

ReLU

Matryoshka SAEs A substantial part of our experimental results, including all figures shown in the main text, use Gemma Scope 2 SAEs. These are Matryoshka SAEs (Bussmann et al., 2025; Kusupati et al., 2022), trained to capture latents at multiple conceptual resolutions within a single run. SAEs trained with a Matryoshka loss, are trained across several nested widths m1 < · · · < mK . For each k, let F (k) denote the first mk activations and (k) Wd the corresponding decoder columns, yielding (k) x̂(k) = Wd F (k) (x) + bd . Training then minimizes a weighted sum of prefix-wise SAE losses, LMat (x) =

K X

  αk ∥x−x̂(k) ∥22 +λk ∥F (k) (x)∥1 .

k=1

(14)

Smaller widths tend to capture coarse, broadly reused latents, whereas larger widths allocate capacity to more specific concepts. The parameter λk controls the sparsity–fidelity trade-off at each width. They are also known to mitigate feature absorption (Bussmann et al., 2025), as the nested-width training objective encourages coarse, reusable latents to be represented already for smaller prefixes.

B

Preliminary Studies

B.1

Union of Properties on LRH Aligned Toy-Model

Motivation and setup. We evaluate SAE compositionality in a controlled LRH-aligned toy setting, following Chanin et al. (2025). The toy model acts as a frozen residual stream simulator in which activations are sparse linear superpositions of known ground-truth feature directions. This gives direct access to the true features and allows us to test whether SAE latent activations compose predictably when multiple features are combined. We define N = {10, 50, 100} ground-truth features in a D = 50 dimensional activation space. The feature directions are the columns of a frozen linear map Wtoy ∈ RD×N , optimized to be approximately pairwise orthogonal and of unit-norm. For a feature coefficient vector a ∈ RN , the toy activation is h = Wtoy a. (15) Each feature fires independently with probability p = 0.2 and unit magnitude when active. We repeated the experiment at both higher and lower sparsity levels and observed qualitatively similar results. Thus, multi-feature activations arise naturally through chance co-occurrence, without imposing an explicit compositional rule. We train standard ReLU SAEs on these activations using the architecture defined Equation (2) and (3). These are trained with reconstruction loss and a norm-scaled L1 sparsity penalty. B.1.1 Union Consistency Evaluation After training, we test whether the SAE latent code satisfies a union property. If two ground-truth features i and j are represented independently, then the signatures for their combined input should approximately match the union of the signatures for the individual inputs. Since the underlying feature directions are approximately orthogonal and the input is their linear superposition, a successful SAE

should resolve the two contributions without suppressing or recruiting additional unrelated latents. To probe this directly, we construct controlled activations from the toy model, simulating LRHaligned residual stream states in which either one or two ground-truth features are active. Let ei denote the one-hot coefficient vector for feature i. Applying the frozen toy-model map to ei isolates the activation induced by feature i alone: hi = Wtoy ei ,

(16)

hj = Wtoy ej ,

(17)

hij = Wtoy (ei + ej ) = hi + hj .

(18)

We then pass each controlled activation through the SAE encoder. Let Fℓ (h) denote the activation of SAE latent ℓ for input h, as defined by the encoder in Eq. 2. Because the encoder uses a ReLU nonlinearity, we define a latent as active whenever its activation is positive: Sx = {ℓ : Fℓ (hx ) > 0},

x ∈ {i, j, ij}. (19)

The expected signature under perfect union consistency is Uij = Si ∪ Sj . We compare the observed set Sij to Uij using |Sij ∩ Uij | , |Sij ∪ Uij | |Sij ∩ Uij | Recallij = , |Uij | |Sij ∩ Uij | Precisionij = . |Sij | Jaccardij =

Figure 5: Jaccard, Recall, and Precision scores, as defined in Equations (20) to (22), for LRH-aligned featurepair combinations, shown for increasing SAE width. Scores are averaged over all ground-truth feature pairs, with semitransparent regions denoting one standard deviation across random seeds. Panels show results for different numbers of ground-truth features: (a) N = 10, (b) N = 50, and (c) N = 100. The red vertical line marks the point where the SAE width equals the number of ground-truth features.

(20) (21) (22)

Recall measures whether expected latents are preserved under composition, while precision measures whether the combined input recruits only expected latents. Thus, low recall indicates lost latents, whereas low precision indicates extra latents. Jaccard similarity summarizes both failure modes in a single set-overlap score. For each trained SAE, we average the metrics over all feature pairs. We then report mean and standard deviation across three independently seeded SAE training runs. B.1.2 Results Results are shown in Figure 5 and Figure 1 in the main text. Overall, all three metrics are high, typically ranging between 0.8 and 1.0. This indicates that, in the controlled LRH-aligned toy setting, the trained SAEs largely preserve the expected union structure of signatures. The strongest, although still moderate, deviation

from union consistency appears for N = 10 ground-truth features, shown in Figure 5 (a). For narrow SAEs, the degradation is mainly recalldriven. Precision remains close to one, while Recall and Jaccard decrease. Thus, the composed input typically does not recruit many unexpected latents, but instead some latents active for the individual features are lost under composition. This suggests that the dominant failure mode is not extralatent recruitment, but a mild suppression of expected latents. A plausible interpretation is that the mild degradation reflects a hedging-like distortion. Since Precision remains close to one, the combined input rarely activates latents outside the expected union. Instead, the lower Recall suggests that some latents active for isolated features are suppressed when features are combined. A plausible explanation might be feature hedging (Chanin et al., 2025), as mixed feature components in capacity-limited latents may cause some expected latents to fall below the ReLU threshold when features are combined. For larger ground truth feature sets N = 50 and N = 100 shown in Figure 5 (b) and (c) respec-

tively, we observe overall higher Jaccard similarity and recall, especially for wider SAEs. For small SAE widths we observe similar effects as for N = 10. Under realistic LRH-style assumptions, the number of underlying features is expected to exceed both the residual stream dimension and the number of available SAE latents (Park et al., 2024; Bricken et al., 2023), making the N = 100 setting the most "realistic" configuration. In this regime, we still observe only mild deviations from set-consistent behavior, indicating that within this controlled linear setting, SAE latent activations mostly satisfy the expected union property. B.1.3 Scaling the Toy-Model Setup To test whether the union-consistency results persist beyond the small D = 50 toy setting, we repeat the same evaluation in a larger activation space with D = 1000. We vary the number of ground-truth features over N ∈ {500, 1000, 5000} and sweep SAE widths L ∈ [50, 1e4 ], as delinearated in Appendix B.1.5. This sweep covers both severely capacity-limited regimes, where L ≪ N , and strongly overcomplete regimes, where L > N . As before, features fire independently with probability p = 0.2. Each SAE is trained on 50M toy activations with three random seeds. For large N , evaluating all feature pairs becomes infeasible, so we estimate the union-consistency metrics using up to 5000 randomly sampled feature pairs per SAE. Results are shown in Figure 6. In the strongly undercomplete regime, union consistency is poor, which can be considered to be expected. When the SAE has far fewer latents than ground-truth features, it cannot allocate independent latents to all directions, and featurepair composition is not reliably resolved. As width increases, however, all three metrics improve sharply. The transition generally occurs around the point where SAE capacity becomes comparable to the ground-truth feature dictionary. However, for larger N = 5e3 , we also observe regimes with L < N in which the SAE already recovers the union property reliably. This is important because LRH-motivated accounts typically assume that, although SAE dictionaries are overcomplete relative to the activation dimension, the number of underlying ground-truth features may be even larger (Park et al., 2024; Bricken et al., 2023). This suggests that set-consistent behaviour does not require a strict one-latent-per-feature allocation, as

Figure 6: Jaccard, Recall, and Precision scores, as defined in Equations (20)–(22), for LRH-aligned featurepair combinations as a function of SAE width in the larger D = 1000 scaling setup. Scores are averaged over randomly sampled ground-truth feature pairs, with semitransparent regions denoting one standard deviation across random seeds. Panels show results for increasing numbers of ground-truth features: (a) N = 500, (b) N = 1e3 , and (c) N = 5e3 . The red vertical line marks the point where SAE width equals the number of ground-truth features. The x-axis is logarithmic.

there may plausibly be a regime of sufficient SAEwidth in which SAEs recover compositional structure despite having fewer latents than ground-truth feature directions. Once the SAE is sufficiently wide, Jaccard similarity, recall, and precision remain high, indicating that the union property can persist in substantially larger settings. These results support the interpretation that the small-scale toy-model findings are not merely artifacts of the low-dimensional setup, but may reflect a more general behaviour of SAEs in controlled LRH-aligned representations. B.1.4 Latent Ground-Truth Features To make the simulation of residual-stream states closer to realistic model geometry, we also sample ground-truth feature directions from the dictionary of a trained Gemma-Scope SAE, whose decoder directions are themselves trained to reconstruct residual-stream states. Specifically, we use the residual-stream SAE of Gemma-3-270M with width 16k at layer 9. D is thereby set to the dimensionality of the model, which is 640 in this

case. From its decoder directions, we greedily select a near-orthogonal subset with pairwise cosine similarity below 0.2, and use the first N selected directions as the ground-truth feature dictionary.6 As before, simulated hidden states are sparse linear combinations of these directions, with each feature firing independently with probability p = 0.2. We then train L1-regularized SAEs on these activations and evaluate the same set-consistency metrics. Evaluation is performed on up to 5e3 randomly

N increases. In this regime, the SAE might have enough capacity to begin differentiating feature directions, but not enough to represent the full dictionary consistently. As a result, composed inputs may suppress latents active for individual features while also recruiting additional latents, leading to reduced performance on the metrics, more comparable to results in Section B.1.3 for capacity limited SAEs. At larger widths, the metrics recover, suggesting that set-consistent behaviour can still emerge under more realistic feature geometry once sufficient capacity is available. B.1.5 Implementation Details Across the toy-model studies above, we evaluate three configurations; a small near-orthogonal setup, a larger scaling setup, and a setup where groundtruth features are sampled from SAE latents using directions from a pretrained Gemma-Scope SAE. Unless stated otherwise, features fire independently with probability p = 0.2, active features have coefficient 1, and SAEs use a reconstruction loss with norm-scaled L1 sparsity. Table 2 summarizes the implementation details. Table 2: Experimental configurations for the LRHaligned toy-model studies. D denotes the activation dimension, N the number of ground-truth features, and L the SAE width.

Figure 7: Jaccard, Recall, and Precision for the unionconsistency evaluation using ground-truth directions sampled from a trained Gemma-Scope SAE decoder dictionary. Scores are shown across SAE widths for increasing dictionary sizes N , with shaded regions denoting one standard deviation across seeds. The red vertical line marks width equal to N and the x-axis is on logarithmic scale.

sampled feature pairs. The results show that union consistency is typically recovered for sufficiently wide SAEs, which alligns essentially with the results from Appendix B.1.3. Interestingly, the metrics can also be high in the strongly compressed regime. This should be interpreted cautiously, as when the SAE is severely capacity-limited, many inputs may collapse onto coarse shared latent patterns, so apparent set consistency need not imply that individual feature directions are faithfully resolved. The strongest failure mode appears at intermediate widths, around SAE width 200, and becomes more pronounced as 6

The threshold of 0.2 is selected pragmatically to obtain a sufficiently large set of directions satisfying this criterion.

Detail

Value

Global setup Feature firing probability SAE objective SAE seeds

reconstruction loss plus norm-scaled L1 penalty {0, 1, 2}

Orthogonal toy model Feature geometry Activation dimension Ground-truth features SAE widths Training activations Evaluation

optimized near-orthogonal directions D = 50 N ∈ {10, 50, 70, 100} L ∈ {8, 10, 12, 15, 20, 30, 40, 50, 70, 100} 15M all feature pairs

Scaling setup Feature geometry Activation dimension Ground-truth features SAE widths Training activations Evaluation

p = 0.2

optimized near-orthogonal directions D = 1000 N ∈ {500, 1000, 5000} L∈ {20, 50, 200, 500, 1e3 , 2e3 , 3e3 , 5e3 , 1e4 } 50M up to 5000 sampled feature pairs

Decoder-sampled setup Feature geometry near-orthogonal directions sampled from a Gemma-Scope decoder dictionary Activation dimension D = 640 Ground-truth features N ∈ {500, 1500, 2500} SAE widths L∈ {20, 50, 200, 500, 1e3 , 2e3 , 3e3 , 5e3 , 1e4 } Training activations 50M Evaluation up to 5000 sampled feature pairs

B.2

Interactive Exploration of SAE Set-Level Structure

To provide intuition for whether joint SAE activations can serve as a meaningful unit of analysis, we complement the controlled evaluations above with an exploratory clustering experiment on natural text. The goal to ask whether texts that activate similar sets of SAE latents also form interpretable groups, placing the signature in the center of analysis instead of individual latents. We sample 30,000 text snippets from The Pile (Gao et al., 2020), truncate each snippet to 100 characters, and prepend the fixed prefix "Fact: ". For each snippet, we run the prompted text through the model and encode the corresponding residual-stream states with a fixed SAE. SAE activations are binarized token-wise, yielding an active/inactive latent signature a(x, t) = 1{F (x, t) > 0} for each input x and token position t. Here F is defined according to Equation (2). A simple union over signatures for each token position in a sequence would typically produce very large sets up to the entire dictionary, making comparisons between longer snippets less nuanced and obscuring which tokens contribute to the similarity. We therefore use a token-level average pairwise maximality score (APM). For two sequences u and v, APM compares each token in u to the best-matching token in v, where token similarity is measured by the intersection size of their SAE signature: APM(u, v) := T

1X max {∥a(u, t) ⊙ a(v, l)∥0 } . T l∈{1,...,L}

(23)

t=1

Here, T and L are the sequence lengths of u and v, ⊙ denotes elementwise multiplication, and ∥ · ∥0 counts the number of shared active latents. Since this score is typically asymmetric, we use the symmetrized version APMsym (u, v) = APM(u, v) + APM(v, u) . 2

(24)

Using this metric, we operationalize the similarity between snippets as follows: On average, each token in one snippet should share many active SAE latents with some best-matching token in the other

snippet. This also preserves token-level information and allows us to identify which parts of a snippet contribute most strongly to the similarity. We use this setup for an interactive cluster browser. After precomputing APM scores for all sampled snippeds pairwise, one can select a seed snippet xi around which a cluster is formed. All snippets xj satisfying APMsym (xi , xj ) ≥ τ

(25)

are assigned to the displayed cluster, where τ is a user-controlled threshold. Higher thresholds produce smaller and typically more tightly related clusters, while lower thresholds reveal broader neighborhoods. For each cluster member, the browser displays a token-level heatmap indicating which tokens have high latent-set overlap with the seed. The user can then select any sampled snipped as a new seed and continue browsing the induced neighborhood structure. We include representative examples for the Gemma3-270M residual-stream SAE with width 16k and big sparsity target at layer 9. To support inspection beyond the selected examples, we provide an interactive browser as supplementary material, allowing readers to explore induced clusters under different seeds and thresholds. This experiment provides qualitative evidence that SAE signatures can expose interpretable structure in natural text. The code and browser will be released upon acceptance. Figure 8 shows representative examples of typical clusters obtained with the latent-set browser. The examples were drawn at random and are representative of the overall quality of clusters. Each line shows a single cluster member and the lines are sorted from highest APM to lowest. We omit the shared "Fact: " prefix for readability.

(a) First ten entries of cluster 14381 obtained with the latent-set browser. The cluster appears to group contexts related to mathematical proofs. Notably, the highest overlap appears to be individual digits. While all sentences share the word "Suppose" this does not meaningfully contribute to the APM.

(b) First then entries of cluster 388 obtained with the latent-set browser. The cluster appears to group contexts related to political actors talking about President Donald Trump. Notably, the highest overlap tokens do not always correspond to "Donald Trump" and often highlight the sentence verb.

(c) First ten entries of cluster 17405 obtained with the latent-set browser. The cluster appears to group contexts related to proper noun disambiguation. Notably, while all samples share the phrase "may refer to" the largest overlap is achieved by the proper nouns themselves, despite being different in each sample.

Figure 8: Representative clusters obtained with the latent-set browser at τ = 25. Each panel shows the first ten members of each cluster. Token-level heatmaps indicate which tokens have high latent signature overlap with the seed snippet.

C

Category-level clustering with AMI.

We ask whether model-induced geometries recover human category boundaries. The human-concepts dataset provides both category labels and typicality scores. Here, we use the labels to test whether items from the same human category also form coherent clusters under model-based similarities. Following the clustering analysis of Shani et al. (2026), we cluster all items into K groups, where K is the number of ground-truth categories, and compare the resulting partition V against the human category partition U . We use Adjusted Mutual Information (AMI), which measures agreement between two partitions while correcting for chance agreement (Shani et al., C 2026). Let U = {Ui }R i=1 and V = {Vj }j=1 be two partitions of N items, with cluster sizes ai = |Ui |, bj = |Vj |, and contingency counts nij = |Ui ∩ Vj |.

Their mutual information is R X C X N nij nij I(U ; V ) = log . N ai bj

(26)

i=1 j=1

For any partition C = {Cj }M j=1 with cluster sizes cj = |Cj |, we write its entropy as H(C) = −

M X cj j=1

N

log

cj . N

(27)

This definition applies to C = U and C = V , respectively. AMI subtracts the expected mutual information E[I(U ; V )] under a fixed-marginal random baseline and normalizes the result: AMI(U, V ) = (28) I(U ; V ) − E[I(U ; V )] . max(H(U ), H(V )) − E[I(U ; V )] Thus, AMI = 1 indicates perfect agreement with human category labels, whereas AMI ≈ 0 indicates chance-level agreement.

For each representation, we convert pairwise similarities into distances d = 1 − s and apply Kmedoids clustering, which operates directly on precomputed distances and is therefore suitable for both cosine- and Jaccard-based similarities. We evaluate four representation families. First, static vocabulary embeddings use cosine similarity between average-pooled raw token embeddings from WE , corresponding to the uncontextualized version of scos in Equation (5) with the layer index dropped. Second, contextual residual-stream states use sℓcos from Equation (5), computed from the prompted template “This is a {word}._”. Third, contextual SAE signatures use the binary target-word signatures from Equation (7) and their Jaccard similarity sℓJac from Equation (8). Finally, uncontextualized SAE signatures are obtained by applying the lowest-layer SAE to raw token embeddings and computing the same Jaccard score. Static embedding and uncontextualized SAE results are summarized in Table 3; layerwise contextual residualstream and SAE results are shown in Figure 9. The static results in Table 3 show weak recovery of human category boundaries. Static embedding AMI is typically around 0.1, indicating that uncontextualized vocabulary embeddings contain minimal human-category-level structure, only weakly recover the human category partition in our setup. This is consistent with findings in Shani et al. (2026). The uncontextualized SAE-Jaccard scores are lower still, suggesting that applying the lowestlayer SAE directly to raw token embeddings does not improve category-boundary recovery over the embedding geometry itself. The contextualized layerwise results in Figure 9 show a clearer category-boundary signal. For residual-stream cosine similarity sℓcos , AMI ranges roughly from near zero to about 0.5, with an overall tendency to increase across layers. Contextual SAE signatures sℓJac show a broadly similar increasing trend, although the detailed curves vary from the embedding related curve. Across SAE types, the layerwise AMI curves are mostly more similar to each other than to the residual-stream cosine curve, suggesting that SAE variants show comparable layerwise agreement with human category labels, but differ more strongly from the corresponding agreement pattern of dense residual-stream similarities. However, the peak AMI values, reported separately in Table 4, show that SAE-based similarities often remain below the corresponding dense residual-

Table 3: Static category-boundary agreement measured by AMI. Static embedding AMI is reported once per model. Static SAE-Jaccard AMI is reported separately for each SAE variant by applying the lowest-layer SAE to raw token embeddings. Model

SAE variant

Emb.

SAE-Jac.

Gemma 3 270M

16k big 16k medium 16k small 262k big 262k medium 262k small

0.1074

0.0294 0.0306 0.0429 0.0555 0.0457 0.0305

Gemma 3 1B

16k big 16k medium 16k small 262k big 262k medium 262k small

0.1222

0.0183 0.0294 0.0229 0.0279 0.0982 0.0085

Gemma 3 4B

16k big 16k medium 16k small 262k big 262k medium 262k small

0.1237

0.0244 0.0155 0.0196 0.0373 0.0734 0.0141

Gemma 3 12B

16k big 16k medium 16k small 262k big 262k medium 262k small

0.1119

0.0136 0.0152 0.0174 0.0443 0.0034 0.0248

Gemma 3 27B

16k big 16k medium 16k small 262k big 262k medium 262k small

0.1019

0.0057 0.0186 0.0089 0.0247 0.0529 0.0208

GPT-2 Small

32k 128k

0.0646

0.0121 0.0213

Llama 3.1 8B

32k 131k

0.0601

0.0227 0.0174

Mistral 7B

65k

0.1070

0.0103

stream state similarities. The gap is smaller than in the static comparison in Table 3 and for the gemma SAEs there are cases where the trend is slightly reversed, but the overall pattern is consistent: SAE latent sets do not outperform dense model representations on the category-boundary task.

D

Typicality Rankings

Below, we discuss the metrics introduced in Section 4 for all investigated models and SAE variants. The results are overall quite robust across models and SAE types, with exceptions discussed onward. Figure 10 shows, for each category, the Spearman correlation between human typicality ratings shum provided by the human typicality dataset (Shani et al., 2026; Rosch, 1973, 1975; McCloskey and Glucksberg, 1978) and uncontextualized cosine similarity vocabulary embedding scores semb , computed from sℓcos in

Figure 9: Layerwise category-boundary agreement for contextualized model representations, measured by AMI. For each layer, pairwise similarities between category instances are converted into distances d = 1 − s, clustered with K-medoids using K equal to the ground-truth number of human categories, and compared against the groundtruth category labels. Results are shown for contextual residual-stream representations using cosine similarity s = sℓcos and for contextual SAE latent signatures using Jaccard similarity s = sℓJac . Higher AMI indicates that the corresponding layerwise representation induces clusters more closely aligned with human category boundaries.

Equation (5) and uncontextualized jaccard based similarity scores on the sae of layer 0. Panel a therefore measures the extent to which the model’s embedding-based similarity structure agrees with the human ranking of how representative an instance is for its category. It shows that the correlations between human typicality judgments and the target-word cosine similarity scores sℓcos are generally weak and overall volatile around zero, depending on the category. "Crime" is standing out as an exception due to its strong anticorrelation for many models and its strong correlation for Llama 3.1 8B, however the overall impression remains. Therefore, these results indicate that the model’s embedding geometry

does not organize category membership in a way that closely matches human typicality judgments, consistent with prior observations in the literature, see (Shani et al., 2026). Interestingly the SAE activations show a comparably diffuse correlation with the human typicality rankings, again with stronger scores for the crime category, however a noisy trend overall, indicating that the structure on the sae latents does not seem to track human typicallity. Panel (c) shows that the scores on the embedding structure align much more closely with sae level analysis as indicated by the overall positive trend. Interestingly, rankings are not equal to one indicating that even though the sae aligns more closely here it decomposition of the latent di-

Figure 10: Spearman rank correlations between human and model-induced typicality rankings. Panel (a) compares human typicality scores shum with the embedding-based typicality score semb , computed from sℓcos in Equation (5). Panel (b) compares Shum with the uncontextualized SAE set-based Jaccard score s0Jac , computed from the layer-0 SAE representation. Panel (c) compares Semb with the same SAE Jaccard score s0Jac . Results are shown across semantic categories for all investigated models. Table 4: Peak layerwise contextualized categoryboundary agreement measured by AMI. Reported values correspond to the maximum AMI across layers for residual-stream cosine similarity and, separately for each SAE variant, the maximum AMI across layers for SAE-set Jaccard similarity. Model

SAE variant

Gemma 3 270M

16k big 16k medium 16k small 262k big 262k medium 262k small

Emb.

SAE-Jac.

0.471

0.421 0.379 0.398 0.463 0.431 0.419

Gemma 3 1B

16k big 16k medium 16k small 262k big 262k medium 262k small

0.488

0.456 0.411 0.415 0.470 0.397 0.391

Gemma 3 4B

16k big 16k medium 16k small 262k big 262k medium 262k small

0.438

0.466 0.412 0.367 0.432 0.410 0.378

Gemma 3 12B

16k big 16k medium 16k small 262k big 262k medium 262k small

0.441

0.461 0.397 0.376 0.409 0.365 0.357

Gemma 3 27B

16k big 16k medium 16k small 262k big 262k medium 262k small

0.397

0.439 0.275 0.385 0.444 0.271 0.359

GPT-2 Small

32k 128k

0.304

0.218 0.217

Llama 3.1 8B

32k 131k

0.429

0.297 0.295

Mistral 7B

65k

0.311

0.289

rections still does something qulaitatively different.

From Figure 11 to Figure 18, each individual figure corresponds to one model, and each row within a figure corresponds to a different SAE type trained for that model. Each panel resolves within-category Spearman rank correlations at different levels of model depth. Across all figures, the columns report complementary rank-correlation analyses. The first four columns show category-wise Spearman correlations, where each score induces a ranking over instances within a category. The first column shows ρS (shum , sℓcos ), i.e. the correlation between human typicality ratings and contextual residual-stream cosine similarity at each layer. The second column shows ρS (shum , sℓJac ), measuring how well human category judgments align with SAE set-based Jaccard similarity. The third column shows ρS (semb , sℓJac ), comparing static embedding cosine similarity with the SAE-induced Jaccard ranking. The fourth column shows ρS (sℓcos , sℓJac ), comparing contextual residual-stream cosine similarity with SAE set-based Jaccard similarity at the same layer. The final column shows the cross-layer consistency of the SAE-based 1 2 scores, i.e. ρS (sℓJac , sℓJac ). This final panel is not category-resolved, but aggregates over the dataset and reports the layer-by-layer agreement between SAE-induced typicality rankings. Since this matrix is symmetric, we display only the diagonal and lower-triangular part of the ℓ1 × ℓ2 matrix. . The comparison between human typicality scores and SAE activation patterns (i.e. the second collumn) is especially relevant. A central motivation in the SAE literature (Templeton et al., 2024;

Figure 11: Layer-dependent Spearman rank correlations for Gemma 3 270M.

Figure 12: Layer-dependent Spearman rank correlations for Gemma 3 1B.

Figure 13: Layer-dependent Spearman rank correlations for Gemma 3 4B.

Figure 14: Layer-dependent Spearman rank correlations for Gemma 3 12B.

Figure 15: Layer-dependent Spearman rank correlations for Gemma 3 27B.

Figure 16: Layer-dependent Spearman rank correlations for GPT-2 Small.

Bricken et al., 2023; Cunningham et al., 2024) is that SAE latents recover interpretable concepts and may therefore provide a representation that aligns more naturally with human-understandable concepts. Under this expectation, one might expect that the correlation between human typicality and SAE set-based Jaccard similarity sℓJac would be substantially stronger than the corresponding result on the hidden state model embeddings, shown in the first collumn in the plot. However as already discussed in Section 4, this is not observed. The correlations between human ratings and SAE set-based Jaccard similarity sℓJac remain weak across models, layers, and SAE types, indicating that SAE activation patterns do not resolve conceptual similarity in a way that is substantially closer to human typicality than the raw model embeddings. By contrast, the third column of plots shows that cosine similarity scores on the raw model embeddings semb and SAE set-based Jaccard scores sℓJac are in most cases qualitatively positively correlated. Thus, although SAE activation patterns do not align well with human conceptualization, they do align closely with the model’s own embedding-based conceptualization structure. GPT 2 and Llama 3.1 8B SAEs are an exception from this trend. Here, we do not observe a consistent trend toward positive rank correlation. Instead, the correlations fluctuate around zero across categories. A trend that holds reliably across SAEs is the strong positive correlation between contextual residual-stream similarity sℓcos and SAE set-based similarity sℓJac . This agreement is partly expected, since the SAE code is a dictionary-based decomposition of the residualstream state at the same layer. However, the result should not be overemphasized, as SAE activations are meant to expose a sparse and more interpretable structure, not simply to reproduce the dense residual-stream geometry. The correlations are therefore informative as they are positive but also not close to one, suggesting that SAE set similarity tracks the model’s residual-stream similarity structure while still measuring something nonidentical. The final column shows cross-layer agreement of SAE set-based Jaccard similarities sℓJac . These correlations are often strong, indicating that SAEinduced similarity rankings are relatively stable across depth. For Gemma Scope 2, the lowersparsity (big) variants typically show higher crosslayer correlations than higher-sparsity variants, suggesting that denser SAE signatures yield smoother,

more layer-stable rankings. Overall, SAE activation sets do not faithfully recover human conceptual typicality. Instead, they more closely track the model’s internal similarity structure, which is known to differ from human judgments (Shani et al., 2026).

Figure 17: Layer-dependent Spearman rank correlations for Mistral 7B.

Figure 18: Layer-dependent Spearman rank correlations for Llama 3.1 8B.

E

Lost Latents

E.1 Choosing a threshold for non-JumpReLU SAEs As discussed in Section 5.3, for a latent that is inactive at a given residual state we distinguish between two cases; it is either not emitted, meaning that its encoder direction is approximately orthogonal to the residual vector, or suppressed, meaning that the corresponding pre-activation in Equation (10) is sufficiently negative (Mayne et al., 2024; Elhage et al., 2021). For JumpReLU SAEs, this distinction is natural, since the learned JumpReLU offset already provides a scale for the decision. For the remaining SAEs, no learned threshold is available, so we introduce an angular tolerance θ for deciding when an encoder direction should count as approximately orthogonal to the residual vector. To motivate this choice, we use a geometric heuristic based on spherical codes (Roger, 2023). Let A(n, ϕ) denote the maximum number of unit vectors in Rn whose pairwise angle is at least ϕ. Known asymptotic bounds (Wyner, 1967) imply that, for fixed ϕ < π/2,
A(n, ϕ) ≈ exp n log(1/ sin ϕ) + o(n) , (29) up to lower-order terms. Strictly speaking, this applies on the unit sphere, whereas SAE preactivations depend on inner products with residual vectors of varying norm. However, for our purposes this mainly introduces a scale factor. The sign of the pre-activation is unchanged, and the qualitative transition between clearly non-orthogonal and approximately orthogonal directions is still governed by the angular term. Since we use the argument only as a heuristic for selecting a pragmatic cutoff, and focus on the location of the “hockeystick” transition rather than on absolute values, this approximation is sufficient. We therefore inspect Λ(ϕ) := n log(1/ sin ϕ), ignoring the lowerorder term, for the residual-space dimensions n relevant to the non-JumpReLU models considered here. The resulting curves exhibit a pronounced “hockey-stick” shape; for very small ϕ, Λ(ϕ) increases sharply, indicating a regime in which extremely many directions may still count as nearly orthogonal, making the distinction between true inactivity and weak alignment highly ambiguous. Beyond this range, the growth becomes less extreme. We therefore choose θ = 0.1. This provides geometric motivation for the choice, while we emphasize that it remains a pragmatic threshold

Figure 19: Value of log Λ(ϕ) = n log(1/ sin θ) as a function of the angle θ, shown for the residual-stream dimensions used by the non-JumpReLU models in our experiments: n = 768 for GPT-2 Small and n = 4096 for both Llama 3.1 8B and Mistral-7B. The dashed vertical line marks the chosen threshold θ = 0.1.

for defining a distinction between deactivated and suppressed in the absence of a learned activation offset. E.2

Additional Results

We report the sweep results for all remaining investigated models and SAE variants, analogous to Figure 4 in the main text. Each figure corresponds to one model, and each row within a figure corresponds to a different SAE type trained for that model. The panel layout follows Figure 4. Panel (a) shows the lost-latent rate R from Equation (11) as a function of the number of prefixed adjectives k, for selected layers across model depth. The inset summarizes the full layerwise trend for k = 1. Panels (b)–(d) focus on k = 1, which is representative because these quantities depend only weakly on k across the investigated models. See Appendix E.2.1 for investigating k-dependence. Panel (b) considers latents that are active for the base prompt t0 but lost on the noun signature of the more specific prompt at layer ℓ. It reports the fraction of these lost latents that were previously active upstream when probing the full content sequence with the SAE of layer ℓ; the inset restricts the same upstream check to noun signatures only. See Section 5.2. Panel (c) analyzes the previously active subset from panel (b), distinguishing whether the corresponding layer-ℓ pre-activations fall into the near-zero regime from Equation (12) or the negative regime from Equation (13). The main plot evaluates this distinction over the full

sequence at layer ℓ, while the inset restricts it to the noun signature. Panel (d) applies the same near-zero/negative distinction to the complementary subset of lost latents that were never previously active under the upstream probing criterion. The main plot evaluates whether these latents remain near-zero or negative throughout the full upstream computation, while the inset restricts the evaluation to noun signatures. Together, panels (b)–(d) distinguish whether lost latents were previously recoverable, and whether their binary loss reflects near-threshold inactivity or stronger negative pre-activation. A shared observation across all investigated models (Figures 20 to 27) is that the lost-latent rate R increases with the number of prefixed adjectives k, typically by roughly 20 percent over the five investigated construction steps. Interestingly, this increase is not linear, as the slope becomes noticeably less pronounced beyond k = 2. A possible interpretation, is that the model may distinguish between a noun modified by only a few adjectives and a qualitatively different regime resembling an extended adjective enumeration. Once the latter regime is reached, the precise number of additional adjectives may become less relevant. This interpretation is further suggested by Appendix E.2.2. However, this interpretation should be understood as suggestive rather than conclusive. A second observed trend is the increase of R with model depth, shown in the inlay plots of collumn a), for the simplest deviation from the base prompt, i.e. at k = 1. Across cases, the lost-latent rate approaches a plateau around 50% in deeper layers. from the main curves in column (a), one can further extrapolate this profile to shift upward by roughly another 20 percent as k increases, which has been verified but is not explicitly shown here. Therefore, the loss rate is consistently smaller in earlier layers, which are commonly associated more strongly with lexical processing (Elhage et al., 2021; Geva et al., 2022). In this regime, the SAEs appear to adhere to a more set-consistent behaviour between the compared inputs. However, at greater computational depth the lost rate R increases, suggesting that the model transitions toward a different internal conceptualization of the more specific prompt, indicating that the SAE might capture additional structure in higher layers

not explained by a "bag-of-features"-view on the dictionary latents. . The results in panel (b) show a structured dependence on model family and depth. For the Gemma SAEs and GPT-2 Small, the fraction of lost latents that had been active upstream generally rises from early layers to a maximum in the middle layers, typically around 50%–80%, then decreases again and often increases once more toward the final layers. The initial increase is partly expected, as for deeper layers, there are more preceding layers in which a latent could have appeared. This makes the subsequent decrease more noteworthy, since it indicates that lost latents in later layers are not simply more likely to have been active before. One possible interpretation is that different phases of the model computation use a greater variety of latent repertoires, so that later-layer losses are not always recoverable as earlier activations of the same SAE directions. Llama 3.1 8B shows a different pattern, with the previously-active fraction remaining comparatively stable at roughly 40% across layers. Mistral-7B reaches relatively high values of roughly 80%, indicating that the lost latents remain relevant in the computation. However, SAEs for only few layers are available, so we avoid drawing strong conclusions about the layerwise trend. The main panel measures upstream activity over all token positions in the full prompt sequence. The inset repeats the analysis using only noun signatures in previous layers. For most Gemma SAEs, the inset closely tracks the main curve, suggesting that when a lost latent was previously active, it was usually active on the noun signature at a previous layer rather than only elsewhere in the sequence. GPT-2 Small and Llama 3.1 8B are partial exceptions. In early layers, the nounrestricted curves are substantially lower than the full-sequence curves, indicating that upstream activity of lost latents is less tightly coupled to the noun position in these models. The plots in column (c) further decompose the previously active lost latents according to their layer-ℓ pre-activations. For most Gemma SAEs, the fraction that is negative over the full sequence is non-zero but typically small, while a larger fraction falls into the near-zero regime. The precise layerwise trend varies across model sizes and SAE variants, but the inset shows a

Figure 20: Lost features under semantically consistent inputs for Gemma 3 270M.

clearer pattern. When the analysis is restricted to the noun signature, the negative fraction remains low, whereas the near-zero fraction is often very high. Thus, for Gemma SAEs, many previously active lost latents show weak rather than negative alignment at the noun position. A broadly similar pattern holds for Llama 3.1 8B, where noun-restricted near-zero rates are substantially higher than the corresponding negative rates. Mistral-7B is the strongest exception, as for both over the full sequence and on the noun signature, almost all previously active lost latents are classified as negative, suggesting that when such latents are lost, they are more strongly removed from the representation. GPT-2 Small shows no comparably clear pattern under this classification, as neither the near-zero nor the negative regime accounts for a substantial fraction of previously active lost latents. Finally, column (d) analyzes lost latents that were never previously active under the upstream probing criterion. For Gemma SAEs, the fraction that is always negative across the full upstream computation is typically low. By contrast, the fraction that remains within the near-zero regime is often larger, frequently increases with layer depth. The inset restricts the same analysis to noun signatures and often strengthens this trend,

particularly in earlier layers. This suggests that many latents classified as never previously active are not persistently negatively aligned throughout the computation. On the noun position, they instead often show only weak alignment with the current residual-stream state. For Llama 3.1 8B, the main panel shows a similar tendency to the Gemma SAEs. The always-negative fraction remains low, while the near-zero fraction accounts for a larger share of never-previously-active lost latents. The noun-restricted inset further increases this fraction, suggesting that several of these latents show weak rather than clear negative alignment on the noun position. GPT-2 Small shows no clear separation, as both fractions remain small. Mistral-7B shows the clearest contrast, as in panel (c), lost latents mostly fall into the negative regime, indicating stronger negative rather than weak subthreshold alignment.

Figure 21: Lost features under semantically consistent inputs for Gemma 3 1B.

Figure 22: Lost features under semantically consistent inputs for Gemma 3 4B.

Figure 23: Lost features under semantically consistent inputs for Gemma 3 12B.

Figure 24: Lost features under semantically consistent inputs for Gemma 3 27B.

Figure 25: Lost features under semantically consistent inputs for GPT-2 Small.

Figure 26: Lost features under semantically consistent inputs for Mistral 7B.

Figure 27: Lost features under semantically consistent inputs for Llama 3.1 8B.

E.2.1 Sensitivity on k In Figure 4 and Figures 20–27, panels (b)–(d) are shown for k = 1. To verify that this choice is representative, we repeat the same analyses for all (k) available construction steps k. Let qm,s,l denote one of the measured quantities for model m, SAE variant s, layer ℓ, and construction step k. We define its k-sensitivity as the layer-averaged range across k: ∆k (q; m, s) =  X  1 (k) (k) (30) max qm,s,l − min qm,s,l , k k |Lm,s | l∈Lm,s

where Lm,s is the set of layers for which the corresponding SAE variant is available. This yields one k-sensitivity value per measured quantity, model, and SAE variant. Lower values indicate that the quantity depends only weakly on the number of added adjectives. Figure 28 summarizes these values for the quantities used in panels (b)–(d). Panel (a) corresponds to the upstream-activity analysis from panel (b), panel (b) to the near-zero/negative decomposition of previously active lost latents from panel (c), and panel (c) to the same decomposition for neverpreviously-active lost latents from panel (d). Main panels show the full-sequence analyses, while insets show the noun-only variants. Overall, the ksensitivity values remain moderately small compared to the layerwise trends observed in the figures, indicating that varying k does not qualitatively change the conclusions. E.2.2

Stepwise Evaluation for Lost-Latent Rates So far, we defined a latent as lost if it was active on the base noun prompt t0 but inactive on a more specific prompt tk , using the neutral prompt template from Section 5. Here, we extend this analysis to

consecutive refinement steps. That is, for each step i, we measure latents that are active on the noun signature of ti but inactive on the noun signature of ti+1 . The rest of the setup remains unchanged. The comparison is made on the noun token positions under the same prompt template. Formally, we define the stepwise lost-latent rate as (i→i+1)

Rl

=

F l (ti ) \ F l (ti+1 ) F l (ti )

,

(31)

thus asking to which extend additional adjectives alter the latent signature. Results are shown in Figure 29. The lost-latent rate remains substantial across refinement steps. However, the largest drop typically occurs from t0 to t1 , i.e. when the representation first changes from a bare noun to an adjective noun phrase. Later refinements still induce losses, but these are usually smaller. This possibly indicates, that the first adjective introduces a qualitatively larger representational shift, whereas subsequent adjectives are incorporated into an already modified noun-phrase representation. Thus, once the model has moved from an isolated noun to description prescribed with adjectives, further refinements may be somewhat easier to accommodate without replacing as much of the SAE signature. The inset shows the layerwise trend for the final consecutive refinement, t4 → t5 .

Figure 28: Sensitivity of the quantities in panels (b)–(d) of Figure 4 and Figures 20–27 to the number of added adjectives k. For each measured quantity q, model, and SAE variant, we compute the layer-averaged range across k, ∆k (q; m, s), as defined in Equation (30). Each point corresponds to one model–SAE pair. Similar to the layout from Figures 20–27, main panels show the full-sequence analyses, while insets show the corresponding noun-only analyses.

Figure 29: Stepwise lost-latent rates for consecutive semantic refinements. Each row corresponds to one model, and panels within a row show the SAE variants evaluated for that model from left to right. For each construction step i, we measure the fraction of latents active on the noun signature of ti that are no longer active on the noun signature of ti+1 , as defined in Equation (31). The inset reports the layerwise trend for i = 4, i.e. the transition t4 → t5 , since the analogous layerwise trend for the first transition t0 → t1 is already shown in Figure 4 and Figures 20–27 (a).

from randomly sampled SAE latent directions. To obtain progressively more complex deviations, we build these perturbations cumulatively. At step k, we add k sampled latent directions fi , each weighted by a random coefficient αi drawn uniformly from an interval centered at µ with width β. The resulting perturbation is rescaled to match the norm of the original residual activation, so as to remain on a comparable scale. Formally, t̃k =

k X i=1

Figure 30: All panels (a)–(c) show the lost rate from Equation (11) for steering-based constructions from the base case t0 as the number of steering steps k increases according to Equation (32). Four layers across model depth are shown. Panel (a) uses only positive coefficients (µ = β/2), panel (b) uses equally likely positive and negative coefficients (µ = 0), and panel (c) uses only negative coefficients (µ = −β/2). Each construction uses up to 2000 latents. Results are aggregated over 50 random steering runs per dataset instance and across the dataset. The width of the distribution is chosen to be β = 10. The inset plots show additionally the trend through model depth at k = 1500 in the construction.

F

Interference

When assuming that the observed violation of set consistency was actually noise, one possible explanation is that the model combines several positively contributing SAE latents in such a way that they result in a negative projection onto another latent direction previously active, as made plausible by Templeton et al. (2024). In that case, some latents would appear absent not because the model explicitly suppresses them (Mayne et al., 2024; Elhage et al., 2021), but as a byproduct of superposition. To probe this, we simulate controlled perturbations of the residual stream using SAE-resolvable directions. For each base prompt t0 , we first extract the residual-stream activation at layer (l) ℓ, denoted Vres , at the target token position. Specifically, t0 is inserted into the template also (l) used above "This is a {t0 }._", and Vres is taken from the last token position of the inserted noun. We then add synthetic perturbations constructed

(l)

αi fi ,

(l) Tk = Vres + t̃k

∥Vres ∥ . (32) ∥t̃k ∥

Here, fi denotes a SAE latent direction from the SAE decoder matrix trained on layer ℓ. Note that this is in accordance with the LRH formulation in Equation (1), as the bias term is implicitly involved (l) in Vres . For each dataset example, we generate 50 random steering trajectories from the base prompt t0 and compute the lost-latent rate from Equation (11) by comparing t0 with t̃k . Reported values are averaged first across trials and then across dataset instances. Throughout, we use a uniform coefficient distribution with width β = 10, a pragmatic choice that induces latent interactions on a meaningful scale and is adequate for the qualitative analysis considered here. This construction serves as a qualitative probe of the origin of latent loss. In particular, it tests whether the observed lost-latent rates from Section 5.1.1 can already be reproduced by interference among positively contributing SAE latents. The perturbation results suggest that positive-latent interference, i.e. µ = β/2, may be one contributing mechanism behind the lost-latent rates observed under real prompts. In particular, Figure 30 (a), and more broadly the results across models in Appendix F.1, show that purely positive steering can already induce loss rates of an order observed under real input constructions from Section 5 for some layers. This is qualitatively consistent with the idea that adding concepts, i.e. moving in positive latent directions, can nevertheless produce interference among active latents, so that some previously active latents are no longer recovered by the SAE. This remains compatible with the set-consistency intuition from Section 5. The more specific prompt may indeed recruit additional latents while preserving the content of the less specific prompt, yet the resulting superposition can still induce latent loss as a side effect. At the same time, this mechanism does not explain the empirical loss patterns in a

sufficiently consistent way across layers and models when compared to the real-input results from Section 5.1.1 and Appendix E.2. This can be red of the inset plots, which show the trend across model depth at step k = 1500 in the construction, which do not align with the trend that the observed lost rate tends to increase with model depth. We also test perturbations with mixed-sign coefficients, i.e. µ = 0, and purely negative coefficients, i.e. µ = −β/2, to assess whether adding negative contributions increases the explanatory power of the construction. Empirically, however, these settings do not yield a clearer match to the observed loss patterns and often induce weaker effects instead. See Figure 4 (b)–(c) and Appendix F.1. As a qualitative side observation, this is in line with the linear representation hypothesis introduced in Section 2, according to which semantic content is primarily encoded by positive latent directions (Park et al., 2024; Elhage et al., 2022; Engels et al., 2025a). Overall, the perturbation experiments suggest that positive-latent interference is a plausible contributing factor, but does not offer a sufficient explanation of the lost-latent rates observed under real inputs. F.1

Additional Results

Similar to Figure 30, the following figures report the results of the latent steering experiment described above for all remaining models and SAE types. Each figure corresponds to one of the investigated models, and each row of panels within a figure corresponds to an SAE type trained on that model. The three columns represent different variants of the experiment. Column (a) shows the steering setup with µ = β/2, that is, using only positive coefficients in the linear construction. Column (b) shows the balanced setting with µ = 0, corresponding to a mixture of positive and negative latents. Column (c) shows the setting µ = −β/2, in which only negative coefficients are used. The inset displays the full layer-wise trend at step k = 1500. From Figure 31 to Figure 35, we observe for the Gemma SAEs that relevant loss rates arise primarily for positive coefficients (column (a), and mainly in the early and middle layers, as shown by the inset plots. By contrast, for mixed (column (b) and purely negative coefficients (column (c), the loss rates remain generally low. As discussed above, this is consistent with the view

that semantic content is encoded primarily in the positive latent direction. Under this interpretation, interference from positive latent projections may contribute to the observed loss in early-to-mid layers and could therefore partially explain the effect reported in Section 5, although this remains speculative. At the same time, this account does not explain the layerwise trend itself. For the other models, this pattern does not hold. For GPT-2 (Figure 36), Llama-3.1-8B (Figure 38), and Mistral-7B (Figure 37), all three coefficient settings yield high loss rates, with steep increases as k grows. In these cases, interference of various kinds could in principle explain the loss rates observed in Appendix E.2. However, results there show that the loss rates on real input data are of similar magnitude across models. Since the steering results differ qualitatively across models, it seems unlikely that this proposed interference effect is the dominant underlying cause. Still, the Gemma results may be seen as more compatible with LRH, insofar as negative latents do not substantially alter the pattern of previously active latents, suggesting an interpretation that meaning is encoded mainly along the positive latent direction, more aligned with interpretations of the strong LRH (Park et al., 2024; Elhage et al., 2022). Why SAEs for the other models behave differently in this respect, and what this implies for LRH more generally, remains an open question for future work.

Figure 31: Feature steering results for Gemma 3 270M.

Figure 32: Feature steering results for Gemma 3 1B.

Figure 33: Feature steering results for Gemma 3 4B.

Figure 34: Feature steering results for Gemma 3 12B.

Figure 35: Feature steering results for Gemma 3 27B.

Figure 36: Feature steering results for GPT-2 Small.

Figure 37: Feature steering results for Mistral 7B.

Figure 38: Feature steering results for Llama 3.1 8B.

Figure 39: Auto-interpretability scores for lost, surviving, and control latents across models and SAE variants. Each row corresponds to one model, and panels within a row show the SAE variants evaluated for that model from left to right. Within each panel, bars report mean auto-interpretability scores for the three latent groups at representative stages of model depth, approximately 25%, 50%, 65%, and 85% of layers. Error bars indicate one standard deviation across evaluated latents. Lost latents are those active for the base prompt t0 but inactive for the composed prompt t1 ; surviving latents remain active in both prompts; control latents are randomly sampled from the SAE dictionary.

F.2

Auto-interpretability of Lost Latents

We additionally ask whether latents lost under composition are less interpretable than latents that remain active. For each noun–adjective pair, we compare the base prompt t0 with the composed prompt t1 and classify latents into three groups: Lost latents, active on t0 but inactive on t1 ; surviving latents, active in both prompts; and randomly sampled control latents. For each group and layer, we evaluate up to 100 latents. We follow the SAEBench auto-interpretability protocol (Karvonen et al., 2025). For each latent, cached SAE activations are used to collect activating examples. During explanation generation, GPT-4o-mini receives the surrounding context with the activating token marked by API-call and produces a natural-language description of the latent. During scoring, the model receives the explanation and held-out sequences, consisting of true activating examples and random non-activating examples, and predicts which sequences contain an activation. The score is the resulting sequence-level classification accuracy. We compute one score per latent and average these scores within the lost, surviving, and control groups for each layer. Latents are discarded if, after masking all top-k activation positions, no remaining activation exceeds 10−6 ; such latents effectively fire only in the top-k region and cannot provide disjoint importance-weighted samples. We therefore cache more than 100 candidate latents per group to retain 100 evaluated latents after filtering. Results across investigated models and SAE types are shown in Figure 39. We report autointerpretability scores for lost, surviving, and control latents at four representative depths, approximately 25%, 50%, 65%, and 85% through the model. The groups occupy a similar score regime, with means often within each other’s error bars, so the results should not be read as a strong separation. Still, lost latents show a weak recurring tendency toward the lowest mean scores. Note, this does not suggest these latents are therefore less relevant and thus discarded by the model under the more specific prompt. It suggests that latents lost under semantic refinement may be somewhat less well captured by current automatic explanations, possibly because they participate in more distributed or context-dependent computa-

tions. This provides weak supporting evidence for the more nuanced view on these latents, also suggested by the pre-activation analyses above.

G

Resources and AI Use

Compute resources. All experiments were performed on a node with four NVIDIA RTX 3090 GPUs. Training the SAEs for the toy-model experiments in Section B.1 required approximately one day of compute. The latent-set clustering experiment in Section B.2 required approximately three hours. The human-concepts experiments following Shani et al. (2026) required approximately one day of compute. The lost-latent experiments in Section 5 and Appendix E.2–F.1 required approximately six days in total. The auto-interpretability analysis in Appendix F.2 required approximately two additional days. Preliminary experiments required approximately one further week of compute. Datasets and software resources. For the toymodel experiments, we build on the repository and SAE-training infrastructure of Chanin et al. (2025), adapting their training setup to our LRHaligned synthetic activation data. For the naturaltext clustering experiment, we sample snippets from The Pile (Gao et al., 2020). For the conceptual-similarity experiments, we use the human-concepts data from Rosch (1973, 1975); McCloskey and Glucksberg (1978) and orient our evaluation procedure around the experimental setup from Shani et al. (2026). The semantic construction experiments were implemented within our own experimental pipeline, no additional external datasets or task-specific code resources are used. For the auto-interpretability analysis, we use the SAEBench auto-interpretability pipeline (Karvonen et al., 2025). All experiments involving pretrained models and SAEs were implemented using SAELens (Bloom et al., 2024) and TransformerLens (Nanda and Bloom, 2022). The pretrained language models and SAE suites used in this work are listed and credited in Appendix A. All external artifacts are used under their publicly available licenses or access terms and only for research purposes. AI use. We used generative AI tools for language refinement, plotting-code assistance, and coding support during implementation checks and debugging. AI-generated suggestions were used only as auxiliary support. The authors remained respon-

sible for the experimental design, implementation decisions, result interpretation, scientific claims, and final manuscript, and reviewed and validated all submitted content. No generative AI system is listed as an author.

Related documents

• Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects#405649
• Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders#229479
• LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling#282798
• MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference#238616
• Why Are LLM Backdoor Defenses Fragmented? A Feature-Level Explanation with Sparse Autoencoders#622439