Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects Phu Gia Hoang∗
UKP Lab, Technical University of Darmstadt
Anwoy Chatterjee∗
Department of Electrical Engineering Indian Institute of Technology Delhi, India
Tanmoy Chakraborty
Department of Electrical Engineering Indian Institute of Technology Delhi, India
Iryna Gurevych
arXiv:2607.24645v1 [cs.LG] 27 Jul 2026
UKP Lab, Technical University of Darmstadt
Subhabrata Dutta†
UKP Lab, Technical University of Darmstadt
Abstract The wide-scale use of sparse autoencoders (SAEs) as interpretability tools is limited by inconsistent links between SAE features and model behavior. Features with clear activation descriptions may have weak or unexpected causal effects; steering can vary across prompts or oppose the intended direction; and activation-based feature selection can miss features that produce the desired output change. Prior work has studied feature geometry inside the model, where features are computed. We instead study the geometry of changes in model logits caused by feature interventions. We introduce Feature-Effect Geometry Analysis (FEGA), an unsupervised framework that removes the same active SAE feature across contexts and analyzes the resulting cloud of logit changes. Across SAE variants, consistent one-dimensional effects are rare: few features behave like reusable directions. To interpret this variation, we distinguish value-like features, tied to static information such as factual attributes, from pointer-like features, associated with context-dependent operations. Valuelike features more often exhibit structured, low-dimensional effects, although these effects typically span several directions. Pointer-like features, by contrast, predominantly exhibit diffuse effects. Our results show that a feature can be interpretable and causally relevant without providing a stable direction for steering.
1
Introduction
Sparse autoencoders (SAEs) map model activations into overcomplete sparse latent spaces whose latents1 are often more interpretable than individual neurons (Bricken et al., 2023; Huben et al., 2024). In mechanistic interpretability (Geiger et al., 2025), however, activation evidence alone does not establish causal contribution. A causal claim requires intervening on the feature and measuring the resulting change in the model. Such interventions underlie SAE feature steering (Templeton et al., 2024; Durmus et al., 2024) and appear in benchmark suites such as SAEBench (Karvonen et al., 2025). This raises a direct empirical question of ∗ These two authors contributed equally to this work. † Corresponding author. 1 In the context of SAEs, we use the terms latent and feature interchangeably in this paper.
1
Prompt
Frozen model tail + logit readout geometry
: Paris is in the country of
LLM
layer
…
layer
last layer
SAE reconstruction baseline:
enc
…
Shared SAE Feature Active in both contexts; ablated to compare downstream e�ects.
…
layer +1
dec
Feature-ablated SAE reconstruction:
LLM
layer
Prompt
…
layer
…
layer +1
last layer
: [{"city": "Tokyo", "country": " Feature E�ect Geometry Analysis (FEGA)
E�ect of ablating feature in context
Compare directional change across contexts
E�ect of ablating feature in context
Figure 1: SAE feature steering should be evaluated by downstream effect geometry. Two contexts, ca and cb , activate the same feature j at layer ℓ. Green paths show the SAE reconstruction baseline; red paths show the feature-ablated reconstruction. Both are propagated through the frozen model tail and logit (ca ) (cb ) readout map ϕ(·), producing one downstream logit effect per context, ∆j,logit and ∆j,logit . feature-effect consistency: how comparable are a feature’s downstream effects when that feature is intervened on across contexts? Existing workflows associate SAE features with concepts or behaviors through highly activating examples, automated descriptions, decoder-based readouts, and benchmark-specific criteria (Bricken et al., 2023; Huben et al., 2024; Karvonen et al., 2025; Arad et al., 2025). These signals can identify what a feature appears to represent, but not how modifying it propagates through the rest of the model. Prior work shows that steering effects can vary across inputs and prompt formulations, sometimes even opposing the intended direction (Tan et al., 2024; Braun et al., 2025); SAE interventions can produce behavioral effects not predicted by activation contexts (Durmus et al., 2024); and activation patterns alone may miss features with the desired output effect (Arad et al., 2025). These gaps point to a missing measurement: computable diagnostics of how intervention effects are organized, distinguishing reusable effect geometries, structured low-dimensional effect spaces, and diffuse effect clouds that resist low-rank explanation. What counts as a comparable effect depends on what remains invariant in a feature’s role across contexts. We borrow the analogy of values and pointers from programming semantics to distinguish between feature roles. In some cases, a feature is tied to relatively context-agnostic information; we call this kind of role value-like. For example, a feature associated with the concept of Paris may be relevant whether the prompt asks for the capital of France, discusses European cities, or compares tourist destinations, because its role is tied to the same underlying information. Such a feature might be expected to produce similar downstream effects across these contexts. In other cases, the invariant is not the information itself but the function the feature supports; we call this kind of role pointer-like. For example, in a copying or rule-following setting, a feature may support the same operation while acting on values supplied by the context. 2 However, prior work does not explore whether such patterns hold for outputs. The key implication is that a feature can 2 Existing SAEs report similar patterns on the input side. For example, https://www.neuronpedia.org/gemma-2-9b-it/ 20-gemmascope-res-131k/78155 demonstrates a feature that activates for every token after a punctuation.
2
make a coherent causal contribution without corresponding to a single shared logit direction, motivating our question of whether its effects exhibit a reproducible geometric structure when the same intervention is repeated across contexts. We study this distinction with tasks that vary whether the model retrieves stable pretrained information or follows prompt-local operations. On the value-like end, RAVEL city-country evaluates attribute disentanglement, as implemented in SAEBench, where features are selected through entity-attribute interventions (Huang et al., 2024; Karvonen et al., 2025). Conversely, synthetic tasks such as Literal Sequence Copying and Word Content rely on randomly sampled tokens to test pattern matching, while PrOntoQA uses fictional predicates and prompt-specified rules to test prompt-local rule completion; these give pointer-like settings that operate without fixed token meanings (Niu et al., 2025; Saparov & He, 2023). Bridging these settings, Token Translation combines an in-context translation format with pretrained lexical associations, serving as a mixed setting (Niu et al., 2025). To evaluate feature behavior across these tasks, Figure 1 illustrates our core intervention method. For a (c) (c) given context c, let hℓ be the activation at the SAE site, with encoding z (c) = E(hℓ ) and reconstruction (c) (c) (c) h̄ℓ = D(z (c) ). We define the reconstruction after ablating feature j as ĥℓ,j↓ = D(z (c) − zj ej ), where ej is (c)
(c)
the jth SAE basis vector. We use h̄ℓ , rather than the original activation hℓ , as the baseline so that the (c) (c) two paths differ only in whether feature j is retained. Comparing ĥℓ,j↓ directly with hℓ would additionally include SAE reconstruction error, confounding feature removal with the replacement of the original activation by its reconstruction. We run the reconstructed baseline and the feature-ablated reconstruction through the frozen model tail and compare their downstream logit readouts: (c)
(c)
(c)
∆j,logit = ϕ(ĥℓ,j↓ ) − ϕ(h̄ℓ ). (c)
The set Ej,logit = ∆j,logit : c ∈ Cjvalid is the feature’s downstream logit-effect cloud, with the valid-context set defined formally in Section 5.1. The local decoder geometry fixes the immediate reconstruction change caused by removing feature j, but it does not determine how that change propagates through later attention blocks, MLPs, residual streams, and final normalization. The intervention-relevant question is, therefore, geometric: what shape does Ej,logit have in the logit-readout space? To this end, we introduce Feature-Effect Geometry Analysis (FEGA), an offline, unsupervised framework for measuring and classifying downstream effect clouds 3 . FEGA treats each feature’s intervention effects as a cloud rather than a single steering object. It first tests whether the effects share one dominant readout direction; if not, it asks how they vary across contexts: through magnitude changes, sign flips along an axis, multiple directional modes, low-dimensional spans, or diffuse structure. We apply FEGA to residual-stream SAEs trained on the post-layer-12 residual stream of Gemma-2-2B (Lieberum et al., 2024) and released in the SAEBench baseline suite (Karvonen et al., 2025). We compare ReLU (Bricken et al., 2023), TopK (Gao et al., 2025), and Matryoshka Batch TopK (Bussmann et al., 2025) variants of width 65k across value-like factual-attribute interventions, pointer-like copying and rule-following tasks, and a mixed token-translation setting. Across this evaluation spectrum, downstream feature effects are substantially more context-dependent than a single-vector interpretation suggests. Our main findings are: • Feature effects rarely collapse to a single direction (Section 6). Across architectures and feature populations, most downstream effect clouds do not reduce to a context-invariant logit vector, suggesting that simple linear steering is the exception rather than the rule. • Pointer-like behavior is associated with a compact, partially shared set of SAE features (Sections 4.3 and 4.4). Across copying, binding, rule application, and translation tasks, most isolated features are task-specific, but a small number of features recur across multiple tasks. This cross-task reuse is more pronounced in TopK and Matryoshka Batch TopK than in ReLU SAEs, while the pointer-like sets remain disjoint from the RAVEL value-like feature sets. 3 Project repository: https://github.com/UKPLab/FEGA
3
• Pointer-like features produce predominantly diffuse downstream effects (Section 6.1). Despite their recurrence across successful task executions, features associated with prompt-local operations rarely act as reusable logit directions. Their effects scatter across output space, consistent with the output target changing with the value being copied, bound, or translated. • Value-like features have more structured downstream effects (Section 6.2). Compared with pointer-like effects, factual-attribute effects more often show low-dimensional organization. This structure typically spans multiple output directions rather than collapsing to a single steering vector. These findings challenge a common assumption in feature-level interpretability: that a latent with a clear description and measurable causal effect can serve as a reliable explanatory or control variable. Prior automated-interpretability work has emphasized quantitative scoring of explanations, including human comparisons, while showing that activation descriptions can miss downstream functions and out-of-distribution behavior (Bills et al., 2023). FEGA extends this perspective from activation description to intervention effect by asking whether a feature’s causal effects form stable geometry across contexts. Our results suggest this stability is often limited: an SAE feature can be interpretable and causally important while lacking a consistent logit-space direction. Pointer-like features show strong context dependence and predominantly diffuse downstream effects. Value-like features show low-dimensional organization more frequently, but their structured effects typically span several output directions rather than forming a single steering vector. Together, these results suggest that feature-level interpretability should distinguish what a feature detects, whether it matters causally, and how its effects vary across contexts. FEGA provides this missing effect-side audit.
2
Related Work
In this section, we restrict the discussion to the literature related to geometry, steering, and semantic multiplicity of SAE features. For a comprehensive overview of SAEs in general, refer to Shu et al. (2025) and Sharkey et al. (2025). Geometry of SAE Features. Uncovering the geometric properties of SAE features has garnered interest recently. One of the foundational works in this direction, by Hindupur et al. (2025), investigates the structural assumptions associated with SAE architectures and how concepts are encoded in model representations. Li et al. (2025) show that SAEs organize concepts differently across scales, while SAEs have also been used to uncover belief-state geometries (Levinson, 2026). Bhalla et al. (2026) show that SAEs can capture manifolds either globally, through groups of atoms whose span contains the manifold, or locally, through features that tile restricted regions. Our work, though related to this line of research in terms of geometric interests, focuses on a different object altogether: effects of SAE features in the downstream logit space. Steering Model Behavior with SAE Features. Prior studies have identified inconsistencies in the role of SAE features in predicting downstream behaviors. Li et al. (2026) observe that even tiny adversarial input perturbations can manipulate concept representations in SAEs. Korznikov et al. (2026) investigate SAEs with frozen (randomly initialized) encoders and decoders; they demonstrate causal editing and sparse probing performance comparable to fully-trained SAEs, consequently questioning the ability of SAEs to learn meaningful features. Wu et al. (2025) highlight that SAE features are unreliable for steering the model behavior, often outperformed by simpler baselines. On the other hand, Arad et al. (2025) show that the steerability is dependent on feature types; they identify two distinct types of SAE features: input features, which mainly capture patterns in the model’s input, and output features, which have a humanunderstandable effect on the model’s output. While most prior attempts in this broad direction uncover many interesting success/failure modes of SAE features, they do not characterize the geometric properties of these features that cause such peculiarities. Polysemanticity of SAE Features. A representation (e.g., a feature or a neuron) is polysemantic when it is associated with multiple unrelated concepts. The original goal of SAEs in mechanistic interpretability is to extract monosemantic features from polysemantic representations of the model (Bricken et al., 2023). 4
Minegishi et al. (2025) propose an evaluation strategy based on polysemous words to determine how good an SAE is in extracting monosemantic features, ultimately highlighting the limitations of most prevalent SAE architectures. The theoretical analysis laid out by Cui et al. (2026) reveals that SAEs fail to fully recover the ground-truth monosemantic features unless the features are extremely sparse. Mencattini et al. (2026) shed light on this problem from a rate-distortion perspective, showing that forcing the SAE to learn monosemantic features comes at the cost of higher reconstruction error and lower sparsity, ultimately leading to a trade-off. While the degree of semantic multiplicity is fundamental to the what is represented by a feature in the concept level, it does not answer the subsequent question of what will happen if a certain feature is strengthened/weakened. Our work, to the best of our knowledge, is the first of its kind to characterize the geometric properties of SAE features in the effect space, tying together the missing pieces of feature geometry, steering and semantic multiplicity.
3
Experimental Setup
This section defines the experimental setting for our downstream geometry analysis. We first specify the base model and SAE variants, then introduce the task spectrum used to sample value-like, pointer-like, and mixed feature populations. Feature isolation itself is described separately in Section 4. 3.1
Model and SAE Configurations
We perform all experiments using Gemma-2-2B (Gemma Team, 2024), analyzing SAEs applied to the postlayer-12 residual stream. To check that our conclusions are not artifacts of a single sparsity mechanism, we evaluate three SAE variants from the SAEBench Gemma-2-2B suite (Karvonen et al., 2025): ReLU, TopK, and Matryoshka Batch TopK. Each SAE has a width (i.e., dictionary size) of 216 , i.e., ∼ 65k. 3.2
The Task Spectrum
We use five tasks spanning synthetic in-context operations to factual attribute retrieval. For each of the four in-context learning (ICL) tasks, we construct 50,000 examples balanced across 1,000 prompt families, giving 50 retained queries per family. We retain only examples with single-token targets for which the unmodified Gemma-2-2B model predicts the correct target as its first answer token. A prompt family denotes a fixed template, demonstration context, or prompt-local mapping, while a query denotes one completion instance generated from that family. This sampling design helps distinguish features associated with reusable promptlocal operations from features associated with incidental tokens in individual prompts. Literal Sequence Copying (LSC). LSC (Niu et al., 2025) is the purely synthetic endpoint of our spectrum. Each prompt family samples a random token pattern P , inserts a target token T after its first occurrence, and repeats P later. The query asks the model to output the token that followed the earlier occurrence of the same pattern. For example: attorney impair georgia all berry enlarged takes any dispute ... return attorney impair georgia all berry
has answer enlarged. Since sampled words are arbitrary, solving LSC requires a minimal pointer-like operation: find the previous occurrence of this pattern and copy what came next. Word Content (WC). Word Content (Niu et al., 2025) keeps a synthetic vocabulary but adds a classification structure. Each prompt family assigns labels to sets of trigger tokens, and each query contains one trigger set plus distractors. For example, a family may define: produce + smell → find, A query such as: 5
names + drop → our.
agreed age miscellaneous reduced might meantime force smell produce
has answer find. WC, therefore, tests the pointer-like operation of identifying prompt-local evidence and emitting the associated label, without relying on semantic shortcuts. PrOntoQA. We adapt PrOntoQA (Saparov & He, 2023) to construct prompt-local rule-completion tasks with fictional predicates and newly generated entity names. Each prompt family provides demonstrations associating source predicates with target predicates. A query introduces a new entity with one of the demonstrated source predicates and asks the model to complete the corresponding rule: Q: Sam is a storpist. Every storpist is a stopin. A: stopin ... Q: Siomdu is a storpist. Every storpist is a A:
Sam is a
with answer stopin. Because predicates such as storpist and stopin are fictional, success depends on retrieving the demonstrated source-to-target predicate binding from context rather than relying on world knowledge. We therefore treat PrOntoQA as a pointer-like task. Token Translation (TT). Token Translation (Niu et al., 2025) combines prompt-local task specification with pretrained lexical knowledge. Each prompt family fixes a source and target language and gives several translation demonstrations. A query then asks for a new translation in the same format: Translate English words into German. English: chair German: Stuhl English: ring German: Ring ... English: apple German:
with answer Apfel. TT is not purely synthetic: the model must follow the translation schema established by the prompt, while the final answer depends on pretrained lexical associations. We therefore treat TT as a hybrid task combining pointer-like schema following with value-like lexical retrieval. RAVEL. RAVEL (Resolving Attribute-Value Entanglements in Language Models), proposed by Huang et al. (2024), forms the semantic endpoint of our spectrum. Unlike the ICL tasks, RAVEL does not provide a prompt-local mapping. Its instances are organized by entity, queried attribute, and natural-language prompt template. We analyze the city entity class, treating Country as the target attribute and attributes such as Language and Continent as non-target controls. A base query such as: The city of Paris is located in the country of
has expected completion France, and a source query such as: The city of Tokyo is located in the country of
has expected completion Japan. RAVEL asks whether transferring the relevant internal representation from the source can make the model answer Japan for the Paris query while preserving the base entity’s non-target attributes, such as its Language or Continent. Thus, factual recall provides the underlying task, while the intervention evaluates whether the target attribute can be causally isolated from other factual attributes. We use the resulting country-associated features as the value-like endpoint of our spectrum because their outputs depend on pretrained factual associations rather than a prompt-local mapping. 6
Feature Source LSC WC PrOntoQA TT RAVEL (City-Country)
Regime
ReLU SAE
TopK SAE
Matryoshka Batch TopK SAE
Pointer-Like Pointer-Like Pointer-Like Hybrid Value-Like
5 25 78 27 12,900
5 9 22 14 2,828
4 14 20 16 3,654
Table 1: Identified feature sets. Number of features selected for each task across Gemma-2-2B SAE variants of width 65k.
4
Isolating Value-Like and Pointer-Like Features
The task spectrum in Section 3.2 ranges from prompt-local operations to factual retrieval. We use this spectrum to construct and compare two task-conditioned feature populations: value-like features associated with pretrained factual attributes, and pointer-like features recurrently associated with prompt-local operations. 4.1
Identifying Value-Like Features
To isolate value-like features, we use the RAVEL attribute-disentanglement setup through SAEBench. SAEBench applies differential binary masking (Chaudhary & Geiger, 2024) over SAE latents to find the smallest feature subset needed for an interchange intervention. Because these features are selected to control localized semantic attributes, they are natural value-like candidates. Starting from 5,000 base records, differential binary masking selects 12,900 ReLU, 2,828 TopK, and 3,654 Matryoshka Batch TopK features. Requiring at least eight active contexts leaves 7,715, 1,167, and 1,750 features, respectively, for downstream geometric analysis. 4.2
Identifying Pointer-Like Features
To identify pointer-like candidates, we analyze the model-correct examples from the four ICL tasks described in Section 3: LSC, WC, PrOntoQA, and TT. At the final token before prediction, we consider an SAE feature active if its activation is positive. We retain a feature if it is active on at least 90% of all examples and on at least 90% of the queries in at least 90% of prompt families. These thresholds select features that are active across many prompts and outputs, rather than only for specific words, queries, or prompt families. This procedure selects between 4 and 78 features for each task–architecture pair; Table 1 reports the full counts. 4.3
Feature Discovery Across SAE Architectures
Table 1 shows how selected set sizes vary across tasks and SAE architectures. No architecture consistently yields the largest set: ReLU identifies the most candidates for WC, PrOntoQA, and TT, whereas all three architectures produce similarly small sets for LSC. The number of selected features is not, by itself, a measure of how well an architecture captures task-related computations. Feature activation patterns and sparsity differ across architectures, making the raw counts difficult to compare directly. We therefore examine cross-task overlap to determine whether features are shared across tasks and use ablation to measure their contribution to task performance. Takeaway 1 Pointer-like recurrence is concentrated in small feature sets across architectures. The number of selected candidates varies across tasks and SAE variants, but no architecture uniformly dominates functional feature discovery.
7
0.0
LSC
1.00
WC
0.2
0.07
0.01
0.03
0.00
LSC
1.00
0.13
0.04
0.00
WC
1.00
0.07
0.00
PrOntoQA
1.00
0.00
TT
1.00
RAVEL
PrOntoQA
TT
RAVEL
C
LS
0.4
C
W
On Pr
QA
to
TT
L VE RA
1.00
C
LS
(a) ReLU
0.6
0.8
0.40
0.17
0.27
0.00
LSC
1.00
0.19
0.21
0.00
WC
1.00
0.24
0.00
PrOntoQA
1.00
0.00
TT
1.00
RAVEL
A
C
oQ nt
W
O Pr
(b) TopK
TT
L VE RA
1.00
1.0
0.20
0.14
0.18
0.00
1.00
0.31
0.36
0.00
1.00
0.29
0.00
1.00
0.00
1.00
C
LS
A
C
oQ nt
W
O Pr
TT
L
VE RA
(c) Matryoshka Batch TopK
Figure 2: Cross-task feature overlap. Intersection over Union (IoU) of the selected feature sets. Several features recur across the ICL tasks, with greater overlap for TopK and Matryoshka Batch TopK than for ReLU. None of the ICL-selected features overlaps with the corresponding RAVEL feature set.
4.4
Shared Features Across In-Context Tasks
If particular SAE features contribute to general ICL and induction-like behavior, they should recur across tasks despite differences in vocabulary, format, and output structure. We therefore compare the selected feature sets across the four in-context tasks. Figure 2 shows clear evidence of such reuse. For TopK, the largest overlap is between LSC and WC, with an IoU of 0.40. For Matryoshka Batch TopK, WC overlaps with TT at 0.36 and with PrOntoQA at 0.31. ReLU exhibits less cross-task reuse, with a maximum IoU of 0.13 between WC and PrOntoQA. Because the selected sets are small, these overlaps point to a compact group of features shared across multiple in-context tasks. This shared component may support computations common to ICL and induction-like behavior, while the remaining features may reflect task-specific demands such as input structure, formatting, or output type. Having identified this shared and task-specific structure, we next use ablation in Section 4.5 to test whether the selected feature sets causally affect task performance. For all three SAE architectures, no features selected from an ICL task appears in the RAVEL City–Country feature set. Under our selection procedures, features associated with prompt-local operations are therefore distinct from those associated with factual attribute disentanglement. This result applies to the selected candidate sets and does not imply that the entire SAE dictionary separates cleanly into pointer-like and value-like regions. Takeaway 2 Some SAE features recur across distinct in-context tasks. Their reuse points to a compact shared component underlying ICL and induction-like behavior, while the selected ICL features remain distinct from the RAVEL-selected factual-attribute features.
4.5
Causal Contribution via Ablation
Recurrence and overlap establish an association with prompt-local behavior, but not a causal contribution. We therefore jointly ablate each selected feature set at the final prompt position and measure the resulting task accuracy. The intervention preserves the SAE reconstruction error: we encode the original residualstream activation, zero the selected SAE coordinates, decode the modified representation, and add the original reconstruction error before patching it back into the model. 8
Task
k
LSC WC PrOntoQA TT
5 25 78 27
LSC WC PrOntoQA TT
5 9 22 14
LSC WC PrOntoQA TT
4 14 20 16
Targeted Ablation Random Control ptarget Accuracy (in %) Accuracy (in %) ReLU SAE 95.7 < 10−300 98.3±0.4 76.5 < 10−300 93.4±4.0 71.3 < 10−300 94.4±1.4 89.9 < 10−300 98.7±0.5 TopK SAE 65.1 < 10−300 95.7±1.1 18.4 < 10−300 93.6±4.0 47.1 < 10−300 95.7±1.8 72.9 < 10−300 98.2±0.6 Matryoshka Batch TopK SAE 64.3 < 10−300 95.8±2.1 19.9 < 10−300 94.7±1.0 17.9 < 10−300 92.1±4.6 66.0 < 10−300 97.5±2.4
prandom 5.3 × 10−15 3.0 × 10−7 1.1 × 10−13 1.6 × 10−10 1.7 × 10−13 4.7 × 10−7 1.1 × 10−9 9.0 × 10−12 1.6 × 10−8 6.8 × 10−16 1.5 × 10−7 9.2 × 10−5
Table 2: Causal effects of identified ICL features. Accuracy after jointly ablating the k selected features at the final prompt position. Results are evaluated on examples answered correctly by the unablated model. Random controls ablate same-sized feature sets matched by activation prevalence and magnitude; values report mean±standard deviation across 20 trials. ptarget is obtained from a one-sided exact paired McNemar test, and prandom from a one-sided one-sample t-test across the random trials.
For comparison, we run 20 control ablations. Each control removes an equally sized set of other features chosen to have similar activation frequency and average activation strength. As shown in Table 2, targeted ablation lowers accuracy in all 12 task–architecture combinations. The effect ranges from a 4.3 percentagepoint drop for ReLU on LSC to an 82.1 percentage-point drop for Matryoshka Batch TopK on PrOntoQA. In every case, the targeted drop is larger than the mean drop across the matched random controls. For example, ablating the 9 TopK WC features reduces accuracy from 100% to 18.4% (p < 10−300 ), whereas the matched random controls retain an average accuracy of 93.6% ± 4.0. These results show that the selected feature sets make a specific causal contribution to task performance beyond what is expected from ablating similarly active feature sets of the same size. They do not imply that every selected latent is individually necessary. Takeaway 3 Selected pointer-like feature sets contribute causally to task performance. Jointly ablating them produces consistently larger drops in accuracy compared to same-sized matched random controls.
4.6
The Need for Downstream Geometric Analysis
The selection, overlap, and ablation results establish task-conditioned feature populations with different functional associations. RAVEL identifies features involved in factual attribute disentanglement, while recurrence and ablation identify small feature sets that are reused across prompt-local tasks and causally affect task performance. These results do not show that a feature’s downstream effects are stable across contexts. A feature may activate reliably across copying prompts, yet its ablation may perturb different logit directions depending on the prompt structure or copied token. Its effect cloud may align with one direction, separate into contextdependent regimes, or remain diffuse in vocabulary space. Thus, after grouping features by the behaviors through which they were isolated and establishing their causal relevance, we ask how their downstream effects vary across contexts. This motivates the geometric analysis that follows. 9
5
Feature-Effect Geometry Analysis (FEGA)
To address the need for downstream geometric analysis established in Section 4.6, we introduce FeatureEffect Geometry Analysis (FEGA). FEGA studies the downstream readout change produced by ablating an active SAE feature and patching the resulting reconstruction back into the model. Across contexts, these effects form an empirical cloud in logit space. FEGA analyzes this cloud to determine whether a feature acts as one consistent steering direction or instead follows another geometric pattern, summarized by a diagnostic profile and a primary geometry label. 5.1
Downstream Feature Effects
To construct each effect cloud, we define an intervention relative to the SAE reconstruction. We first specify this reconstruction-relative baseline, then formalize feature zeroing, downstream readout effects, valid contexts, and the sense in which downstream effects can depart from the local decoder direction. Contexts and SAE Reconstruction. A context c determines both the prompt and the task-specific (c) analysis position. Let hℓ ∈ Rdmodel be the original model activation at the SAE input site in layer ℓ. The SAE encoder and decoder are E : Rdmodel → Rdsae and D : Rdsae → Rdmodel , giving: (c)
(c)
z (c) = E(hℓ ) ∈ Rdsae ,
h̄ℓ = D(z (c) ) ∈ Rdmodel . (c)
All feature-ablation comparisons below are reconstruction-relative: the baseline path patches h̄ℓ , rather (c) than the original activation hℓ , so that the compared paths differ only in whether feature j is retained and are not confounded by SAE reconstruction error. (c)
Feature-Zeroing Intervention. For feature j ∈ {1, . . . , dsae }, define the active context set Cj = {c : zj 0}. On an active context, feature-zeroing removes only coordinate j from the SAE code: (c)
(c)
(c)
zj↓ = z (c) − zj ej ,
>
(c)
ĥℓ,j↓ = D(zj↓ ), (c)
(c)
where ej ∈ Rdsae is the jth feature basis vector. We then patch h̄ℓ and ĥℓ,j↓ into the same SAE site and target position in separate forward passes, so the two paths differ only in whether feature j is retained in the reconstruction. (c)
Frozen Tail and Logit Readout. Let rℓ : Rdmodel → Rdmodel denote the frozen model tail from the SAE site in layer ℓ to the vector presented to the output embedding at the target position, with patched activation u inserted at that site. For Gemma-2-2B, this vector is taken after the model’s final normalization. The superscript (c) records that the surrounding prompt is fixed, while the model weights remain unchanged. With unembedding matrix WU ∈ R|V|×dmodel , the corresponding linear logit readout, before the output soft cap, is (c)
(c)
ϕℓ (u) = WU rℓ (u) ∈ R|V| . Removal Effects. The canonical FEGA object is the logit-space removal effect: (c)
(c)
(c)
(c)
(c)
∆j = ϕℓ (ĥℓ,j↓ ) − ϕℓ (h̄ℓ ) ∈ R|V| . (c)
Thus, ∆j is always oriented as ablated readout minus reconstruction-baseline readout. A positive coordinate (c)
of ∆j
means that the corresponding token logit increases after feature j is removed; the opposite direction,
(c) −∆j , describes what the retained feature contributed before removal. Appendix A gives the sign-orientation
details.
10
Rather than storing a vocabulary-dimensional vector for every context, we retain the corresponding removal effect at the input to the output embedding, (c)
δj (c)
(c)
(c)
(c)
(c)
= rℓ (ĥℓ,j↓ ) − rℓ (h̄ℓ ) ∈ Rdmodel . (c)
Its induced linear-logit effect is ∆j = WU δj . Thus, FEGA measures geometry in linear-logit space while computing its magnitudes and inner products from the smaller stored vectors. Effect Clouds and Valid Contexts. For feature j, FEGA analyzes the finite empirical cloud of valid (c) logit removal effects Ej = {∆j : c ∈ Cjvalid }. We retain a context only when the two readout vectors and their difference are finite and the induced linear-logit effect is larger than numerical zero. Specifically, n o (c) Cjvalid = c ∈ Cj : ∆j is finite and non-zero . (c)
(c)
We validate this implementation on a held-out subset where both δj and explicit ∆j are materialized. The cloud Ej is not the full distribution of all possible contexts; it is the sampled set of valid effects under the chosen data set, feature-selection rule, target position, reconstruction baseline, and removal-effect orientation. Why Downstream Effects Are Needed? For a linear SAE decoder with decoder matrix WD ∈ Rdmodel ×dsae , zeroing one feature gives the local reconstruction perturbation: (c)
(c)
(c)
(c)
ĥℓ,j↓ − h̄ℓ = D(z (c) − zj ej ) − D(z (c) ) = −zj WD,:,j . This local decoder-removal term has a fixed direction scaled by the feature activation. The measured effect (c) ∆j , however, is defined after the patched activation passes through later attention blocks, MLPs, residual additions, final normalization, and the unembedding. These downstream computations can make the logit removal effect context-dependent even when the local decoder-removal direction is fixed. The central question is, therefore, geometric: does the sampled logit-effect cloud Ej behave like one stable directed removal effect, or does it vary across contexts in a structured way? Later in this section, FEGA equips these effects with logit-space geometry, separates magnitude from direction, and tests whether a feature forms a directed ray, an axis-like pattern, multiple context regimes, a low-dimensional effect span, or a diffuse cloud. 5.2
Logit-Induced Effect Geometry
We formalize the geometry used throughout the FEGA diagnostics. Each removal effect is decomposed into a magnitude and a direction; directions are compared through a shared kernel; and the corresponding logit-space quantities are computed from pre-logit effects using the unembedding Gram matrix. Magnitude-Direction Split. All FEGA diagnostics are defined on the downstream logit-effect cloud. nj (ci ) Let Ej,logit = {∆j,logit }i=1 again denote the retained logit-removal effects for feature j, where c1 , . . . , cnj enumerate the valid contexts. For each retained effect, define its logit magnitude and normalized direction as: (ci ) ∆j,logit (i) (ci ) (i) (i) ∥2 , vj = mj = ∥∆j,logit , ∥vj ∥2 = 1. (i) mj (i)
The normalized directions vj
capture the shape of the effect cloud after removing variation in effect mag-
(i) nitude. The magnitudes mj are used later as evidence-strength and magnitude-heterogeneity information.
Shared Directional Kernel. FEGA summarizes directional agreement with a context-by-context kernel. Each entry in this kernel asks how similar two retained normalized effects are under the induced logit (i) geometry. Let Vj ∈ Rnj ×|V| be the matrix whose ith row is vj . The pairwise directional similarities are: 11
(i)
(k)
(Kj )ik = ⟨vj , vj ⟩2 ,
Kj = Vj Vj⊤ ∈ Rnj ×nj .
Positive entries indicate that two removal effects point in similar logit directions, negative entries indicate opposing directions, and entries near zero indicate little directional agreement. Thus, Kj is the shared object used by the directional diagnostics below. The non-zero eigenvalues of Kj are the squared singular values of Vj , giving the standard dual-PCA view of the normalized logit-effect cloud (Schölkopf et al., 1998). The axis, span, residual, and effective-rank diagnostics all reuse this kernel or its centered version. Appendix D gives the full derivation. (c)
(c)
Pre-Logit Gram Computation. Since ∆j,logit = WU ∆j,pre , the same logit inner products can be computed by pre-logit inner products without explicitly materializing expensive vocabulary-dimensional logit vectors. Define the unembedding Gram matrix: G = WU⊤ WU ∈ Rdmodel ×dmodel . For any two valid contexts ci and ck , (c )
(c )
(c )
(c )
i i k k ⟨∆j,logit , ∆j,logit ⟩2 = (∆j,pre )⊤ G∆j,pre .
Therefore, the same magnitudes and kernel entries are computed directly using pre-logit effects as: (i)
mj =
q
(c )
(c )
(c )
i i (∆j,pre )⊤ G∆j,pre ,
(Kj )ik =
(c )
i k (∆j,pre )⊤ G∆j,pre
(i)
(k)
.
mj mj
This is only a computational shortcut: the geometry remains the Euclidean geometry of the corresponding logit effects. Appendix B gives the validation checks and failure cases. 5.3
Geometry Diagnostics
Starting from the normalized effect directions, FEGA tests progressively different geometric explanations: one signed direction, one unsigned axis, several directional modes, a shared low-dimensional span, and lowdimensional residual variation around the mean. Directed-Ray Concentration. The first directional question is whether the normalized effects behave like repeated observations of the same directed removal movement. Here, agreement means pointing in the same direction: opposing effects should lower the score rather than be treated as the same pattern. FEGA summarizes this alignment using the mean off-diagonal similarity in the directional kernel: Cray (j) =
1
X
nj (nj − 1)
i̸=k
(Kj )ik .
High Cray means that most retained removal effects point along a common logit direction, supporting the directed-ray family shown in Figure 3. P (i) (i) Equivalently, let Sj = i vj . Since each vj is unit-normalized in logit space, Cray (j) =
∥Sj ∥22 − nj . nj (nj − 1)
Appendix C gives the trace-corrected Gram form and derives this identity. Low Cray is ambiguous: it may arise from antipodal, multimodal, low-dimensional, or diffuse structure. Negative values indicate substantial cancellation or opposition. Thus, directed-ray concentration tests one geometric hypothesis: do these removals repeatedly point in the same direction? 12
Main Geometry Labels Directed Ray
Global Span Spectrum low-dimensional subspace
residual disagreement is low dimensional
Feature 57176 | ReLU SAE
Feature 16392 | Matryoshka Batch TopK SAE
Feature 47789 | TopK SAE
most directions repeat one effect Normalized directions
2D class view
Normalized directions
Centered Residual Spectrum
2D class view
Normalized directions
2D class view
Secondary Flags & Fallback Long-Tail Spectrum Flag
low-energy tail may complicate rank estimation
Magnitude Heterogeneity Flag
effect strength varies across contexts
e.g. Feature 37457 of TopK SAE 𝐶𝑉𝑚 = 1.31; 𝑚10% = 116.70; 𝑚90% = 1837.27
(𝟏)
𝑪𝒓𝒂𝒚 = 𝟎. 𝟖𝟕; 𝑺𝒔𝒑𝒂𝒏 = 𝟎. 𝟖𝟕;
𝑬𝒓𝒆𝒔 = 𝟎. 𝟏𝟑;
𝒏 = 𝟔𝟒
(𝟐)
(𝟐)
(𝟐)
𝑺𝒔𝒑𝒂𝒏 = 𝟎. 𝟗𝟎 𝑼𝒔𝒑𝒂𝒏 = 𝟎. 𝟑𝟔 𝑫𝒔𝒑𝒂𝒏 = 𝟎. 𝟏𝟒
𝒏 = 𝟏𝟑
𝑬𝒓𝒆𝒔 = 𝟎. 𝟓𝟗
(𝟐)
𝑺𝒓𝒆𝒔 = 𝟎. 𝟖𝟑
𝒏 = 𝟑𝟎
Fallback: One Dimensional Diffuse strong 1D evidence but fails the others
Axis or Antipodal
Directional Mixture
Unresolved Diffuse
both signed sides are populated
several directed modes
no simple geometry detected
Feature 62802 | TopK SAE
Feature 21437 | ReLU SAE
Feature 58529 | TopK SAE
Normalized directions
2D class view
Normalized directions
2D class view
Normalized directions
Feature 623 | Matryoshka Batch TopK SAE Normalized directions
2D class view
(𝟏)
𝑪𝒓𝒂𝒚 = 𝟎. 𝟕𝟗 𝑺𝒔𝒑𝒂𝒏 = 𝟎. 𝟖𝟏 𝟏
𝑪𝒓𝒂𝒚 = −𝟎. 𝟎𝟑; 𝑺𝒔𝒑𝒂𝒏 = 𝟎. 𝟖𝟏; 𝑩𝒂𝒙𝒊𝒔 = 𝟎. 𝟒𝟐; 𝒏 = 𝟏𝟐
𝒎𝒊𝒏 𝚫𝐦𝐢𝐱 = 𝟎. 𝟒𝟓; 𝝅𝒎𝒊𝒏 = 𝟎. 𝟏𝟕; 𝑪𝒓𝒂𝒚 = 𝟎. 𝟕𝟑; 𝜿𝒎𝒊𝒏 = 𝟗. 𝟏𝒆 + 𝟓; 𝒏 = 𝟏𝟖
2D class view
𝑩𝒂𝒙𝒊𝒔 = 𝟎 𝒏 = 𝟏𝟖
(𝟏)
𝑪𝒓𝒂𝒚 = 𝟎. 𝟎𝟑; 𝑺𝒔𝒑𝒂𝒏 = 𝟎. 𝟎𝟐; 𝒓𝒔𝒑𝒂𝒏 = 𝟏𝟕. 𝟕; 𝒓𝒄𝒕𝒓 = 𝟏𝟕. 𝟏; 𝒏 = 𝟔𝟒
Figure 3: Overview of FEGA geometry labels and secondary flags. Each card shows a representative downstream logit-effect cloud after removing one SAE feature. The left panel visualizes normalized effect directions; the right panel gives a two-dimensional view for visualization only. The six main cards illustrate the strict geometry families and the unresolved outcome, rather than the complete reporting vocabulary. The orange panel shows two secondary qualifications and the one-dimensional diffuse fallback. Partial span and residual fallbacks, together with the three non-geometric terminal outcomes, are defined in Section 5.5 and Appendix H. The printed statistics are the diagnostics used by the tests. Axis or Antipodal Structure. A low directed-ray score may occur when effects share one axis but use opposite orientations. Contexts near u and −u cancel under signed similarity even though the cloud remains one-dimensional. To capture this structure, let the leading eigenpair of Kj satisfy: Kj q1 = λ1 q1 ,
λ1 ≥ λ2 ≥ · · · .
The leading eigenvector q1 assigns weights to the retained contexts. Combining these weights with the sampled effect directions recovers the leading axis in logit-effect space: Vj⊤ q1 u1 = √ . λ1 + ϵ The corresponding signed projection scores are: Kj q1 s= √ , λ1 + ϵ (i)
with si = ⟨vj , u1 ⟩2 up to the numerical safeguard. This is the dual-PCA recovery step: diagonalize the context kernel, then map the leading component back to a logit-effect axis. Appendix D gives the full derivation and the λ1 > 0 condition. The sign of u1 is arbitrary, so only the occupancy of both orientations matters. We measure it using the axis-split fraction: Baxis,j =
min{|{i : si > 0}|, |{i : si < 0}|} . nj 13
Axis evidence requires more than a non-zero split: it combines strong one-dimensional structure, weak directed-ray evidence, and non-negligible occupancy on both sides of the axis. This supports the axis or antipodal family shown in Figure 3. Directional Mixture Structure. Some clouds contain several coherent directions rather than one ray or one sign-split axis. An axis contains directions near u and −u on the same line, whereas a directional mixture contains prototype directions that need not be collinear or antipodal. (i)
The mixture diagnostic models the normalized logit directions vj with a von Mises-Fisher mixture, a distribution for unit vectors on a Euclidean sphere (Banerjee et al., 2005). Because the likelihood is defined on the logit sphere, this diagnostic uses the normalized logit directions themselves (as defined in Section 5.2). For M directional modes, the mixture density is: (i)
p(vj ) =
M X
(i) πr C|V| (κr ) exp κr µ⊤ , r vj
r=1
where Cd (κr ) is the vMF normalizing constant, µr is the prototype direction for mode r, κr measures how tightly contexts PM concentrate around that prototype, and πr is the fitted probability mass of mode r, with πr ≥ 0 and r=1 πr = 1, describing how much of the effect cloud is associated with that directional regime. A context may support several modes to different degrees and is assigned to its best-matching mode only for mode-specific summaries. (Banerjee et al., 2005). Candidate mode counts are selected using the Bayesian information criterion (BIC) (Schwarz, 1978). Bayesian information criterion (BIC) makes the proposed multi-mode explanation pay for its extra prototype directions, concentration parameters, and mixture weights. Accepted mixtures must contain more than one mode. Each mode must satisfy the fitted-mass requirement πr ≥ 0.10, and each within-mode concentration must be defined from at least two hard-assigned contexts. The modes must additionally exhibit sufficient within-mode coherence and improve coherence relative to the pooled cloud. The improvement is summarized by:
∆mix =
M X
πr Cray,r − Cray,global .
r=1
Here, πr describes how much probability mass the fitted mixture assigns to mode r, while Cray,r measures the directional coherence of the contexts whose best-matching mode is r; Cray,global is the corresponding concentration of the full cloud. A positive ∆mix indicates that the proposed modes are more coherent than the full cloud. The remaining gates require valid concentration estimates and stable assignments under resampling. Appendix F gives the complete fitting and model-selection protocol, the evaluation of the high-dimensional normalizer, and the assignment-stability calculation. An accepted mixture supports multiple directional regimes in the sampled logit-effect cloud, but does not by itself establish behavioral polysemanticity. The modes may instead reflect task regimes, prompt templates, token positions, or other context variables. Global Span Spectrum. If the cloud is not explained by a ray, axis, or directional mixture, the global span tests whether all normalized effects lie within a low-dimensional logit-effect subspace. Unlike a mixture, which separates contexts into local directions, the global span summarizes the full cloud with one shared subspace. Because Kj is built from nj retained contexts, it contains at most nj non-zero eigenvalues. We treat any remaining components within the dmodel -dimensional readout space as having zero energy. For λ1 ≥ λ2 ≥ · · · , the k-span sufficiency score is: Pk λℓ (k) . Sspan = P ℓ=1 λ + ϵ ℓ ℓ 14
This score measures the fraction of directional energy explained by the first k components. Since high cumulative energy alone can hide a dominant first component or a long spectral tail, FEGA also reports the component-use share: λk , ℓ λℓ + ϵ
(k) Uspan =P
and the post-k spectral-drop ratio: (k) Dspan =
λk+1 . λk + ϵ
(k)
(k)
Here Sspan measures whether k dimensions are sufficient, Uspan measures whether the kth component is (k) meaningfully used, and Dspan measures whether there is a clean drop after k. These statistics are interpreted together with effective rank (Section 5.4). Passing the gates with k = 2 yields a global two-dimensional directional-subspace label (Figure 3), while passing them with k ∈ {3, 4, 8} yields a global k-dimensional label. Appendix I gives representative spectra. Centered Residual Spectrum. The preceding diagnostics analyze the whole cloud of normalized directions. This can miss a feature with a reliable average effect plus a smaller, systematic context-dependent deviation: the mean direction may dominate the global spectrum, while the variation around it still has low-dimensional structure. The centered residual spectrum isolates this variation by subtracting the average normalized direction before computing a spectrum. Let H=I−
1 11⊤ ∈ Rnj ×nj , nj
Kctr = HKj H.
The matrix H performs row-centering in context space: it subtracts the sample mean direction from each retained context. Since Kj = Vj Vj⊤ , Kctr = HVj Vj⊤ H = (HVj )(HVj )⊤ is the Gram matrix of the mean-centered logit directions. The fraction of directional energy remaining after centering is: Eres =
tr(Kctr ) . tr(Kj ) + ϵ
Low Eres indicates that the mean direction already explains most of the cloud. When residual energy is nontrivial, the centered eigenvalues η1 ≥ η2 ≥ · · · define: Pk (k) ℓ=1 ηℓ P Sres = . ℓ ηℓ + ϵ The residual low-dimensional label should be driven by nontrivial residual energy together with high small-k sufficiency. FEGA makes this claim for k ∈ {2, 3, 4}; a one-dimensional residual may be noted as a descriptive fallback, but is not treated as a supported residual-dimensionality claim. When one of these dimensions is supported, the cloud has an average direction and its context-dependent deviations occupy a small subspace. Tiny residual energy should not drive a residual-structure claim. Appendix E gives the derivation. 15
Diagnostic Directed ray Axis split
What It Detects One shared direction One axis, two signs
Key Readout Cray Baxis,j
Guardrail Low score has many causes. Sign split is not temporal flipping. Accepted modes M , coherence gain Modes must be populated ∆mix and stable.
vMF mixture
Several coherent directions
Span spectrum
One shared low-D subspace
Sspan , Uspan , Dspan
Residual spectrum
Eres , Sres
Secondary flags Selected-family stability
Low-D variation around the mean Strength and failure modes Confidence in the selected result
Reporting label
Final summary family
(k)
(k)
(k)
High energy alone can hide tails.
(k)
CVm,j , long-tail, instability flags Family-specific intervals, assignment agreement, leave-out, sample-size, or angle evidence Primary label plus flags
Residual energy must be nontrivial. Flags qualify the label. Availability is reported separately. Descriptive, not mechanistic.
Table 3: FEGA diagnostic map. Each row tests one property of the same effect cloud. The key readout gives the statistic used for the test; the guardrail states what the statistic alone should not be taken to imply. 5.4
Evidence Strength and Secondary Flags
The diagnostics above identify the geometric family best supported by the full cloud. FEGA then evaluates only the stability evidence relevant to that selected family and, where applicable, its selected dimension. Sample size, spectral tails, and magnitude heterogeneity remain companion summaries; none of these checks replaces the full-sample label. Effective Rank. The span statistics evaluate a chosen value of k. Effective rank provides a complementary summary of how many components the relevant spectrum uses overall. For the global span diagnostic, this spectrum is given by the eigenvalues of Kj ; for the centered residual diagnostic, it is given by the eigenvalues of Kctr . Thus, effective rank acts as a safeguard: it helps distinguish a clean low-dimensional spectrum from a spectrum with one large component plus a long tail. For a non-negative spectrum s1 , . . . , sr , define: sℓ . q=1 sq + ϵ
pℓ = Pr The entropy and participation-ratio ranks are:
! rent = exp −
X
1 rPR = P 2 . ℓ pℓ
pℓ log(pℓ + ϵ) ,
ℓ
Ignoring the numerical safeguard, both equal k for a uniform k-component spectrum. Participation-ratio rank emphasizes components carrying substantial energy, while entropy rank is more sensitive to long lowenergy tails. FEGA uses rPR as the more conservative dimensionality summary and rent as a long-tail warning. Neither statistic defines a geometry family on its own; instead, they qualify span and residual (k) (k) claims made from Sspan and Sres . Magnitude Heterogeneity. The primary geometry family describes the normalized removal directions (i) vj . Because normalization removes effect size, FEGA reports magnitude heterogeneity separately using the (i)
logit-effect magnitudes mj defined in Section 5.1: (i)
CVm,j =
stdi (mj ) (i)
meani (mj ) 16
.
Sampled effect cloud ℰ𝑗 : sufficient full-sample evidence?
no
valid-context and zero-effect evidence gates
Insufficient evidence reported separately
yes Select the first supported family in the fixed order below. For dimensioned families, retain the smallest supported 𝑘
1.
Most directions repeat one effect?
yes
Directed Ray
one stable removal direction
directed-ray concentration 𝐶𝑟𝑎𝑦 is high no
2.
Same axis with both signed sides populated?
yes
strong 1-D span; axis-split 𝐵𝑎𝑥𝑖𝑠 on both sides
Axis or Antipodal
one axis, two orientations
no
3.
Several coherent directed modes?
yes
vMF mixture accepted; coherence gain Δ𝑚𝑖𝑥 > 0
Directional Mixture
several prototype directions
no
Shared low-dimensional subspace? 𝑘 𝑘 𝑘
4.
span spectrum 𝑆𝑠𝑝𝑎𝑛 , 𝑈𝑠𝑝𝑎𝑛 , 𝐷𝑠𝑝𝑎𝑛 sufficient
5.
Residual disagreement is low dimensional? (𝑘)
yes
Global Span Spectrum
whole cloud in a plane / k-D span
no yes
nontrivial 𝐸𝑟𝑒𝑠 , high 𝑆𝑟𝑒𝑠
Centered Residual Spectrum
mean direction plus low-D variation
no fallback
No strict family is supported otherwise
One Dimensional Diffuse
Unresolved Diffuse
Secondary flags: magnitude heterogeneity (𝐶𝑉𝑚,𝑗 ), long-tail spectrum, and family-specific stability evidence qualifies the selected family but does not replace them.
Figure 4: FEGA label assignment and qualification. Full-sample diagnostics select the first supported geometry family and, where applicable, its smallest supported dimension. Family-specific stability tests then qualify this selection; directional-mixture acceptance already includes assignment stability. High CVm,j adds a magnitude-instability flag, indicating that effect strength varies across contexts even when the normalized directions support a coherent geometry. This flag qualifies the primary label but does not replace it. For example, a feature can be assigned a directed-ray or low-dimensional-span label while still having unstable effect magnitudes. Long-tail spectral flags similarly qualify low-dimensional summaries. Sensitivity to omitted contexts is incorporated into the selected-family stability result described below. Appendix I gives examples, and Appendix H lists the flag triggers. Stability and Evidence Strength. FEGA reports insufficient effect evidence when fewer than eight valid non-zero contexts remain. Otherwise, it first selects the most specific family supported by the full cloud and, for a dimensioned family, the smallest supported dimension. Stability analysis then considers only evidence relevant to that selected result. The relevant evidence may be a ray-concentration interval, family-local leave-out and sample-size checks, a principal angle at the selected dimension, or the assignment agreement already computed for an accepted directional mixture. Stability changes the confidence attached to the result, not its family or dimension. Observed instability and unavailable evidence are reported separately, while fallback and terminal outcomes are marked as not evaluated. Appendix G specifies the family-specific procedures, and Appendix H lists the reporting thresholds. The thresholds are reporting gates rather than universal constants. 5.5
Geometry Labels
Figure 4 summarizes the FEGA label-assignment procedure. FEGA records the full point-diagnostic profile for each feature together with stability evidence for the selected family, but assigns one primary geometry label to that feature’s sampled logit-effect cloud for empirical summaries. 17
(a) ReLU
(b) TopK LSC
Directed ray
WC
TT
Global kD directional subspace
(c) Matryoshka Batch TopK PrOntoQA Unresolved high dimensional or diffuse
Figure 5: FEGA atlas for pointer-like features. UMAP projections of downstream logit-effect geometry across four in-context tasks. Marker shape denotes the task and color denotes the FEGA label; features with undefined geometry are omitted. Features shared across all four tasks are shown at higher opacity. Clouds with insufficient valid effects are excluded before geometry selection. For each remaining cloud, FEGA considers the families in the fixed order directed ray, axis or antipodal structure, accepted directional mixture, global low-dimensional span, and centered residual low-dimensional structure, as shown in Figure 4. Ray, axis, span, and residual selection use full-sample diagnostics; directional-mixture acceptance is the sole exception because it already includes its standalone assignment-stability audit. The first supported family is retained, and dimensioned families use the smallest supported k. Subsequent family-specific stability checks qualify this fixed choice rather than searching for a replacement. This priority gives more specific explanations precedence: a directed ray, for example, is also low-dimensional, but is more informative than a generic span. Appendix H gives the exact gates and priority rules. If no strict family passes, FEGA reports the first descriptive fallback supported by the full cloud: onedimensional diffuse evidence, the first anchored global span, the first anchored centered residual span, or unresolved high-dimensional or diffuse evidence. Directed rays, axes, and accepted directional mixtures are strict selections rather than separate fallback branches. Fallback outcomes describe the nearest supported structure and are not subjected to the selected-family stability protocol. The three terminal outcomes answer different questions. Insufficient effect evidence means that the initial evidence gate failed; geometry metrics unavailable means that the evidence gate passed but every family test lacked the quantities needed for evaluation; and undefined geometry means that family tests were attempted but neither a strict family nor a descriptive fallback was supported.
6
Geometric Analysis of Downstream Effects
Having formalized FEGA, we apply it to isolated features across our evaluation spectrum, from synthetic pattern matching to factual attribute retrieval. This lets us systematically ask how downstream effect geometry varies with a feature’s computational role. 6.1
The Geometry of Pointer-Like Features
We first examine features isolated from the four ICL tasks: LSC, WC, PrOntoQA, and TT. We call these features pointer-like candidates as they recur across model-correct examples in tasks involving prompt-local copying, lookup, rule completion, or schema following. 18
Task
Selected
Undef.
LSC WC PrOntoQA TT
5 25 78 27
2 20 66 17
LSC WC PrOntoQA TT
5 9 22 14
4 5 13 7
LSC WC PrOntoQA TT
4 14 20 16
2 5 10 8
Unres.
G-kD Res.-lowD 1D diff. ReLU SAE 3 0 0 0 5 0 0 0 11 1 0 0 10 0 0 0 TopK SAE 1 0 0 0 3 1 0 0 8 0 0 0 7 0 0 0 Matryoshka Batch TopK SAE 2 0 0 0 7 1 0 0 9 0 0 0 8 0 0 0
Ray
G-2D
Axis
Multi
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 1 0
0 0 0 0
0 0 0 0
0 0 0 0
0 1 1 0
0 0 0 0
0 0 0 0
0 0 0 0
Table 4: Primary FEGA labels for pointer-like features. Geometry labels for features isolated from the four ICL tasks across Gemma-2-2B SAE variants of width 65k. Selected is the number of features active across at least 90% examples and on at least 90% queries in at least 90% prompt families per ICL task; Undef. denotes eligible features for which family tests were attempted, but neither a strict family nor a descriptive fallback was supported. Other columns denote unresolved/diffuse (Unres.), global k-dimensional subspace (G-kD), residual low-dimensional (Res.-lowD), one-dimensional diffuse (1D diff.), directed ray (Ray), global 2D subspace (G-2D), axis or antipodal (Axis), and directional mixture (Multi).
Figure 5 and Table 4 show a clear pattern. Of the 239 selected features, 159 are classified as having undefined geometry. Among the 80 mapped cases, 74 are classified as unresolved high-dimensional or diffuse. Only three form directed rays, and three occupy global low-dimensional subspaces. Thus, when pointer-like candidates have sufficient evidence for geometric analysis, stable low-dimensional structures are rare. This pattern is consistent with their context-dependent targets. A feature may support a similar operation across prompts, but copying “apple” and copying “car”, for instance, require effects on different output logits. As the relevant value changes, the downstream effect can change with it. A shared functional role therefore need not produce a shared direction in logit space. TT provides a similar example. The prompt specifies the target language, but the correct translation changes with the source word. A feature shared across TT examples therefore contributes to predicting many different output tokens, so its effect need not follow a fixed direction in logit space. Takeaway 4 Mapped pointer-like effects are overwhelmingly diffuse. Among pointer-like candidates with a mapped geometry, stable low-dimensional structures are rare, consistent with their effects changing as the prompt-local target changes.
6.2
The Geometry of Value-Like Features
We next evaluate features isolated from the RAVEL city-country attribute task. Since these latents are selected by differential binary masking to support attribute editing (e.g., changing a city’s country attribute), they are natural candidates for stable conceptual control vectors. FEGA tests whether such value-like features produce a consistent one-dimensional downstream effect across contexts. Figure 6 shows the diversity of the mapped RAVEL geometries, but it does not display the entire selected population. Features with insufficient effect evidence and features for which no geometry family is supported are omitted from the atlas and reported separately in Table 5. Among the remaining cases, isolating a factual attribute at the representation level still does not imply a single downstream ray. 19
ReLU SAE 2,955 geometry-eligible features
Directed ray
Axis or antipodal
TopK SAE 726 geometry-eligible features
Directional mixture
Centered Residual Spectrum
Matryoshka Batch TopK SAE 991 geometry-eligible features
Global 2𝐷 directional subspace Unresolved diffuse
Global 𝑘𝐷 directional subspace
1𝐷 diffuse
Figure 6: FEGA atlas for value-like features. UMAP projections of the RAVEL features for which FEGA assigns a primary geometry label. Features with insufficient effect evidence or undefined geometry are omitted from the atlas and retained in the complete counts in Table 5. Colors indicate the primary FEGA label. Table 5 quantifies this pattern. Of the 12,900 ReLU, 2,828 TopK, and 3,654 Matryoshka Batch TopK features selected by differential binary masking, 5,185, 1,661, and 1,904, respectively, lack sufficient effect evidence for a geometry claim. Among the geometry-eligible features, undefined geometry accounts for 4,760 of 7,715 ReLU features, 441 of 1,167 TopK features, and 759 of 1,750 Matryoshka Batch TopK features. Unresolved high-dimensional or diffuse evidence accounts for a further 1,860, 277, and 452 features, respectively. Together, the G-kD, residual-low-dimensional, 1D-diffuse, and G-2D categories contain 1,078 ReLU features, 424 TopK features, and 530 Matryoshka Batch TopK features, corresponding to 14.0%, 36.3%, and 30.3% of the eligible populations. Directed rays remain much rarer, with 5, 14, and 6 features, respectively. This geometric heterogeneity is consistent with the complexity of natural language generation. Even if an individual SAE feature encapsulates a concept such as the country “France”, ablating it from the model’s residual stream may alter vocabulary probabilities along multiple axes. Depending on the grammar and phrasing of the prompt, suppressing that feature might boost probabilities for competing geographic entities (e.g., “Japan”), alter regional language associations (e.g., shifting toward “Spanish”), or produce a localized syntactic adjustment. In structured cases, the feature’s downstream effects therefore occupy a low-dimensional subspace rather than a single direction. This shows that understanding what conceptual knowledge a feature encodes is distinct from predicting how its intervention will geometrically perturb the output space. Takeaway 5 Value-like candidates exhibit low-dimensional structure more often than pointer-like candidates. Their structured effects usually span multiple directions, while directed rays remain rare.
7
Discussion and Conclusion
A common assumption in SAE-based steering is that an isolated feature provides a stable, one-dimensional control direction. Our results show that such directions are rare. Value-like features tied to factual attributes exhibit low-dimensional structure more often than pointer-like features, but many cases remain undefined or 20
SAE ReLU TopK Matryoshka Batch TopK
Selected Insuff. Undef. Unres. G-kD
Res.- 1DRay G-2D Axis Multi lowD diff.
12,900 2,828
5,185 1,661
4,760 441
1,860 277
795 243
278 165
2 3
5 14
3 13
12 0
0 11
3,654
1,904
759
452
352
164
3
6
11
2
1
Table 5: Primary FEGA labels for value-like features. Geometry outcomes for features selected from the RAVEL city-country task. Selected is the number of positive MDBM-mask entries. Insuff. denotes features with fewer than eight valid nonzero effects. Undef. denotes eligible features for which family tests were attempted, but neither a strict family nor a descriptive fallback was supported. The remaining columns denote unresolved high-dimensional or diffuse geometry (Unres.), global k-dimensional subspaces (G-kD), residual low-dimensional structure (Res.-lowD), one-dimensional diffuse structure (1D diff.), directed rays (Ray), global two-dimensional subspaces (G-2D), axes or antipodal structure (Axis), and directional mixtures (Multi).
unresolved, and directed rays are uncommon. Thus, even when a factual intervention has a structured effect, it can perturb multiple output directions rather than following a single steering vector. Among pointer-like candidates with mapped geometries, effects are overwhelmingly diffuse or high-dimensional, consistent with their downstream directions changing as the prompt-local target changes. These differences have direct implications for steering. When a value-like effect spans a low-dimensional subspace, steering along a single direction may also alter related output probabilities. The prompt-local operations studied here pose a stronger challenge for static steering: their effects suggest that fixed, contextinvariant vectors may be insufficient for controlling copying, binding, and similar operations. Such interventions may instead need to adapt to the prompt and to the value currently being operated on. These findings also affect how SAEs should be trained and evaluated. Cross-task overlap is greater for TopK and Matryoshka Batch TopK than for ReLU SAEs, suggesting that SAE architecture influences how features associated with prompt-local operations are organized. However, no architecture consistently yields the largest candidate sets, and the selected sets remain architecture-dependent. Our results therefore motivate training objectives and evaluations that complement factual recall with pointer-like functions and other context-dependent computations. Overall, SAE interpretation should move beyond asking only what a feature represents. For reliable steering, auditing, and editing, we must also ask how intervening on that feature geometrically perturbs the model’s downstream output space.
8
Limitations and Future Work
FEGA provides a framework for evaluating downstream feature stability, but our study has several boundaries. First, all experiments use Gemma-2-2B with 65k-width SAEs applied to the post-layer-12 residual stream. Although we compare ReLU, TopK, and Matryoshka Batch TopK variants, feature organization may change across model scales, SAE widths, or intervention layers. Larger models may also require different SAE capacities to isolate pointer-like mechanisms. Whether the predominance of diffuse geometry among mapped pointer-like candidates persists at larger scales is therefore an open empirical question. Second, our task spectrum uses structured prompts designed to separate computational roles. LSC and RAVEL represent useful endpoints: prompt-local copying on one side and factual attribute editing on the other. In more naturalistic generation, pointer-like routing and value-like retrieval may interact within the same computation. Future work should apply FEGA to less constrained settings, such as multi-step or chain-of-thought generation, where prompt-local operations and semantic knowledge are jointly used. 21
Our conclusions are also conditional on the features selected by each task and on the at most 64 active contexts retained per feature. Moreover, many selected features either lack sufficient effect evidence or do not receive a defined geometry label. Our claims about the distribution of mapped geometries should therefore not be extended to these cases. FEGA describes the geometry observed for a feature over its retained context sample; it does not claim that the same label must hold over every possible prompt. Stability analyses measure sensitivity within this sample but cannot eliminate the underlying selection boundary. Finally, our measurements are taken at the target-position logit readout. This choice is behaviorally motivated, since logits determine the model’s next-token distribution, but it also aggregates all transformations downstream of the SAE site. A feature with diffuse logit effects may still produce a low-dimensional local perturbation near that site. Applying FEGA at successive layers could reveal whether and where a localized feature effect disperses into a high-dimensional vocabulary-level effect.
Acknowledgments P. Hoang, S. Dutta, and I. Gurevych acknowledge the support of the LOEWE Distinguished Chair “Ubiquitous Knowledge Processing”, LOEWE initiative, Hesse, Germany (Grant Number: LOEWE/4a//519/05/00.002 (0002)/81). T. Chakraborty acknowledges the support of the Rajiv Khemani Young Faculty Chair Professorship in AI and the NVIDIA Academic Grant Program. A. Chatterjee acknowledges the support of the Google PhD Fellowship.
References Dana Arad, Aaron Mueller, and Yonatan Belinkov. SAEs Are Good for Steering - If You Select the Right Features. CoRR, abs/2505.20063, 2025. doi: 10.48550/ARXIV.2505.20063. URL https://doi.org/10. 48550/arXiv.2505.20063. David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In Nikhil Bansal, Kirk Pruhs, and Clifford Stein (eds.), Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2007, New Orleans, Louisiana, USA, January 7-9, 2007, pp. 1027–1035. SIAM, 2007. ISBN 978-0-898716-24-5. URL http://dl.acm.org/citation.cfm?id=1283383.1283494. Arindam Banerjee, Inderjit S. Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the Unit Hypersphere using von Mises-Fisher Distributions. Journal of Machine Learning Research, 6(46):1345–1382, 2005. URL http://jmlr.org/papers/v6/banerjee05a.html. Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Jack Merullo, Atticus Geiger, and Ekdeep Singh Lubana. Do Sparse Autoencoders Capture Concept Manifolds? CoRR, abs/2604.28119, 2026. doi: 10.48550/ARXIV.2604. 28119. URL https://doi.org/10.48550/arXiv.2604.28119. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https: //openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023. Joschka Braun, Carsten Eickhoff, David Krueger, Seyed Ali Bahrainian, and Dmitrii Krasheninnikov. Understanding (Un)Reliability of Steering Vectors in Language Models. In ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025. URL https://openreview.net/forum?id= JZiKuvIK1t. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html. 22
Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 6077–6101. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/bussmann25a.html. Maheep Chaudhary and Atticus Geiger. Evaluating open-source sparse autoencoders on disentangling factual knowledge in GPT-2 small. CoRR, abs/2409.04478, 2024. doi: 10.48550/ARXIV.2409.04478. URL https://doi.org/10.48550/arXiv.2409.04478. Jingyi Cui, Qi Zhang, Yifei Wang, and Yisen Wang. On the limits of sparse autoencoders: A theoretical framework and reweighted remedy. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=DSOTgzeH3w. Esin Durmus, Alex Tamkin, Jack Clark, Jerry Wei, Jonathan Marcus, Joshua Batson, Kunal Handa, Liane Lovitt, Meg Tong, Miles McCain, Oliver Rausch, Saffron Huang, Sam Bowman, Stuart Ritchie, Tom Henighan, and Deep Ganguli. Evaluating Feature Steering: A Case Study in Mitigating Social Biases, 2024. URL https://anthropic.com/research/evaluating-feature-steering. Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tcsZt9ZNKD. Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26(83): 1–64, 2025. URL http://jmlr.org/papers/v26/23-0058.html. Gemma Team. Gemma 2: Improving Open Language Models at a Practical Size. CoRR, abs/2408.00118, 2024. doi: 10.48550/ARXIV.2408.00118. URL https://doi.org/10.48550/arXiv.2408.00118. Sai Sumedh R. Hindupur, Ekdeep S Lubana, Thomas Fel, and Demba Ba. Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (eds.), Advances in Neural Information Processing Systems, volume 38, pp. 11649–11699. Curran Associates, Inc., 2025. URL https://proceedings.neurips.cc/ paper_files/paper/2025/file/110d919b4a711f25962a7cd5961f4955-Paper-Conference.pdf. Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8669–8687, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.470. URL https://aclanthology.org/ 2024.acl-long.470/. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse Autoencoders Find Highly Interpretable Features in Language Models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=F76bwRSLeK. Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Isaac Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, Demian Till, Matthew Wearden, Arthur Conmy, Samuel Marks, and Neel Nanda. A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), ICML 2025, Vancouver, BC, Canada, July 1319, 2025, Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. URL https: //proceedings.mlr.press/v267/karvonen25a.html. 23
Anton Korznikov, Andrey V. Galichin, Alexey Dontsov, Oleg Rogov, Ivan V. Oseledets, and Elena Tutubalina. Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? CoRR, abs/2602.14111, 2026. doi: 10.48550/ARXIV.2602.14111. URL https://doi.org/10.48550/arXiv. 2602.14111. Matthew Levinson. Finding Belief Geometries with Sparse Autoencoders. CoRR, abs/2604.02685, 2026. doi: 10.48550/ARXIV.2604.02685. URL https://doi.org/10.48550/arXiv.2604.02685. Aaron J. Li, Suraj Srinivas, Usha Bhalla, and Himabindu Lakkaraju. Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders. In Vera Demberg, Kentaro Inui, and Lluís Marquez (eds.), Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2026 - Volume 1: Long Papers, Rabat, Morocco, March 24-29, 2026, pp. 5940–5957. Association for Computational Linguistics, 2026. doi: 10.18653/V1/2026.EACL-LONG.279. URL https: //doi.org/10.18653/v1/2026.eacl-long.279. Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The Geometry of Concepts: Sparse Autoencoder Feature Structure. Entropy, 27(4):344, 2025. doi: 10.3390/E27040344. URL https://doi.org/10.3390/e27040344. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 278–300, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.blackboxnlp-1.19. URL https://aclanthology.org/ 2024.blackboxnlp-1.19/. Daniel W. Lozier. NIST digital library of mathematical functions. Ann. Math. Artif. Intell., 38(1-3):105– 119, 2003. doi: 10.1023/A:1022915830921. URL https://doi.org/10.1023/A:1022915830921. Modified Bessel function integral, Equation 10.32.2. Tommaso Mencattini, Francesco Montagna, and Francesco Locatello. The rate-distortion-polysemanticity tradeoff in saes. CoRR, abs/2605.14694, 2026. doi: 10.48550/ARXIV.2605.14694. URL https://doi. org/10.48550/arXiv.2605.14694. Gouki Minegishi, Hiroki Furuta, Yusuke Iwasawa, and Yutaka Matsuo. Rethinking evaluation of sparse autoencoders through the representation of polysemous words. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=HpUs2EXjOl. Jingcheng Niu, Subhabrata Dutta, Ahmed Elshabrawy, Harish Tayyar Madabushi, and Iryna Gurevych. Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview. net/forum?id=10QqO1tM1H. Abulhair Saparov and He He. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=qFVVBzXxR2V. Bernhard Schölkopf, Alexander J. Smola, and Klaus-Robert Müller. Nonlinear Component Analysis as a Kernel Eigenvalue Problem. Neural Comput., 10(5):1299–1319, 1998. doi: 10.1162/089976698300017467. URL https://doi.org/10.1162/089976698300017467. Gideon Schwarz. Estimating the dimension of a model. The annals of statistics, pp. 461–464, 1978. Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeffrey Wu, Lucius Bushnaq, Nicholas GoldowskyDill, Stefan Heimersheim, Alejandro Ortega, Joseph Isaac Bloom, Stella Biderman, Adrià Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, William 24
Saunders, Eric J. Michaud, Stephen Casper, Max Tegmark, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath. Open problems in mechanistic interpretability. Trans. Mach. Learn. Res., 2025, 2025. URL https://openreview.net/forum?id=91H76m9Z94. Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. CoRR, abs/2503.05613, 2025. doi: 10.48550/ARXIV.2503.05613. URL https://doi.org/10.48550/arXiv. 2503.05613. Daniel Chee Hian Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià GarrigaAlonso, and Robert Kirk. Analysing the Generalisation and Reliability of Steering Vectors. In The Thirtyeighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview. net/forum?id=v8X70gTodR. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy. Scipy 1.0-fundamental algorithms for scientific computing in python. CoRR, abs/1907.10121, 2019. URL http://arxiv.org/abs/1907.10121. Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), ICML 2025, Vancouver, BC, Canada, July 13-19, 2025, volume 267 of Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. URL https://proceedings.mlr.press/v267/wu25a.html.
A
Sign Convention for Removal Effects
FEGA orients each effect as the downstream change caused by removing a feature. For context c, the logit-space removal effect is: (c)
(c)
(c)
(c)
(c)
∆j = ϕℓ (ĥℓ,j↓ ) − ϕℓ (h̄ℓ ), (c)
where, h̄ℓ
(c)
(c)
= D(z (c) ) is the full SAE reconstruction and ĥℓ,j↓ = D(z (c) − zj ej ) is the reconstruction with (c)
feature j zeroed. Thus, ∆j,t > 0 means that removing feature j raises token t’s logit, so the retained feature (c)
had been suppressing it; ∆j,t < 0 means removal lowers the logit, so the feature had been supporting it. (c)
Before unembedding, the corresponding removal effect is δj
(c)
(c)
(c)
(c)
(c)
(c)
= rℓ (ĥℓ,j↓ ) − rℓ (h̄ℓ ), with ∆j = WU δj .
This sign fixes the interpretation of positive and negative token changes, but not FEGA’s geometry. Let (c ) (c ) vi = ∆j i /∥∆j i ∥2 be the normalized logit-effect direction for feature j in context ci . If one instead used the opposite convention, describing what the retained feature contributed before removal, every direction in the feature cloud would flip: ṽi = −vi . Hence ṽi⊤ ṽk = (−vi )⊤ (−vk ) = vi⊤ vk . The same holds when computing logit geometry from pre-logit effects with G = WU⊤ WU , since (−δi )⊤ G(−δk ) = δi⊤ Gδk . 25
Quantity atol Linear-logit reconstruction 10−6 Logit norm 10−6 Logit inner product 10−5 Cosine 10−5 Cray 10−4 (1) Sspan 10−4 (1) Sres 10−4
rtol 10−6 10−5 10−5 10−5 10−4 10−4 10−4
Max. absolute error 0 7.67 × 10−13 1.17 × 10−10 8.92 × 10−8 5.59 × 10−9 0 0
Table 6: Gram–logit equivalence on the bounded validation sample. Each entry is the largest absolute discrepancy across the eight selected features and their eight retained contexts; kernel comparisons include all context pairs. A comparison passes when |a − b| ≤ atol + rtol|b|. All seven comparisons passed. Model replay used bfloat16 arithmetic, while the Gram and equivalence calculations used float64. Thus, magnitudes, pairwise similarities, spectra, effective-rank summaries, and FEGA geometry labels are unchanged by flipping all contexts for the same feature. The sign only matters for externally oriented quantities, such as positive versus negative token effects, agreement with a target direction, or signed intervention scores. Throughout the paper, FEGA uses the removal orientation above.
B
Readout-Gram Equivalence and Validation
FEGA geometry is defined in logit space, but most diagnostics require only logit-space inner products. We, therefore, retain the pre-logit removal effects in their original context order than storing vocabularyr rather ⊤ (c) (c) Gδj > 10−12 . This set is δj dimensional logit effects. A context enters the effect cloud only when (c)
formed once and used throughout our analysis. Since ∆j contexts ci and ck , (c )
(c ) ⊤
(c )
∆j i , ∆j k
2
= WU δj i
(c)
= W U δj
(c )
WU δ j k
and G = WU⊤ WU , for any retained
(c )
(c )
= (δj i )⊤ Gδj k .
Thus, any diagnostic depending only on logit-space inner products can be computed exactly in pre-logit coordinates through the unembedding Gram matrix, including magnitudes, cosines, pairwise kernels, directed-ray concentration, span and centered spectra, and effective-rank summaries. We evaluated this equivalence on a deterministic sample from the Gemma-2-2B ReLU SAE at layer 12 (width 216 , trainer 0) for the RAVEL city–country task. The sample contains the eight lowest-index eligible features, 1, 4, 17, 26, 44, 47, 50, and 56, and the first eight retained contexts for each feature. As Table 6 shows, the Gram and explicit linear-logit calculations agreed for every comparison under the stated tolerances; the largest absolute difference was 8.92×10−8 , for a cosine. In contrast, the returned post-softcap logit differences departed from the corresponding linear-logit effects by as much as 1.44, and are therefore not used to define FEGA geometry.
C
Directed-Ray Concentration Fast Identity (i)
Fix a feature j, write vi = vj for its n = nj normalized logit-effect directions, and let K = Kj be the directional kernel with entries Kik = vi⊤ vk . The directed-ray score is the mean ordered off-diagonal similarity, Cray (j) =
X 1 1⊤ K1 − tr(K) vi⊤ vk = . n(n − 1) n(n − 1) i̸=k
The second equality subtracts the diagonal terms from the full kernel sum. This trace-corrected form is exact for the stored rows. When effects are stored in pre-logit form, the same entries Kik are computed using the Gram metric from Appendix B. 26
The fast formula follows by summing the directions. Let Sj = exact unit normalization gives tr(K) = n, and hence Cray (j) =
Pn
2 i=1 vi . Since ∥Sj ∥2 =
⊤ ⊤ i,k vi vk = 1 K1,
P
∥Sj ∥22 − n . n(n − 1)
Thus, the fast formula is the trace-corrected identity with the diagonal replaced by n. If numerical safeguards make row norms only approximately one, we use the trace-corrected form instead. This identity also gives the intended interpretation: Cray (j) is high when normalized removal effects repeatedly point in the same direction. A low value only shows that a shared directed component is weak; it does not distinguish antipodal, multimodal, low-dimensional, or diffuse structure.
D
Dual Span Spectrum (i)
Fix a feature j, and let Vj ∈ Rnj ×|V| have rows vj , the normalized logit-effect directions for the valid (i)
(k)
contexts. The directional kernel is Kj = Vj Vj⊤ , with entries (Kj )ik = (vj )⊤ vj . When effects are stored in pre-logit form, Appendix B gives the same entries using the unembedding Gram matrix; the geometry is still the Euclidean geometry of the induced logit directions. The spectrum of Kj is the dual-PCA spectrum of the effect cloud. If Vj = U ΣQ⊤ is an SVD, then Kj = Vj Vj⊤ = U Σ2 U ⊤ ,
Vj⊤ Vj = QΣ2 Q⊤ .
Thus, the nonzero eigenvalues of Kj are the squared singular values of the normalized logit-direction matrix. The nj × nj context kernel therefore contains the complete nonzero spectrum, allowing FEGA to diagonalize this kernel instead of constructing principal directions in vocabulary-sized logit space (Schölkopf et al., 1998). Because the effects pass through a dmodel -dimensional readout, spectral components not represented by Kj but still within that space are assigned zero energy; indices beyond dmodel do not denote available readout directions. For λ1 ≥ λ2 ≥ · · · , the span sufficiency score: Pk λℓ (k) Sspan (j) = P ℓ=1 λ + ϵ ℓ ℓ is the fraction of logit-direction energy captured by the top k principal directions. This score asks whether k directions are enough; component-use shares and post-k spectral-drop ratios are then used to avoid mistaking a dominant first component or a long spectral tail for a clean k-dimensional cloud. The leading axis used by the axis diagnostic is recovered from the same kernel. For an eigenpair Kj qℓ = λℓ qℓ with λℓ > 0, define: Vj⊤ qℓ , uℓ = √ λℓ + ϵ
Kj qℓ sℓ = Vj uℓ = √ . λℓ + ϵ
Here uℓ is the recovered logit-space principal direction, and sℓ contains the signed projection scores of all retained contexts onto that√direction. Ignoring the numerical safeguard, the √ recovered axes are orthonormal ⊤ λℓ qℓ . If λℓ is zero or numerically because u⊤ ℓ um = qℓ Kj qm / λℓ λm . For an exact positive eigenpair, sℓ = negligible, the corresponding axis is not a defined readout direction and should not drive a geometry label. For the axis or antipodal diagnostic, FEGA uses the leading defined component s1 . Since the sign of u1 is arbitrary, only the occupancy of both sides of the axis matters: the axis-balance score is the smaller fraction of retained contexts with positive and negative entries in s1 . This tests whether an apparently one-dimensional cloud is better interpreted as an undirected axis than as one directed ray. 27
E
Centered Residual Spectrum
The centered residual spectrum asks whether there is structured context-dependent variation after removing the average normalized effect direction. Fix a feature j, and let Vj contain its nj normalized logit-effect directions as rows. With Hj = I − n1j 11⊤ , row centering gives Hj Vj , and since Kj = Vj Vj⊤ , the centered kernel is: Kj,ctr = Hj Kj Hj . Thus, Kj,ctr is the Gram matrix of the centered directions. Since the trace of a Gram matrix equals total squared row norm, tr(Kj ) = ∥Vj ∥2F and tr(Kj,ctr ) = ∥Hj Vj ∥2F . The residual-energy statistic is, therefore, tr(Kj,ctr ) , tr(Kj ) + ϵ
Eres (j) =
the fraction of normalized directional energy left after subtracting the mean direction. This diagnostic asks a different question from directed-ray concentration. A feature may have a reliable average removal direction while still varying systematically across contexts; Eres (j) measures how much such variation remains. Let η1 ≥ η2 ≥ · · · be the eigenvalues of Kj,ctr . The centered-residual sufficiency score is: Pk ηℓ (k) Sres (j) = P ℓ=1 . ℓ ηℓ + ϵ This score asks whether remaining deviations are low-dimensional. It is interpreted with Eres (j): high (k) Sres (j) is meaningful only when the centered residual energy is large enough to matter, and weak when the deviations around the mean are numerically small.
F
Directional Mixtures and Model Selection
FEGA fits the directional-mixture candidates for every feature with at least eight retained directions, independently of the preceding ray diagnostic; the family priority is applied only after these candidates have been evaluated. (i)
For feature j, the fitted data are yi = vj
∈ S|V|−1 . If effects are stored in pre-logit form, this diagnostic (c )
(c )
materializes the corresponding logit direction as yi = WU δj i /∥WU δj i ∥2 , since vMF fitting requires explicit logit coordinates rather than only pairwise Gram entries. An M -component von Mises–Fisher mixture has likelihood pM (yi ) =
M X
πr C|V| (κr ) exp(κr µ⊤ r yi ),
r=1
∥µr ∥2 = 1,
κr ≥ 0,
M X
πr = 1,
r=1
where µr is the prototype direction of mode r, κr measures its concentration, and πr is its fitted probability mass. These weights describe how the fitted model distributes the effect cloud across its directional modes: a context may contribute to several modes to different degrees and is assigned to its best-matching mode only when a mode-specific summary is required (Banerjee et al., 2005). The normalizer Cd (κr ) is retained because BIC compares full likelihoods; changing κr changes not only cosine alignment but also how probability mass is distributed on the sphere. d/2−1
κ Writing d = |V|, the normalizer is Cd (κ) = (2π)d/2 I
, κ > 0, with the uniform-sphere limit Cd (0) =
d/2−1 (κ) Γ(d/2) . We evaluate log C (κ) in float64 using an exponentially scaled modified Bessel function. d d/2 2π
If this evaluation underflows or is otherwise non-finite, we evaluate the equivalent mode-centered spherical integral 28
by adaptive quadrature and retain it only when the estimated relative integration error is at most 10−11 (Virtanen et al., 2019; Lozier, 2003). The implementation was checked for 2 ≤ d ≤ 256,000 over the reachable concentration range, extending to 1.28 × 1015 at d = 256,000. The largest observed error in the cancellation-safe quantity log Cd (κ) + κ was 2.33 × 10−10 , below the declared 10−8 absolute bound. For each eligible cloud, we fit every M ∈ {1, 2, 3, 4} by soft EM (Banerjee et al., 2005) with k-means++ initialization (Arthur & Vassilvitskii, 2007). Each mode count receives four deterministic starts, each limited to 200 iterations. A start stops when the summed squared change in its prototype directions is no greater than 10−6 times the mean coordinate-wise variance of the fitted unit directions. For a fixed M , we retain the start with the greatest finite full log likelihood; because replacement requires strict improvement, an exact tie retains the earlier start. A failed start does not suppress the remaining starts, and a mode count is unavailable only when none produces a finite full likelihood. Candidate mode counts are selected by BIC (Schwarz, 1978), BICM = −2 log LM + νM log nj ,
νM = M (d − 1) + M + (M − 1).
The three terms in νM count unit mean directions, concentration parameters, and free mixture weights. Beginning with the smallest finite mode count, a larger M is selected only when its BIC is more than 10−9 lower; otherwise, the smaller model is retained. BIC proposes a statistical fit, but FEGA accepts a multi-mode geometry only after additional gates. The selected model must have M > 1, each mode must have fitted mass πr ≥ 0.10, and its within-mode concentration is computed only when at least two contexts have that mode as their best match, within-mode directed-ray concentrations must be defined and sufficiently coherent, and clustering must improve over the pooled cloud: ∆mix (j) =
M X
πr Cray,r (j) − Cray (j).
r=1
The selected fit is accepted as a directional mixture only when ∆mix (j) ≥ 0.10, every mode has fitted mass πr ≥ 0.10, every κr is finite, every within-mode directed-ray concentration is defined and at least 0.70, and assignment stability is at least 0.80. A within-mode concentration is computed only when at least two contexts have that mode as their best match; this hard-assignment requirement is distinct from the fitted mass πr . For a selected M > 1, assignment stability is evaluated on eight deterministic subsets drawn without replacement, each containing min{nj , max[M, ⌈0.8nj ⌉]} contexts. Each subset is refitted at the already selected mode count, and its assignments are compared with the corresponding full-fit assignments using the adjusted Rand index. Assignment stability is the mean of the eight scores only when every refit succeeds and remains finite; otherwise, it is unavailable. For M = 1, assignment stability is not applicable. FEGA reports the selected M , ∆mix (j), the smallest fitted mode mass, the weakest within-mode concentration, assignment stability, and the smallest fitted κr . A rejected or unavailable fit provides no accepted multi-mode evidence, while selection of M = 1 should not be read as proof that the cloud is truly unimodal. An accepted mixture indicates multiple directional regimes in the sampled effect cloud, not by itself multiple semantic meanings of the feature.
G
Stability Protocols
FEGA first selects a geometry family from the complete effect cloud and, where applicable, the smallest supported dimension. It then evaluates only the stability evidence associated with that selected result as shown in Table 9. For small or structured context sets, FEGA also uses leave-out sensitivity. Leave-one-out or leave-groupout checks remove a context, entity group, or template group and recompute the evidence required by the 29
Evidence gate Minimum retained contexts
Threshold 8
Maximum zero-effect filter fraction
0.30
Magnitude-instability flag threshold
1.00
90th percentile of the maximum principal angle
30◦ , 30◦ , 35◦
Interpretation Below this, valid nonzero effects are too few for a geometry-family claim. A retained coherent cloud is weak evidence if many candidate contexts showed no measurable effect. Large relative strength variation is reported as a flag, not as its own direction family. The selected 1D, 2D, or higher-D subspace should reappear across the resampled context subsets.
Table 7: FEGA evidence gates. These gates check whether a sampled feature-effect cloud has enough usable evidence for a geometry claim. They are conservative because normalization can make small or rare effects appear geometrically organized. The angle row gives the 1D, 2D, and higher-D resampling thresholds. Directional-family gate Directed-ray concentration
Threshold 0.80
(1)
Single-axis sufficiency, Sspan
0.80
Ray participation-rank boundary, d = 2
1.45
Minimum axis-balance mass
0.15
Minimum mixture-gain improvement
0.10
Minimum mode mass Minimum within-mode ray concentration
0.10 0.70
Minimum assignment stability
0.80
Interpretation Valid effects should repeatedly align with one directed readout direction. The cloud should be nearly one-dimensional before deciding between ray and sign-split axis. Flags ray-like cases whose spectrum is close to a low-dimensional span regime. Both sides of the leading axis need enough contexts to support an axis label. Clustering should make modes more ray-like than the pooled cloud. Accepted modes should not be tiny leftover clusters. Each accepted mode should be internally coherent, not merely separated. Directional modes should persist under resampling, not reflect a fitting accident.
Table 8: Directional-family gates. These thresholds decide whether low or ambiguous global ray concentration is better explained by a directed ray, an unsigned axis, or several coherent directional modes selected family. Rather than reclassifying each subset, FEGA asks whether the evidence continues to support that family and, for dimensioned families, the same smallest k. Failures weaken the confidence or add an instability flag; they do not replace the full-sample family or dimension. For a selected axis, global span, or centered residual family, subspace stability compares the full-sample subspace with resampled subspaces only at the retained dimension: k = 1 for an axis and k = k ∗ for a span or residual result. Let U and W be bases for the two subspaces, orthonormal in the geometry used to store the effects: Euclidean for logit directions, or G-orthonormal for pre-logit coordinates. The singular values of U ⊤ W in logit coordinates, or U ⊤ GW in pre-logit coordinates, give the principal angles θℓ = arccos(clip(sℓ , −1, 1)) ,
ℓ = 1, . . . , k.
For nj ≥ 32, FEGA draws 20 deterministic subsets without replacement, each containing ⌈0.75nj ⌉ contexts. For each subset, it converts the k principal angles to degrees and retains the largest one; the reported statistic is the linearly interpolated 90th percentile of these 20 maxima. The selected subspace passes this check when the statistic is at most 30◦ for k ∈ {1, 2} and at most 35◦ for k > 2. Each basis is obtained by eigendecomposing a symmetrized induced row kernel. An eigenvalue below −10−5 makes the comparison unavailable; otherwise, negative eigenvalues are treated as numerical roundoff, and only eigenvalues strictly greater than 10−8 are retained. If the full cloud or any required subset has rank below k, the principal-angle evidence is likewise reported as unavailable. Directional mixtures instead use the fixed-mode adjusted-Rand protocol in Appendix F; they do not use the scalar intervals or principal-angle calculation described here. The evidence threshold is eight retained contexts. If nj < 8, FEGA reports insufficient effect evidence and does not perform stability qualification. For strict non-mixture selections, complete evidence is exploratory when 8 ≤ nj < 32 and accepted when 32 ≤ nj ≤ 64. An accepted directional mixture instead inherits the 30
Selected result Directed ray Axis or antipodal Directional mixture Global span at k Centered residual at k Fallback or terminal result
Retained stability evidence Cray interval and family-local leave-out and sample-size checks. Cray interval, raw one-dimensional angle, and family-local leave-out and sample-size checks. Assignment stability from the selected standalone vMF fit. Raw angle at k and family-local checks for strict dimensions through k. Centered-residual angle at k and family-local checks for strict dimensions through k. No stability protocol; reported as not evaluated.
Table 9: Selected-family stability evidence. Stability qualifies the result selected from the full cloud without reopening the family or dimension decision. Span or residual gate (k) Span sufficiency, Sspan
Threshold 0.90 (k)
Component-use share, Uspan , k = (2, 3, 4, 8)
(0.08, 0.05, 0.03, 0.01) (1.60, 2.30, 3.00, 5.00)
Span participation rank, rspan,PR , k = (2, 3, 4, 8) (k) Post-k span drop, Dspan
0.60
Centered residual energy
0.10
(k)
0.80
Residual sufficiency, Sres
Centered participation rank, rctr,PR , k = (2, 3, 4) Long-tail flag, Ltail
(1.50, 2.20, 2.90) 1.50
Interpretation The selected k-span should explain most normalized directional energy. The selected component should carry nontrivial energy, not merely complete the span. The spectrum should meaningfully use about k dimensions; this is a lower-bound gate. Lower is cleaner: energy should drop after k rather than continue as a tail. Residual spectra matter only when enough variation remains after removing the mean direction. The selected residual span should explain most centered residual directional energy. Residual structure should meaningfully use the reported residual dimension. A long-tail flag is added when entropy rank greatly exceeds participation rank.
Table 10: FEGA span and residual gates. These gates test whether a low-dimensional span is sufficient, whether its selected components are actually used, and whether the spectrum has a clean stopping point. Residual gates apply only after enough centered variation remains. assignment-stability evidence of its standalone fit. Fallback and terminal outcomes retain their point-state interpretation and are marked as not evaluated.
H
Geometry Reporting Gates
The geometry label is a reporting layer over pre-declared evidence gates, not a discovery procedure for internal mechanisms. Before testing a geometry family, FEGA assigns the insufficient-evidence terminal whenever the valid-effect count is unavailable or below eight, or the near-zero filter removes more than 30% of the loaded contexts. Remaining clouds are tested in priority order: directed ray, axis or antipodal structure, accepted directional mixture, global low-dimensional subspace, and centered residual low-dimensional structure. This order favors specific explanations: a directed ray is also low-dimensional, but the ray label is more informative than a generic span label. If no strict family passes, FEGA reports a nearest-family fallback only when partial evidence remains interpretable. For example, strong one-dimensional span evidence without enough signed agreement for a ray or enough balanced sign split for an axis is reported as one-dimensional diffuse evidence, rather than being forced into either family. The unresolved high-dimensional or diffuse label is itself an evidence-bearing fallback: it is assigned only when no family anchor applies and the cloud satisfies the high-dimensional/diffuse or long-tail condition. If neither a strict family nor a descriptive fallback is supported, FEGA reports undefined geometry. The threshold profile has three layers: evidence gates decide whether the cloud is reliable enough to label; directional-family gates test for one ray, one sign-split axis, or several locally coherent modes; and span or 31
Profile Stable direction, unstable strength High sufficiency, weak second component
Numbers Directions align, but magnitudes (0.01, 0.02, 0.03, 10.0) give large CVm,j . Spectrum (0.91, 0.08, 0.01, . . .) gives (2) (2) high Sspan but weak Uspan .
Clean two-dimensional span
Spectrum (0.50, 0.45, 0.03, 0.02, . . .) (2) gives nontrivial Uspan and (2)
Long-tailed span
Insufficient retained evidence Many zero effects
Dspan = 0.03/0.45. Spectrum (0.35, 0.30, 0.25, 0.10, . . .) has high small-k sufficiency but (2) Dspan = 0.25/0.30. Only five finite nonzero contexts remain after filtering. Most active contexts are removed by τ0 before normalization.
Label pressure Directed-family label plus magnitude-instability flag.
Trap avoided Reading directional stability as uniformly reliable control. Two dimensions suffice, but Inferring a genuine the evidence is mostly 1D. plane from sufficiency alone. The second component is Collapsing a real plane substantive and followed by into a ray or axis. a drop. The third component weakens a clean 2D label. Report insufficient effect evidence, not a geometry family. Retained directions may be coherent, but only for the nonzero subset.
Turning a broad spectrum into a crisp low-D claim. Letting sample noise choose the classifier. Treating rare measurable effects as feature-wide consistency.
Table 11: Illustrative FEGA profiles. These examples separate span sufficiency, component use, post-k spectral drop, and magnitude variation. Together, they show why a geometry label should be qualified by strength, tail, and evidence flags. residual gates test whether a low-dimensional claim is clean rather than driven by a dominant component or a long spectral tail. For global spans, FEGA tests k ∈ {2, 3, 4, 8} in increasing order and selects the smallest supported dimension. The strict centered-residual test instead considers k ∈ {2, 3, 4}. A one-dimensional residual may still be retained as a descriptive fallback when its residual-energy and sufficiency conditions hold, but it does not carry the same evidential status. If neither a strict test nor an informative fallback is supported, FEGA does not force a low-dimensional label. FEGA records long-tail spectra using Ltail = rspan,ent /(rspan,PR + ϵ); high values warn that many small spectral components complicate a crisp rank interpretation. Each strict full-sample label is accompanied by selected-family confidence and a separate evidence-availability status. Complete evidence with no observed crossing gives accepted confidence, or exploratory confidence when the context count is below 32. Any completed boundary crossing, principal-angle failure, or selected-dimension mismatch gives unstable confidence. If required evidence is unavailable and no instability is observed, no confidence value is assigned; observed instability may coexist with unavailable auxiliary evidence. Fallback and terminal results retain their pointstate confidence and are marked as not evaluated. Long-tail spectra and magnitude heterogeneity remain separate secondary descriptors.
I
Illustrative Geometry Profiles
This appendix collects the FEGA reporting tables used to make the diagnostic procedure interpretable. Table 7 states the evidence gates that determine whether a sampled effect cloud is usable for a geometry claim. Table 8 gives the directional-family gates used to distinguish directed rays, unsigned axes, and multiple coherent modes. Table 10 gives the span and residual gates used to assess low-dimensional sufficiency, component use, spectral drops, and residual variation. Table 11 then gives small schematic spectra and context summaries showing why FEGA reports several diagnostics rather than collapsing the effect cloud to a single scalar. These examples are illustrative, not additional experiments; they clarify how sufficiency, component use, spectral tails, residual variation, zero-effect filtering, and magnitude instability affect the final reporting label.
32