Transformers with Selective Access to Early Representations
arXiv:2605.03953v1 [cs.LG] 5 May 2026
Skye Gunasekaran UC Santa Cruz [email protected]
Téa Wright UC Berkeley [email protected]
Rui-Jie Zhu UC Santa Cruz [email protected]
Jason Eshraghian UC Santa Cruz [email protected]
Abstract Several recent Transformer architectures expose later layers to representations computed in the earliest layers, motivated by the observation that low-level features can become harder to recover as the residual stream is repeatedly transformed through depth. The cheapest among these methods add static value residuals: learned mixing coefficients that expose the first-layer value projection V1 uniformly across tokens and heads. More expressive dense or dynamic alternatives recover finer-grained access, but at higher memory cost and lower throughput. The usefulness of V1 is unlikely to be constant across tokens, heads, and contexts; different positions plausibly require different amounts of access to early lexical or semantic information. We therefore treat early-representation reuse as a retrieval problem rather than a connectivity problem, and introduce Selective Access Transformer (SATFormer), which preserves the first-layer value pathway while controlling access with a context-dependent gate. Across models from 130M to 1.3B parameters, SATFormer consistently improves validation loss and zero-shot accuracy over the static value-residual and Transformer baselines. Its strongest gains appear on retrieval-intensive benchmarks, where it improves over static value residuals by approximately 1.5 average points, while maintaining throughput and memory usage close to the baseline Transformer. Gate analyses suggest sparse, depth-dependent, head-specific, and category-sensitive access patterns, supporting the interpretation that SATFormer learns selective reuse of early representations rather than uniform residual copying. Our code is available at github.com.
1
Introduction
Information flow across depth has motivated a broad family of architectural modifications to the Transformer. Although the residual stream provides a direct pathway for propagating token representations, each layer transforms those representations through attention, normalization, and feed-forward computation. Prior work has argued that low-level lexical, structural, and token-specific features computed in early layers can become difficult to recover deeper in the network [Zhou et al., 2025]. A natural response is to expose later layers to earlier representations, specifically the first-layer value projection V1 , through additional cross-layer pathways. This idea underlies work on residual scaling and normalization, gated residuals, dense layer aggregation, and dynamic cross-layer routing [Bachlechner et al., 2021, Touvron et al., 2021, Wang et al., 2024, Srivastava et al., 2015, Pagliardini et al., 2024, Xiao et al., 2025, Zhu et al., 2024]. Preprint.
These methods largely share an underlying framing: cross-layer reuse is a connectivity problem, where the design question is how many pathways to add between layers and how richly to combine them. The simplest instantiation, value residual learning (ResFormer) [Zhou et al., 2025], exposes V1 to all subsequent layers through a learned scalar coefficient per layer. This mechanism is computationally cheap, but it applies the same amount of V1 to every token and every attention head within a layer. More expressive methods such as DenseFormer, MUDDFormer, and HyperConnections recover finer-grained access by aggregating over wider layer histories or generating dynamic dense pathways, but these gains come with higher memory cost and lower throughput [Pagliardini et al., 2024, Xiao et al., 2025, Zhu et al., 2024]. This leaves a more targeted question: can a single early-value pathway become competitive if the model learns control over when and where that pathway is used? The usefulness of V1 should vary across tokens, heads, layers, and contexts: a token that depends on prompt-specific lexical or semantic information may benefit from direct access to early features, while another token or head may not need that access at all. A layer-wise scalar cannot express this variation, because it exposes or suppresses V1 broadly across the layer. To put this into practice, we introduce Selective Access Transformer (SATFormer), in which a computationally cheap per-token, per-head gate modulates how much of V1 enters each layer’s value computation. Empirically, this control-based design yields favorable performance-efficiency trade-offs. Across 130M-1.3B parameter models, SATFormer consistently improves validation loss and zero-shot accuracy over the baselines. The gains are particularly visible on retrieval-intensive evaluations, where SATFormer outperforms ResFormer by nearly 1.5 average points. At the same time, SATFormer maintains throughput and memory usage close to the Transformer and ResFormer baselines, while improving on their perplexity at all scales. We further analyze the learned gating behavior to understand how SATFormer uses its retrieval pathway. Our contributions are as follows: • A control-based framing for early-representation reuse. We treat V1 access as a question of control across tokens, heads, and depth over a single early pathway, rather than connectivity over many pathways. • The SATFormer architecture. SATFormer replaces ResFormer’s per-layer scalar with a per-token, per-head gate from a single linear projection of the current hidden state, adding negligible parameters and runtime cost. • Pareto-favorable scaling and retrieval. SATFormer improves validation loss over Transformer and ResFormer at every evaluated scale (130M–1.3B), with the largest zero-shot gain over ResFormer at 1.3B (+0.67 average accuracy). On retrieval-intensive benchmarks, it gains approximately 1.5 points over ResFormer and matches or exceeds dense alternatives such as MUDDFormer at roughly 1.8× their throughput. • Mechanistic evidence for structured access. Gate analyses reveal sparse, depth-dependent, head-specific, and token-specific access, concentrated in a subset of heads. Ablation and logit-lens experiments confirm that both the V1 pathway and the input-dependent structure of the gate are functionally consequential to language modeling performance.
2
Related Work
Prior work improves depth-wise information flow by changing either the stability of the residual stream or the amount of cross-layer history available to later computation. Residual scaling and normalization methods such as ReZero [Bachlechner et al., 2021], LayerScale [Touvron et al., 2021], and DeepNorm [Wang et al., 2024] improve optimization and signal propagation without adding explicit access to a large layer history. Gated residual mechanisms such as Highway networks [Srivastava et al., 2015] similarly introduce control over residual updates. A second line of work increases cross-layer access more directly: multi-stream recurrence methods such as HyperConnections [Zhu et al., 2024] and mHC [Xie et al., 2025] expand the residual stream into multiple pathways, while aggregation and routing methods such as DenseNet-style connectivity [Huang et al., 2017], ELMo-style learned layer mixtures [Peters et al., 2018], DenseFormer [Pagliardini et al., 2024], MUDDFormer [Xiao et al., 2025], MRLA [Fang et al., 2023], LAuReL [Menghani et al., 2024], and Attention Residuals [Team et al., 2026] combine information from earlier layers more explicitly. 2
Q2
K2
…
…
… Q2
V2
K2
V2
X1
X1
Q1
K1
V1
Q1
K1
⨁
DWA Conv Attention
Proj
…
…
Attention/FFN V1
X0
X0
a. Resformer
b. SATFormer
…
⨁ ⨁
⨁
…
Q
K
V
DA Module
h1
h2
c. Hyperconnections
X:i
Xi
d. Muddformer
R
X1
Attention
X0
e. Denseformer
Figure 1: ResFormer provides efficient but layer-wise static access to V1 ; DenseFormer, MUDDFormer, and HyperConnections increase cross-layer access through denser or more dynamic pathways at substantially higher cost. SATFormer preserves the single early-value pathway of ResFormer but makes access token- and head-dependent through a computationally cheap gate, achieving greater adaptability without increasing cross-layer connectivity. These methods trace an efficiency–granularity frontier: static scalar connections are inexpensive but coarse, whereas dense aggregation and dynamic routing provide finer-grained access at higher memory cost and lower throughput. SATFormer targets a different point on this frontier: adaptive control over a single early-value pathway, rather than denser access to many previous representations. The closest prior work to SATFormer is value residual learning (ResFormer) introduced by [Zhou et al., 2025]. ResFormer addresses the dilution of initial token information by adding a direct connection from the first-layer value projection to the value vectors of subsequent layers. For a Transformer layer n > 1, the modified value vector is computed as Vn′ = λn1 V1 + λn2 Vn
(1)
where V1 is the value representation from the first layer, λn1 is typically a learned layer-wise scalar eλ
′
initialized to λscale · PNn,1eλ′ where λscale is initialized to the total layers N and λn2 is held constant j=1
j,1
across layers. Note, in the definition above we refer to the highest performing ResFormer variant, Learnable ResFormer Plus. We focus our empirical comparisons on four methods that instantiate distinct points along the efficiency–granularity frontier. Value residual learning [Zhou et al., 2025] addresses the dilution of initial token information by adding a direct residual connection to the value vectors of all subsequent layers, using learned layer-wise scalar coefficients. HyperConnections [Zhu et al., 2024] provides a learnable alternative to traditional residuals by replicating hidden states into multiple pathways. MUDDFormer [Xiao et al., 2025] extends this to the most flexible setting, generating dense connection weights dynamically for each sequence position via a lightweight MLP. DenseFormer [Pagliardini et al., 2024] introduces a Depth-Weighted-Average module after each Transformer block, computing a weighted average over all preceding representations. SATFormer targets a narrower and more controlled design point. Rather than aggregating over all previous layers, expanding the residual stream, or routing across a dense layer history, it asks whether a single early-value pathway can become competitive when access to that pathway is made input-dependent. SATFormer does not increase the amount of cross-layer history available to the model; it increases the model’s control over when and where one early representation is reused.
3
Methodology
Previous work exposes V1 through layer-wise scalar coefficients, making early-value access uniform across tokens and heads within a layer [Zhou et al., 2025]. To enable more dynamic input-aware gating, we replace the static scalar λn in Equation 1 with an input-dependent mixing coefficient. This (n) allows the model to perform context-aware value retrieval. Let xt ∈ Rdmodel denote the normalized 3
hidden state for the token at position t in layer n. SATFormer computes a per-token, per-KV-head (n) gate αt,j as h i (n) (n) αt,j = ReLU xt Wα(n) , j
(2)
(n)
where Wα ∈ Rdmodel ×Nkv is a single linear projection, Nkv is the number of key-value heads, and j ∈ {1, . . . , Nkv } indexes the KV head. The value used by attention is then (n)
(1)
′ Vt,j,r = Vt,j,r + αt,j · Vt,j,r ,
(3)
where r indexes the per-head value dimension. The ReLU activation keeps the mixing coefficient non-negative and allows the gate to become exactly zero. A deeper comparison of gating mechanisms can be found in Appendix B.1. Design rationale. SATFormer makes three deliberately minimal choices. First, the gate is per-token and per-head because the usefulness of V1 varies along both axes: a token-level gate alone would ignore head-level specialization, while a head-level gate alone would make access uniform across the sequence, preventing the model from selecting particular tokens whose early representations should be preserved. Second, the gate is computed with a single linear projection from the current normalized hidden state. This makes access depend on the representation already available to the layer, while adding only dmodel Nkv parameters per layer. Third, we use a ReLU gate so that access is non-negative and can be exactly zero. Non-negativity keeps the pathway interpretable as additive reuse of the first-layer value stream, while exact zeros allow the model to fully disengage from V1 when early information is unhelpful. Further ablation studies regarding gating functions can be found in Appendix B.1. This construction preserves the same early-value source as ResFormer, but changes the access policy: rather than applying a single layer-wise coefficient, SATFormer lets each token and KV head choose how much of V1 to reuse.
4
Experiments
We organize our evaluation around the central claim of this paper: selective access to early value representations provides an efficient alternative to dense cross-layer connectivity, with particular benefits for in-context retrieval and language-modeling performance. We therefore begin with retrieval-intensive benchmarks, where selective access should be most directly useful. We then evaluate wall-clock efficiency, throughput, and memory usage to test whether SATFormer preserves the computational profile of sparse connectivity. Finally, we report standard language modeling, zero-shot, and scaling results to verify that these targeted gains do not come at the cost of model quality. Pretraining setup. Along with SATFormer, our experiments include Transformer, ResFormer, DenseFormer, MUDDFormer, and HyperConnections as baselines. We train all models under matched data, optimizer, and hardware conditions across three main scales: Small (130M parameters, 5B tokens, η = 10−3 ), Medium (340M parameters, 10B tokens, η = 1.5 × 10−3 ), and Large (760M parameters, 20B tokens, η = 2 × 10−3 ). We additionally train Transformer, ResFormer, and SATFormer at the XL scale (1.3B parameters, 30B tokens) to study scaling relative to the closest static value-residual baseline. All models are trained on 2 NVIDIA H200 GPUs, with configurations closely following Yang et al. [2024]. We train on a randomly sampled subset of FineWeb-Edu and use the Llama-2 tokenizer with a vocabulary size of 32,000. Training uses a sequence length of 4K tokens and the AdamW optimizer with weight decay 0.1 and gradient clipping 1.0. The learning rate η follows a cosine annealing schedule, decaying to a minimum of 0.1 × ηbase after a linear warmup period covering the first 1% of training tokens. Model configurations. Table 1 reports the model configurations used at each scale. For each baseline, we use the strongest or recommended variant from the corresponding prior work. DenseFormer uses a dilation factor of 4 and a Depth-Weighted-Average (DWA) period of 5. MUDDFormer uses 4
full Query, Key, Value, and Residual connectivity with parameter reallocation and pre/post Depth Aggregation normalization. HyperConnections uses the dynamic connection variant with expansion rate n = 2, tanh activation, and layer normalization. ResFormer follows the Learnable ResFormer Plus variant defined by the weighted residual sum in Equation 1. 4.1
Retrieval-Intensive Evaluation
Table 1: Model Configurations.
We first evaluate SATFormer on retrieval-intensive benchmarks, where preserving prompt-specific information should be especially useful. As shown in Table 2, SATFormer achieves the highest average score among the evaluated architectures, narrowly surpassing MUDDFormer and improving over ResFormer by approximately 1.5 average points.
VARIANT
dmodel
L
nheads
df f
S MALL M EDIUM L ARGE XL
768 1024 1536 2048
11 18 19 24
12 16 24 32
3072 4096 6144 5632
This comparison is particularly informative because SATFormer and ResFormer both expose the first-layer value stream. Their main difference is that ResFormer mixes this stream using layer-wise static coefficients, whereas SATFormer conditions access on the current hidden state at the token and head level. The observed improvement therefore suggests that the benefit is not simply due to making V1 available, but may also depend on controlling when and where the early representation is reused. Table 2: Performance on retrieval-intensive tasks. Input length was truncated to 2k tokens during evaluation. Evaluation protocol follows Arora et al. [2024]. M ODEL T RANSFORMER D ENSE F ORMER MUDDF ORMER H YPER C ONNECTIONS R ES F ORMER SATF ORMER
T RIVIAQA
SWDE
SQUAD
NQ
FDA
D ROP
AVG
52.9 52.66 54.32 55.27 53.9 54.26
29.43 32.31 36.81 25.83 27.9 32.4
34.91 35.32 34.78 36.72 34.88 36.96
12.06 11.24 12.06 12.82 11.46 12.16
19.14 16.42 20.32 15.78 19.41 21.96
20.07 22.66 20.84 24.86 22.8 21.41
28.085 28.435 29.855 28.546 28.392 29.858
Importantly, SATFormer achieves this retrieval performance while preserving a computational profile close to the baseline Transformer and ResFormer. In contrast, MUDDFormer attains a similar retrieval average but trains substantially more slowly under matched hardware. These results are consistent with the hypothesis that selective early-value access is useful in settings where performance depends on retaining prompt-specific lexical or semantic information, while avoiding the higher cost of dense cross-layer connectivity. 4.2
Efficiency Benchmarking
Train Loss vs. Wall-Clock Time ISO-compute (10h)
SATFormer (ours) MUDDFormer HyperConnections Transformer ResFormer DenseFormer
3.5 3.0
ISO-Compute Checkpoint (Zoomed)
2.60
Model
Train Loss
Train Loss
4.0
2.5
2.55 2.50 2.45
0
10
20
30
40
Wall-clock time (hours)
50
60
7
8
9
10
11
Wall-clock time (hours)
12
13
All models compared at 10h in wall-clock time ( Transformer step 11,693). HyperConnections and MUDDFormer train ~3× slower, completing only ~15% of training by this point.
Figure 2: (Left) Train loss plotted against wall-clock time in hours. The dashed vertical line indicates the 10-hour wall-clock checkpoint used for comparison. (Right) A zoomed-in view of the 10-hour checkpoint highlighting the loss differentials. Metrics computed from the 760M training run. Final validation loss and zero-shot accuracy do not fully capture the practical cost of dense crosslayer connectivity. For instance, architectures that expand the residual stream, aggregate over many previous layers, or generate dynamic dense pathways may improve endpoint performance while 5
making substantially slower wall-clock progress. We therefore evaluate training efficiency directly, focusing on wall-clock loss, throughput, and memory usage under matched hardware. Figure 2 compares training loss as a function of wall-clock time. At the 10-hour checkpoint, SATFormer achieves a loss of 2.4515 compared to 2.4557 for ResFormer, 2.4703 for the baseline Transformer, and 2.4965 for DenseFormer. In contrast, HyperConnections and MUDDFormer make substantially slower wall-clock progress in this setup, reaching losses of 2.5396 and 2.5672 at the same checkpoint. Table 3: Training Throughput (toks/sec) & Memory Table 3 reports memory usage and through- Usage (GB). Metrics computed at the 760M scale. put at the 760M scale with a batch size 16, 4 gradient accumulation steps, sequence M ODEL M EMORY T HROUGHPUT length 4096, 2 NVIDIA H200 GPUs, and T RANSFORMER 85.76 170,308 5 warmup steps. SATFormer maintains D ENSE F ORMER 93.32 145,639 throughput close to the baseline TransMUDDF ORMER 90.52 90,443 former and ResFormer while using comH YPER C ONNECTIONS 124.73 94,534 parable memory. In terms of efficiency, R ES F ORMER 89.59 163,826 SATFormer achieves roughly 1.75× higher SATF ORMER 86.03 164,839 throughput and uses 31% less memory than HyperConnections, 1.82× higher throughput and 5% less memory than MUDDFormer, and 4% less memory than ResFormer. Thus, while dense dynamic methods and static value residuals can achieve strong endpoint performance, SATFormer provides a more favorable wall-clock Pareto trade-off in throughput-constrained training regimes. 4.3
General Language Modeling and Zero-Shot Evaluation Table 4: Final Validation Loss Values.1 We next evaluate whether SATFormer’s retrieval and efficiency VARIANT TF DF MF HC RF SAT gains come at the cost of gen2.707 2.704 2.658 2.670 2.674 2.648 eral language-modeling quality. S MALL M EDIUM 2.461 2.452 2.415 2.410 2.429 2.415 Table 4 reports final validation 2.256 2.256 2.233 2.234 2.246 2.242 loss, while Table 5 reports zero- L ARGE XL 2.152 N/A N/A N/A 2.150 2.139 shot performance across language modeling and downstream benchmarks. These broader evaluations serve as a general-purpose validation of SATFormer beyond the targeted retrieval and efficiency settings. We first focus on the comparison between SATFormer and ResFormer, since this isolates the effect of replacing static value-residual access with input-dependent selective access. SATFormer improves final validation loss over ResFormer at every evaluated scale: 2.648 vs. 2.674 at 130M, 2.415 vs. 2.429 at 340M, 2.242 vs. 2.246 at 760M, and 2.139 vs. 2.150 at 1.3B. This suggests that selective access to V1 improves the training objective relative to static value mixing without requiring dense connectivity. This is the central efficiency advantage of selective access: it increases the granularity of cross-layer reuse without substantially increasing the amount of cross-layer connectivity. One result warrants direct acknowledgment: while SATFormer improves over ResFormer at every evaluated scale, denser alternatives still achieve higher endpoint zero-shot accuracy in some settings. At the Medium scale, HyperConnections outperforms SATFormer in average zero-shot accuracy (46.04 vs. 45.81), and at the Large scale, MUDDFormer and HyperConnections achieve higher average scores (49.67 vs. 48.81). However, SATFormer achieves roughly 1.75 − 1.82× higher throughput than these denser alternatives in our efficiency benchmark. This is consistent with SATFormer’s design intent: it is not meant to uniformly dominate all dense connectivity mechanisms on endpoint accuracy, but to provide a favorable Pareto trade-off between adaptivity, model quality, and training efficiency. Scaling behavior. The scaling results on core language modeling suggest that SATFormer’s benefits may depend not only on parameter count, but also on the number of opportunities the model has to reuse V1 across depth. Notably, the XL setting (1.3B parameters) shows SATFormer’s strongest zeroshot gains over ResFormer (+0.67 average accuracy), consistent with deeper models providing more layers at which selective V1 access can be applied. This interpretation is further supported by the gate analysis in Section 5, where V1 utilization remains sparse in early layers and increases substantially 1 TF: Transformer, DF: DenseFormer, MF: MUDDFormer, HC: HyperConnections, RF: ResFormer, SAT: SATFormer.
6
Table 5: Zero-shot performance across language modeling and downstream tasks. Best results are bolded. Task abbreviations: Wiki (WikiText-103), LMB (Lambada), PIQA (Physical IQA), Hella. (HellaSwag), Wino. (Winogrande), ARC-e/c (ARC Easy/Challenge), SIQA (Social IQA). M ODEL
W IKI . PPL ↓
LMB. PPL ↓
LMB. PIQA ACC ↑ ACC ↑
T RANSFORMER D ENSE F ORMER MUDDF ORMER H YPER C ONNECTIONS R ES F ORMER SATF ORMER
41.81 41.80 39.45 38.45 39.68 37.98
121.31 134.53 115.12 95.54 106.80 84.23
22.26 21.62 21.70 23.71 23.38 25.79
T RANSFORMER D ENSE F ORMER MUDDF ORMER H YPER C ONNECTIONS R ES F ORMER SATF ORMER
29.46 28.92 27.97 26.83 27.84 26.98
59.29 49.68 49.44 35.21 42.38 45.84
27.38 29.17 27.77 33.86 30.47 28.90
T RANSFORMER D ENSE F ORMER MUDDF ORMER H YPER C ONNECTIONS R ES F ORMER SATF ORMER
21.95 21.77 21.07 21.14 21.71 21.24
24.27 24.57 22.30 21.63 25.06 24.92
36.72 37.16 38.11 38.36 37.03 36.72
T RANSFORMER R ES F ORMER SATF ORMER
18.83 18.62 18.47
18.66 18.28 16.44
40.38 40.55 43.56
H ELLA . ACC _ N ↑
W INO . ARC- E ACC ↑ ACC ↑
ARC- C ACC _ N ↑
SIQA B OOL Q ACC ↑ ACC ↑
AVG . ↑
25.51 23.89 23.81 25.68 23.89 25.00
37.00 35.93 36.28 37.25 35.93 36.49
62.08 60.37 57.92 55.41 61.41 59.88
42.15 41.70 41.66 41.80 41.94 42.71
24.57 25.68 26.11 26.70 25.68 27.22
38.02 37.87 37.15 36.74 38.18 37.36
62.14 61.01 59.33 61.89 60.55 61.44
44.71 44.84 44.80 46.04 45.62 45.81
61.11 61.70 61.74 60.90 59.97 61.91
30.89 29.52 30.80 30.03 29.61 29.86
37.92 39.15 38.84 39.25 39.25 38.79
60.61 55.35 58.38 62.17 60.15 58.96
48.67 48.44 49.67 49.67 48.79 48.81
65.61 65.48 65.69
32.93 33.87 32.59
39.71 39.66 40.63
56.17 58.53 59.48
51.23 51.29 51.96
S MALL : 130M PARAMS / 5B TOKENS 61.21 61.32 60.88 61.31 60.07 61.97
30.57 30.90 31.48 31.63 31.00 32.12
51.85 50.43 52.49 50.98 51.30 50.91
46.68 49.12 48.70 48.44 48.53 49.54
M EDIUM : 340M PARAMS / 10B TOKENS 65.07 65.40 65.72 64.85 64.74 65.89
36.71 37.01 37.04 38.72 38.03 38.30
50.83 48.15 51.07 51.06 51.46 51.07
52.95 54.42 54.17 54.54 55.85 56.27
L ARGE : 760M PARAMS / 20B TOKENS 67.14 68.17 68.28 67.84 66.87 68.34
44.20 44.28 45.53 45.93 44.81 45.25
50.75 52.17 55.64 52.95 52.64 50.67
XL: 1.3B PARAMS / 30B TOKENS 69.85 69.36 68.49
54.85 52.88 54.30
50.40 50.02 51.01
in deeper layers, and by the additional 760M thin-and-long study in Appendix B.2, which shows that reallocating capacity toward depth improves SATFormer more than ResFormer. These results suggest that the comparatively muted 760M endpoint result is at least partly configuration-dependent, and that selective access benefits from depth as well as scale. A more complete scaling study separating the effects of depth, width, and training tokens remains an important direction for future work.
5
Mechanistic Analysis
Mean Gate (α) per Layer × Head 18 17 16
5.1
15 14 13
15
12 11
10
10 9
5
8
Mean α
Layer index
The goal of this section is to test a narrower mechanistic prediction of the architecture: if selective access is doing useful work, the learned gate should not behave like a dense residual shortcut. Instead, access to V1 should be sparse, concentrated in particular heads or layers, and functionally important when perturbed. We focus on three questions: First, is access to V1 sparse or broadly active? Second, which layers are most functionally sensitive to perturbing the learned gate, and how does this compare to where gate activation is largest? Third, do different token categories receive different amounts of access?
7 6
0
5 4 3 2 1 0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23
KV Head index
Figure 3: Mean gate activation across layers and KV heads. Early layers show little access to V1 , while a subset of later-layer heads activates sharply, indicating that SATFormer uses the early-value pathway selectively rather than uniformly.
Sparse and Depth-Dependent Access
Figure 3 shows the mean gate activation across layers and KV heads. The learned access pattern is highly non-uniform. In the early and middle layers, most heads assign little or no weight to V1 , 7
indicating that SATFormer does not simply propagate the first-layer value stream through the network as a dense residual shortcut. Instead, access remains sparse for much of the model depth. This pattern rules out the simplest interpretation of SATFormer as merely learning a rescaled value residual. If the gate were only compensating for the absence of ResFormer’s layer-wise coefficient, access would be distributed broadly across heads within each layer. Instead, the model allocates V1 access sparsely: most heads remain close to inactive, while a smaller set of later-layer heads carries most of the early-value reuse. Thus, SATFormer learns a division of labor across heads and depth, where the early-value pathway is reserved for specialized computations rather than used as a uniform shortcut. 5.2
Gate Intervention Analysis
As shown in Figure 4, both interventions degrade performance relative to the unablated SATFormer. Zeroing the gate tests whether access to V1 matters at all, while replacing the gate with its mean preserves the average scale of access but removes token- and head-dependent selectivity. The fact that both interventions increase perplexity shows that SATFormer benefits from both components: the early-value pathway itself and the learned structure controlling when that pathway is used. Gate Ablation: ΔPPL per Layer (baseline = 9.4496) 1.75
Transformer SATFormer
105
1.25
Perplexity (log scale)
ΔPPL vs. baseline
Logit Lens: Per-Layer Perplexity
Zero gate (V residual removed) Mean gate (constant mixing)
1.50
1.00 0.75 0.50 0.25
104
Performance Crossover (SATFormer < Transformer)
103
102
0.00 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
0
Layer index
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18
Layer index
Figure 5: SATFormer begins to outperform the baseline Transformer in layer-wise perplexity near the same depth range where V1 gate activation increases.
Figure 4: We measure the change in perplexity after either zeroing the gate at a single layer or replacing the input-dependent gate with its mean value. Ablating some earlier layers produces larger perplexity increases.
The intervention results also show that gate magnitude alone is not a complete measure of functional importance. Later layers have the largest average gate activations, but some earlier layers produce large perplexity increases when perturbed. This indicates that small early uses of V1 can affect representations that are amplified or reused later in the network. The logit-lens probe in Figure 5 is consistent with this picture: SATFormer initially lags the baseline Transformer, then crosses over in the same later-depth regime where gate activation becomes stronger. Together, the ablation and logit-lens results show that selective access is not only visible in the gate statistics, but also consequential for model predictions. 5.3
Category-Specific Access
Finally, we ask whether the gate treats different kinds of early features differently. Because V1 itself is high-dimensional and not directly labeled, we use sparse autoencoders as a coarse labeling tool: SAE features with similar top-activating tokens are grouped into broad semantic, functional, and structural categories. This analysis should be read as a descriptive probe of which feature types receive more access, not as a causal decomposition of SATFormer’s downstream gains. Details of the SAE training and labeling pipeline are given in Appendix C. Figure 6 shows that category differences are small in early layers, where access to V1 is weak overall. In later layers, however, features labeled as semantic receive higher average gate activation than structural or functional features. This provides a more interpretable view of the sparse late-layer access pattern from Figure 3: the heads that re-access V1 are not only active late in depth, but also tend to assign more access to content-bearing early features. This does not establish that semantic 8
Category-Specific Gating Intensity 7 6
Semantic Functional Structural
Mean α
5 4 3 2 1 0
2
4
6
8
10
12
14
16
18
Layer index
Figure 6: Gating is nearly uniform in the first half of the model, where access to V1 is generally weak. In later layers, semantic features receive stronger V1 access than structural and functional features, suggesting that SATFormer preferentially re-accesses content-bearing early representations.
gating causes the retrieval gains in Table 2, but it supports the intended picture of SATFormer as selectively reusing early token information rather than uniformly copying the first-layer value stream. Across these analyses, SATFormer’s learned gate behaves as a selective and functionally consequential access mechanism. Access to V1 is sparse in early layers and concentrated in a subset of later-layer heads; per-layer ablations show that disrupting the gate increases perplexity, with especially large effects in some earlier layers; logit-lens analysis shows that SATFormer begins to outperform the baseline Transformer near the later-layer gating transition; and category-level analysis shows stronger access for semantic than structural features. These findings support the interpretation of SATFormer as a computationally cheap mechanism for targeted early-representation reuse; however, we do not claim full causality and leave fine-grained decomposition as future work.
6
Conclusion
We introduced SATFormer, a computationally cheap interpretation of value residual learning that replaces static layer-wise access to the first-layer value stream with token- and head-dependent gating. Across 130M to 1.3B parameter models, SATFormer improves validation loss over ResFormer at every evaluated scale and improves average zero-shot accuracy over ResFormer at 130M, 340M, and 1.3B, including a +0.67 point gain at 1.3B. On retrieval-intensive benchmarks, SATFormer achieves the highest average score among all evaluated architectures (29.86 average across TriviaQA, SWDE, SQuAD, NQ, FDA, and DROP), outperforming ResFormer by roughly 1.5 average points and narrowly surpassing MUDDFormer at a fraction of the throughput cost. At the same time, SATFormer maintains throughput and memory usage close to the baseline Transformer and ResFormer, achieving roughly 1.75× higher throughput than HyperConnections and using 31% less memory. These results support a targeted but important conclusion: early representations need not be made available through dense connectivity to be useful. SATFormer occupies a useful point in the architecture design space: substantially cheaper than dense cross-layer routing, more adaptive than static value residuals, and especially effective in retrieval-oriented settings. A single early-value pathway, when equipped with token- and head-dependent control, can provide an efficient mechanism for targeted reuse. SATFormer therefore reframes cross-layer reuse as a problem of selective access rather than maximal connectivity, offering a simple and computationally efficient design point for future Transformer architectures. 9
Broader Impacts and Safeguards This work studies architectural mechanisms for improving information flow in Transformer language models. Our experiments are conducted at relatively small research scales, and the trained models are intended for controlled empirical analysis rather than deployment. As such, the models evaluated in this work are unlikely to directly pose substantial real-world risks on their own. At the same time, SATFormer is a general-purpose architectural modification and could be applied to larger language models. Improvements in language-modeling efficiency or performance may therefore contribute to both beneficial and harmful downstream capabilities, depending on how such models are trained, released, and used. Potential risks include misuse of stronger language models for deception, spam, harmful content generation, or other malicious applications. We therefore emphasize that this work should be used in accordance with standard safeguards for language-model development, including careful dataset curation, capability and misuse evaluations, controlled release practices when appropriate, and monitoring for downstream harms. The intended benefit of this research is to improve the efficiency and interpretability of Transformer architectures. By studying selective access to early representations, we aim to better understand how models reuse information across depth and to identify architectural designs that improve performance without requiring substantially denser or more computationally expensive connectivity.
References S. Arora, A. Timalsina, A. Singhal, B. Spector, S. Eyuboglu, X. Zhao, A. Rao, A. Rudra, and C. Ré. Just read twice: closing the recall gap for recurrent language models. arXiv preprint arXiv:2407.05483, 2024. T. Bachlechner, B. P. Majumder, H. Mao, G. Cottrell, and J. McAuley. Rezero is all you need: Fast convergence at large depth. In Uncertainty in artificial intelligence, pages 1352–1361. PMLR, 2021. H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. Y. Fang, Y. Cai, J. Chen, J. Zhao, G. Tian, and G. Li. Cross-layer retrospective retrieving via layer attention. arXiv preprint arXiv:2302.03985, 2023. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023. G. Menghani, R. Kumar, and S. Kumar. Laurel: Learned augmented residual layer. arXiv preprint arXiv:2411.07501, 2024. M. Pagliardini, A. Mohtashami, F. Fleuret, and M. Jaggi. Denseformer: Enhancing information flow in transformers via depth weighted averaging. Advances in neural information processing systems, 37:136479–136508, 2024. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In M. Walker, H. Ji, and A. Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/ N18-1202. URL https://aclanthology.org/N18-1202/. R. K. Srivastava, K. Greff, and J. Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015. 10
K. Team, G. Chen, Y. Zhang, J. Su, W. Xu, S. Pan, Y. Wang, Y. Wang, G. Chen, B. Yin, et al. Attention residuals. arXiv preprint arXiv:2603.15031, 2026. H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 32–42, 2021. H. Wang, S. Ma, L. Dong, S. Huang, D. Zhang, and F. Wei. Deepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761–6774, 2024. D. Xiao, Q. Meng, S. Li, and X. Yuan. Muddformer: Breaking residual bottlenecks in transformers via multiway dynamic dense connections. arXiv preprint arXiv:2502.12170, 2025. Z. Xie, Y. Wei, H. Cao, C. Zhao, C. Deng, J. Li, D. Dai, H. Gao, J. Chang, K. Yu, et al. mhc: Manifold-constrained hyper-connections. arXiv preprint arXiv:2512.24880, 2025. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. S. Yang, J. Kautz, and A. Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024. Z. Zhou, T. Wu, Z. Jiang, F. Obeid, and Z. Lan. Value residual learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28341–28356, 2025. D. Zhu, H. Huang, Z. Huang, Y. Zeng, Y. Mao, B. Wu, Q. Min, and X. Zhou. Hyper-connections. arXiv preprint arXiv:2409.19606, 2024.
A 1 2 3
SATFormer PyTorch Implementation
import torch import torch . nn as nn import torch . nn . functional as F
4 5 6 7 8 9 10 11
class S A T F o r m e r R e s i d u a l ( nn . Module ) : def __init__ ( self , hidden_size , num_kv_heads , gate = ’ relu ’) : super () . __init__ () self . num_kv_heads = num_kv_heads self . gate = gate # Lightweight linear projection : hidden_size -> num_kv_heads self . alpha_proj = nn . Linear ( hidden_size , num_kv_heads , bias = False )
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
def _apply_gate ( self , logits ) : if self . gate == ’ relu ’: return F . relu ( logits ) elif self . gate == ’ sigmoid ’: return torch . sigmoid ( logits ) elif self . gate == ’ softmax ’: # Competition across heads ; scaled return F . softmax ( logits , dim = -1) * elif self . gate == ’ s oft ma x _s i gm oi d ’: # Combined across - head competition return F . softmax ( logits , dim = -1) * num_kv_heads elif self . gate == ’ tanh ’: return torch . tanh ( logits ) return logits
to match unit i nitial izatio n self . num_kv_heads and per - head gating torch . sigmoid ( logits ) * self .
27 28 29 30 31 32
def forward ( self , v_n , v_1 , hidden_states ) : # Compute token - dependent mixing coefficients alpha # logits shape : [ Batch , Seq , Heads ] logits = self . alpha_proj ( hidden_states . to ( v_n . dtype ) ) alpha = self . _apply_gate ( logits ) . unsqueeze ( -1) # [B , T , H , 1]
33
11
# Apply mixing : V ’ _n = V_n + alpha ( h ) * V_1 return v_n + alpha * v_1 . to ( v_n . dtype )
34 35 36
# --- Integration into Model Forward Pass --# In SATFor merMod el . forward () : 39 V1 = None 40 for layer_idx , layer in enumerate ( self . layers ) : 41 # Each layer returns ( hidden_states , ... , v_base ) 42 # v_base is the raw V projection from the current layer ’s Attention module 43 outputs = layer ( hidden_states , V1 = V1 ) 44 hidden_states = outputs [0] 45 v_base = outputs [ -1] 37 38
46
# Step 1: Capture V1 from Layer 0 if layer_idx == 0: V1 = v_base # Store raw projection for all subsequent layers
47 48 49 50
# Step 2: Propagation # For layers n > 0 , the ’ V1 ’ variable now contains the stored projection . # It is passed into the Attention module of the next layer via ’ layer (...) ’.
51 52 53
B
Ablation Study
B.1
Gating Analysis Ablation: Gating Function Performance
3.6
Gating Type
Loss
3.4
Relu Softmax Softmax + sigmoid Identity Tanh Sigmoid
Final Convergence
3.2 3.0 8000
2.8 2000
4000
Training Step
6000
9500
8000
Figure 7: Impact of Gating Functions on Convergence. Several gating mechanisms used to compute the token-wise mixing coefficient α at the smallest scale (130M parameters and 5B tokens). To identify the optimal mechanism for token-dependent value mixing, we evaluate several candidate (n) gating functions for the coefficient αt,j .We tested six specific gating behaviors (Note: we use the standard PyTorch Kaiming uniform initialization for all linear layers): 1. ReLU (Default): Produces sparse, non-negative weights, allowing the model to completely zero out the connection if/when V1 is redundant. 2. Sigmoid: Provides a smooth, bounded [0, 1] probability-like weight for each head. 3. Softmax: Computes a convex combination across the head dimension, forcing heads to ”compete” for a shared V1 budget (scaled by Nkv to maintain signal parity). 4. Softmax + Sigmoid: A hybrid approach combining across-head competition with an individual per-head on/off gate. 5. Tanh: Allows for subtractive mixing ([−1, 1]), enabling the model to suppress features in the current layer using the original input. 6. Identity: A linear baseline (W h) used to verify the necessity of non-linear activation. 12
As shown in Figure 7, the ReLU and Softmax gates demonstrate the most stable convergence and lowest final loss (although ReLU maintains a significant lead over Softmax). Surprisingly, Sigmoid and Tanh perform the worst, possibly suggesting that bounded gating limits the expressivity enabled with value mixing. Overall, ReLU is simultaneously the simplest and strongest performing gating function, exhibiting a favorable inductive bias for enabling deeper expressivity with value mixing. B.2
Depth vs. Width Study
The main 760M results show that SATFormer improves both validation loss and zero-shot language modeling over ResFormer, however its advantage is less pronounced than at the 130M, 340M, and 1.3B scales. To test whether this behavior reflects the architecture itself or the particular width vs. depth allocation used at 760M, we ran an additional matched-scale experiment using a “thin-and-long” configuration (1536 dmodel , 24 L, 16 nheads , 4352 df f ). This configuration reallocates capacity toward depth while remaining in the same approximate parameter regime. While both value-residual architectures benefit from the deeper allocation, SATTable 6: Depth vs. Width Final Validation Loss Former benefits more strongly. In Table 6, ResFormer improves from 2.2461 M ODEL VALIDATION L OSS to 2.2396, whereas SATFormer improves R ES F ORMER -L ARGE -T HIN 2.2396 from 2.2424 to 2.2313, corresponding to R ES F ORMER -L ARGE 2.2461 approximately a 1.7× larger reduction for SATF ORMER -L ARGE -T HIN 2.2313 SATFormer. The resulting SATFormerSATF ORMER -L ARGE 2.2424 Large-Thin model also outperforms the corresponding ResFormer-Large-Thin model by 0.0083 validation loss. Table 7: Depth vs. Width zero-shot performance across language modeling tasks. M ODEL
W IKI . LMB. PPL ↓ PPL ↓
R ES F ORMER SATF ORMER
20.73 20.42
21.81 22.04
LMB. PIQA ACC ↑ ACC ↑
H ELLA . ACC _ N ↑
W INO . ARC- E ACC ↑ ACC ↑
ARC- C ACC _ N ↑
SIQA B OOL Q ACC ↑ ACC ↑
AVG . ↑
39.01 38.19
46.61 46.58
51.54 52.64
31.06 31.74
39.10 39.36
59.76 58.47
49.53 49.81
67.85 68.17
61.32 63.30
The zero-shot results in Table 7 show a similar reversal: in the original 760M configuration, ResFormer narrowly outperformed SATFormer in average accuracy, but under the thin-and-long allocation SATFormer takes a larger step forward, improving to 49.81 average accuracy compared with 49.53 for ResFormer. These results support the interpretation that SATFormer benefits disproportionately from depth. This is consistent with the main mechanistic analysis, where gate activation is sparse in early layers and becomes stronger and more head-specialized in later layers. The thin-and-long study suggests that the weaker relative 760M endpoint result is not evidence against the selective-access mechanism, but rather reflects sensitivity to the allocation of parameters between depth and width.
C
Additional Mechanistic Interpretability
This appendix provides additional details for the interpretability analysis in Section 5. The main paper uses three pieces of evidence: sparse depth-dependent gating, logit-lens behavior, and categoryspecific gate activation. Here, we describe how the token-feature categories were constructed and provide supplementary analyses showing that the broad content of V1 is similar across models, while SATFormer changes how those early features are accessed. C.1
SAE Training and Feature Labeling
To obtain coarse categories for V1 features, we train Sparse Autoencoders (SAEs) [Cunningham et al., 2023] on the first-layer value activations of the 760M-parameter Transformer and SATFormer models. The SAEs use a Top-K activation function with k = 32 and an expansion factor of 16. We train both models for 100M tokens using Adam with learning rate 5 × 10−4 and a cosine annealing schedule. The resulting SAEs achieve Fraction of Variance Explained above 99%, indicating that the sparse codes preserve most of the variance in the underlying activation space. 13
We use the SAEs only to construct a coarse probe of which types of early features receive stronger gate activation. The goal is not to provide a complete semantic decomposition of V1 , but to distinguish broad classes of early representations that may be accessed differently by SATFormer. After training, we identify and categorize high-confidence features using a three-stage automated pipeline: 1. Density filtering and MonoScore ranking. We remove dead or ultra-sparse features with fewer than 10 activations, as well as overly dense features with activation density above 0.5%. For each remaining feature, we compute a MonoScore based on the similarity structure of its top activating tokens. 2. LLM consistency scoring. We select the top 20% of features by MonoScore for qualitative evaluation. Using Qwen3-30B-A3B-Instruct-2507-FP8 [Yang et al., 2025] served with vLLM [Kwon et al., 2023], we perform a consistency check over two non-overlapping sets of highly activating tokens. A feature is retained only if both subsets receive a feature-purity score of at least 7 out of 10. 3. Taxonomic categorization. Retained features are assigned to a predefined taxonomy of linguistic and structural categories. The taxonomy is designed to capture broad distinctions relevant to early-token representations, including semantic fragments, structural tokens, code-related tokens, and tokenizer artifacts. For visual clarity, we aggregate the fine-grained SAE feature labels in Table 8 into three broader categories used in the main-text analysis: 1. Semantic: Semantic Root Fragments, Entity Marker Fragments, Logic Quantifier Keywords, Morphological Suffixes, and Functional Leading Space. 2. Functional: Multilingual Unicode, Byte-Level Fallbacks, and Numerical Digit Fragments. 3. Structural: Structural Punctuation and Code Formatting Symbols. Table 8: Taxonomy of SAE feature categories for V1 activations. Feature Category Description Examples Semantic Root Fragments Entity Marker Fragments Morphological Suffixes Logic Quantifier Keywords Functional Leading Space Numerical Digit Fragments Structural Punctuation Multilingual Unicode Code Formatting Symbols Byte-Level Fallbacks
C.2
Core meaning-bearing subwords Capitalized fragments often used in proper names Subwords that modify parts of speech Connectives and quantifiers that direct sentence flow Tokens that represent the start of a new word Numbers, decimals, or ordinal indicators Symbols used for syntax, grouping, or boundaries Valid non-ASCII scripts Programming syntax or whitespace/indentation Raw UTF-8 bytes or non-standard characters
struct, trans, ology Mc, San, Al ing, ed, s, ly if, however, all, but The, Apple, Run 12, 4, .5, th ., :, (, ” α, é, ñ, ß {, \n, ++ <0x0A>
Supplementary Gate Statistics
The heatmap in the main paper shows that large gate activations are concentrated in a subset of later-layer heads. Figure 8 provides a complementary aggregate view of the same behavior. Mean gate activation remains low in early layers and increases in later layers, while the fraction of exactly zero gate values decreases. At the same time, the coefficient of variation across heads increases, indicating that later-layer access is not only stronger but also more head-specialized. This supports the interpretation in Section 5: SATFormer does not use V1 as a uniform shortcut. Instead, the model learns a sparse access pattern that becomes stronger and more differentiated across heads in deeper layers. 14
Gate Profile by Layer Mean Gate (α)
Head Specialisation (CV)
10
80%
6
4
2
Fraction α = 0
2.0
CV across heads
Mean α
8
0
Gate Sparsity (α = 0)
2.5
1.5
1.0
3
5
7
9
11
13
15
17
0.0
60%
50%
0.5
1
70%
40% 1
3
Layer index
5
7
9
11
13
15
17
1
3
Layer index
5
7
9
11
13
15
17
Layer index
Figure 8: Gate profile by layer. Mean gate activation increases in later layers, while gate sparsity decreases and head-level variation grows. This provides an aggregate view of the sparse, depthdependent access pattern shown in the main-text heatmap. C.3
Feature Distribution in V1
We also compare the distribution of automatically labeled V1 features across the baseline Transformer and SATFormer. SATFormer produces a larger number of automatically verified monosemantic features under our pipeline, with 3,939 retained features compared to 3,261 for the Transformer. We interpret this as evidence that the accessible feature structure differs between the models, but not as a direct measure of representational richness. This metric can be affected by SAE training dynamics, activation geometry, tokenizer artifacts, and the automated labeling procedure. Figure 9 shows that the broad category distribution of labeled V1 features is similar across the two architectures. This is important for the main analysis: SATFormer’s category-specific gate behavior does not appear to arise from a large shift in the coarse content of V1 . Instead, the evidence is more consistent with the view that SATFormer primarily changes how early features are accessed through depth.
Distribution of V1 SAE Feature Categories by Architecture
89.5% 82.6%
Semantic Root Fragments Entity Marker Fragments
3.2%
Structural Punctuation
3.0% 4.1%
Logic Quantifier Keywords
2.8% 4.3%
Morphological Suffixes
0.7% 0.4% 0.6%
5.8%
2.0%
Numerical Digits Limitations. The SAE-based categories Fragments should be interpreted as a coarse analysis 0.3% Functional 0.3% Leading Space tool rather than a definitive semantic labeling of early representations. The Llama0.1% Multilingual 0.2% Unicode 2 tokenizer produces many subword fragByte Level ments and byte-level artifacts, which limits 0.1% Fallbacks the granularity of linguistic categories. In Code Formatting addition, the labeling pipeline relies on an SATFormer Symbols Transformer LLM consistency check and therefore may 0.1% 1% 10% 100% inherit model-driven biases. For this reaFraction of Total SAE Features (%) Log Scale son, we use the SAE labels only to support the qualitative conclusion that SATFormer gate activation varies across broad feature Figure 9: Transformer and SATFormer have similar category distributions, suggesting that SATFormer does categories. not substantially change the content of V1 .
15