arXiv CSopen access

Demystifying Variance in Circuit Discovery of LLMs

Demystifying Variance in Circuit Discovery of LLMs

Frank Zhengqing Wu

Francesco Tonin

Volkan Cevher

Laboratory for Information and Inference Systems (LIONS) École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

arXiv:2606.16920v1 [cs.LG] 15 Jun 2026

{zhengqing.wu,francesco.tonin,volkan.cevher}@epfl.ch

Abstract Circuit discovery is a key technique in mechanistic interpretability to pinpoint the model components that are crucial for performing a given task. Although the current state-of-the-art method (EAP-IG) performs well on the metric of (un)faithfulness, it suffers from substantial variability. This includes resampling variance, where the circuit changes when we probe with a new batch of data from the same distribution; rephrasing variance, where the discovered circuit shifts when the prompts are rephrased; and sample-wise variance, where a circuit with low population unfaithfulness exhibits large fluctuations in unfaithfulness across individual samples. This paper studies the roots of these variances. We demonstrate that CEAP, our new circuit discovery method that improves upon EAP-IG with a theoretical guarantee, can substantially lessen resampling variance. We further show that rephrasing variance arises because prompts with different templates tend to activate different circuits in the model. This leads us to argue that it may be challenging to find a comprehensive circuit that explains and controls the model’s behavior on a task, which can be expressed in countless templates, suggesting that LLMs may be inherently hard to steer. We show that sparsity, which has been claimed to form more compact and interpretable task circuits, fails to solve this problem. Regarding sample-wise variance, we argue that it is largely benign: extremely poor unfaithfulness scores often stem from how unfaithfulness is defined, rather than from defects in the measured circuits. We show that the magnitude of unfaithfulness is affected by selective contribution scaling, a neural mechanism that accounts for the extremely poor scores sometimes observed.

1

Introduction

Circuit discovery aims to find the key pathways a model uses to perform a task. It usually does this by first evaluating the importance of all components, then selecting the most important ones to form the circuit that is considered responsible for carrying out the task. In recent years, this area has advanced along two main axes: greater efficiency and broader generality. On the efficiency side, methods are becoming more automated and easier to parallelize. To eliminate the need for extensive manual work, [1] introduced an automated framework that evaluates the importance of model components by applying causal interventions to each in turn. As such a method scales poorly, [2–4] proposed gradient-based circuit discovery approaches that leverage the parallelism provided by modern deep learning hardware. In particular, edge attribution patching (EAP) [3, 4] can evaluate all the components of a model in two forward and one backward passes. However, this level of efficiency comes at the cost of precision. To balance these two aspects, [5] interpolated the model behavior into multiple steps and applied EAP to each step, yielding EAP-IG (integrated gradients). On the generality side, earlier work [4, 6] often focused on single-sample probing. This methodology was unable to provide much insight into how models handle a task as a whole [4]. Subsequently, researchers transitioned to methods that assess the significance of model components over a task Preprint.

distribution with many samples, typically through computing the average importance across those samples [2, 5, 7, 8]. By steering the discovered task circuits, [9, 10] enhanced the model’s performance on those tasks, highlighting the practical value of circuit discovery. Nevertheless, several works revealed great variances in the circuit discovery algorithms, calling their reliability into question. In this work, we discuss three forms of such variance. The first is resampling variance [11], which means that the identified circuit may differ when the probing prompts are resampled from the same underlying distribution. We show that such variance can be reduced by our new circuit discovery method, conductance-based EAP (CEAP), which achieves the same level of efficiency as EAP-IG. The second type of variance is rephrasing variance [11], where alternative phrasings of the same input can produce substantially different circuits. We show that this is because different templates activate distinct circuits in the model. A recent work [12] also observed template dependence for a different circuit-discovery algorithm. Together, our results suggest the ubiquity of this phenomenon. While [12] proposed to shift the granularity of circuit discovery from tasks to templates, we take the stance that the dependence of circuits on templates reveals a profound shortcoming of current models and notions of tasks. Since it is impractical to exhaust all templates, the generality goal of circuit discovery might be unattainable: it is infeasible to find comprehensive task circuits, and one cannot reliably predict the outcome of task circuit steering when the model is faced with unseen templates. We investigate sparse training [13] as a potential remedy, as it tends to produce more compact task circuits, which might compel the model to rely on a single circuit for all templates. Nonetheless, we do not find strong evidence that sparsity adequately mitigates this issue. The third form of variance is sample-wise variance [14], in which the circuit quality metric, unfaithfulness, varies sharply across samples. We show that the poor unfaithfulness scores arise from a negative correlation between the metric and the magnitude of the model’s behavior, rather than from deficiencies in the discovered circuit itself. We account for this using a neural mechanism that we term selective contribution scaling. Taken together, our contributions are the following: • We introduce CEAP, whose underlying component selection strategy is more principled than that of EAP-IG. We support this claim by proving that conductance satisfies additive order preservation—an intuitive and desirable property that IG fails to meet. We further demonstrate that CEAP lowers resampling variance. • We show that template-induced rephrasing variance is widespread and study its implications for circuit steering. We find that sparsity does not solve this problem, despite its prima facie appeal. • We explain extreme sample-wise unfaithfulness through selective contribution scaling, showing that large unfaithfulness values reflect the metric geometry rather than catastrophic circuit failure.

2

Background: Evaluating Subgraphs With (Un)faithfulness

Figure 1: The computational graph of a mini Transformer. The residual addition is split into separate edges. The black edges are the ones included in the subgraph/circuit, while the grey ones are excluded. During a patched run, modules will take the results from previous ones through black edges but the grey edges will be fixed at activations corresponding to the corrupted input. Each red dotted line intercepts a set of edges that is necessary and sufficient for producing the output.

Circuit discovery aims to identify a subgraph (circuit) from the model’s full computational graph that best reproduces the full model’s behavior (illustrated in Figure 1). A widely used measure of the circuit quality is faithfulness [5]. In the current setup, the behavior of a task is usually characterized by how a task-specific output metric changes when the input changes from a base point (referred to as corrupted input in previous literature) to the clean input. For instance, when examining subject-verb agreement (SVA) [15], we might quantify how much output probability shifts from plural verbs to 2

singular verbs as the input changes from one that cues a plural verb (corrupted input) to one that cues a singular verb (clean input). To quantify this change in general, for a given input x, we use the metric prob diff M (x; x0 , x1 ) := P(C(x1 )|x) − P(W (x0 )|x), where x1 is a clean input, like “The book on the table", and x0 is a corrupted input like “The books on the table". C(x1 ) is the set of tokens that x1 encourages, and W (x0 ) is the set of competing tokens encouraged by x0 instead. Continuing our example of SVA, C(x1 ) is singular verbs, and W (x0 ) is plural verbs. In the following, when the clean-corrupted pair is clear from context, we shorten M (x; x0 , x1 ) to M (x). Besides prob diff, another commonly used metric is logit diff [4, 5], which, in this case, is the summation of all the logits corresponding to singular verbs minus those of the plural ones. [1, 5] also recommended using KL divergence as a metric. Nevertheless, we observed numerical issues with this approach, detailed in Appendix C. We use subscripts G and G′ to denote evaluation on the full computational graph and selected subgraph, respectively. The behavior of the full model and the selected circuit can be written as a subtraction of metrics, QΓ (x0 , x1 ) := MΓ (x1 ) − MΓ (x0 ), (1) where Γ ∈ {G, G′ }. The metrics for the full model in Equation (1) can each be obtained via two standard forward passes. By contrast, MG′ (·) is obtained by patching: edges outside G′ are fixed to their activations under x0 , while edges inside G′ remain responsive to the current input through the selected subgraph [5], as visualized in Figure 1. By this design, MG′ (x0 ) = MG (x0 ). Finally, the faithfulness [5] of G′ with respect to G is defined as Φ(G′ , G; x0 , x1 ) =

QG′ (x0 , x1 ) 1 . QG (x0 , x1 )

The best value for Φ(G′ , G; x0 , x1 ) is 1, as it is the faithfulness yielded by choosing the entire graph. For the ease of discussion, in what follows, we use unfaithfulness, defined as U (G′ , G; x0 , x1 ) := |1 − Φ(G′ , G; x0 , x1 )|, to evaluate the subgraph. The smaller this metric, the better, which clarifies the presentation.

3

Conductance-based EAP (CEAP)

To find a subgraph that best explains the model’s behavior on a certain task, we first quantify the importance of the edges in the model with a scoring method, after which we select the most important edges, which constitute the subgraph. Naturally, the quality of the subgraph hinges upon the soundness of the scoring method. In this section, we discuss how the conductance-based scoring method, which CEAP builds on, improves upon the previous IG-based scoring method. Suppose we are studying a metric function M (·) ∈ C(Rdin , R), where din is the input dimension. The value of the metric given any input can be written as a function of the activations on the edges, M (x) = M ({ae (x)}e∈E ). Here, ae (x) ∈ Rde are the activations on edge e, and E denotes the set of all edges. We study the process where we change the input from x0 to x1 . The importance we attribute to each component i of the activation ae is its conductance [17]: Definition 3.1. The conductance of a scalar activation aei for a scalar-valued function M (·), evaluated with respect to the corrupted-input pair (x0 , x1 ), is given by: Z Z 1 X ∂M ∂M (x0 + α(x1 − x0 )) ∂aei ICond (aei ) = daei = (x1j − x0j ) dα, (2) ∂a ∂aei ∂xj ei γ 0 j∈[din ]

where γ is the path that aei traverses when the input moves from x0 P to x1 in a straight line. Then, we define the importance/conductance of the edge e to be: ICond (e) = i∈[de ] ICond (aei ). 1 This value was previously called normalized faithfulness in previous literature [5, 16]. As the “unnormalized" faithfulness

is not the emphasis of this paper, we will refer to the normalized faithfulness as faithfulness for brevity.

3

Comparison with previous methods. The current state-of-the-art patching method (EAP-IG [5]) uses integrated gradients (IG) [18], which scores each activation component with: Z 1 ∂M (x0 + α(x1 − x0 )) IIG (aei ) = [aei (x1 ) − aei (x0 )] dα. (3) ∂aei 0 P Then, the IG method scores an edge with IIG (e) = i∈[de ] IIG (aei ). When evaluating edges across multiple samples, the edge scores are computed as the average over all samples. After all edges have been scored, circuit discovery proceeds by retaining only those edges whose scores have the largest absolute values. In this paper, we use the greedy method [5]. This method first finds a circuit with a target edge number and prunes the childless or parentless edges.2 The excluded edges are patched as described in the previous section. Different scoring methods, IG or conductance, result in different patching: EAP-IG and CEAP. Overall, to obtain the most informative graph, we need to give higher absolute-value scores to edges whose patching causes greater changes at the network output, which motivates us to study the following property. Definition 3.2 (Additive order preservation). P Consider a function P that can be decomposed additively into several branches: R ∋ F (x; B) = b∈B fb (yb ) = b∈B fb (gb (x)), P where B is the set of branches, and x and yb may be vectors. We also denote R ∋ F (x; B ′ ) = b∈B ′ fb (yb ) for any B ′ ⊆ B. Suppose we study the process where x moves from x0 to x1 . For convenience, we denote ∆F (B ′ ) = F (x1 ; B ′ ) − F (x0 ; B ′ ) and ∆fb = fb (x1 ) − fb (x0 ). For a scoring function that assigns a scalar to an edge, I(yb ) ∈ R, we say it satisfies additive order preservation if |∆F (B) − ∆F (B \ {b1 })| = |∆fb1 | > |∆fb2 | = |∆F (B) − ∆F (B \ {b2 })| ⇐⇒ |I(yb1 )| > |I(yb2 )|.

Remark 3.3. Note that |∆F (B) − ∆F (B \ {b})| exactly equals to the importance of "activation" yb measured with activation patching [4], which freezes yb to be yb (x0 ) while letting other activations freely move with upstream inputs. Additive order preservation conveys that, if the patching of one branch causes the whole function to change its behavior more than the patching of another, then the former branch should be assigned a score higher in its absolute value than that of the latter, so that the former can be prioritized to be admitted into the subgraph. This property is particularly desirable when we study unfaithfulness, which evaluates a subgraph by how much the behavior is affected when the complement of the subgraph is patched. Remark 3.4. The sum-of-branches type of functions discussed in Definition 3.2 can be of significant relevance if we wish to interpret the residual stream of currently prevalent architectures, as the residual stream is always a summation of previous branches. More broadly, additive order preservation is a minimal desideratum for scoring when applied to general functions. General functions may have other ways to mix the outputs of branches. The appropriate desiderata for more complicated mixings may not be immediately intuitive. However, the first-order Taylor expansion of any mixing always appears in the form of sum-of-branches functions, so studying the latter still provides insight. To show that conductance is a more principled scoring method, we prove: Theorem 3.5. Conductance satisfies additive order preservation, while IG does not. The proof is in Appendix E. In the following, we derive the discretized version of conductance and IG, which shows how the two quantities are computed numerically. The discretized computation also sheds light on why IG appears less principled. Let us break down the input movement x0 → x1 into K evenly spaced segments and denote the endpoints of each segment by xk := x0 + k/K(x1 − x0 ), where k ∈ {0, 1, · · · , K}. We also denote the activations corresponding to these input endpoints by akei := aei (xk ). The discretized IG can be written as 0 K−1 X ∂M (xk ) aK ei − aei dis IIG (aei ) = . (4) K ∂aei k=0

One can interpret this formula as a refined version of gradient×input [4]. The latter scores the
M (x0 ) 0 edge with aK ei − aei aei . Equation (4), in comparison, computes the average gradient instead 2 The target edge number and the final edge number are usually close. We do not distinguish between them in our discussion.

4

Pairwise Jaccard index

EAP-IG prob diff

plural_0

plural_1

CEAP prob diff

plural_2

plural_3

plural_4

plural_5

0.8 0.6 0.4 0.2

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

Number of edges

Figure 2: Comparison of PJI yielded by CEAP and EAP-IG (using GPT-2 XL for a subset of SVA)

of using one gradient at one endpoint. Another more fine-grained interpretation of Equation (4) is 0 aK (xk ) ei −aei that it is a weighted sum of the gradients ∂M . Nevertheless, it ∂aei , with all weights equal to K may be undesirable for all the weights to be the same. After all, the rate of change of M with respect
(xk ) to eei is roughly ∂M only when aei lies within the segment of akei , ak+1 , whose length is not ei ∂aei

0 aK ei −aei due to the nonlinearity of aei (x). This motivates a new formula that rectifies the K ∂M (xk ) weightings of ∂aei : K−1 X ∂M (xk )
dis ak+1 − akei , (5) ICond (aei ) = ei ∂aei k=0

necessarily

which is exactly the discrete form of conductance [19]. The “weighted-sum of gradients" perspective of Equations (4) and (5) also suggests that IG can be taken as an approximation to conductance, which explains its efficacy shown by [5]. 3.1

CEAP Reduces Circuit Variance Under Data Resampling

[11] noted that EAP-IG finds significantly different circuits when we resample the dataset from the same distribution. We will show that such an instability is ameliorated by CEAP. Following the setup in [5], we conducted experiments on GPT-2 small, GPT-2 XL, Pythia-160M, and Pythia-2.8B over three datasets: SVA [15], IOI [7], and greater-than [8].3 The templates for generating these datasets are shown in Appendix D. Experimental details are in Appendix A. In our experiments, for each task, we used a full dataset of roughly 10000 samples and subsampled 4 smaller datasets (without replacement), each with 1000 samples. We performed circuit discovery for the 4 datasets and measured the stability of the found circuits using pairwise Jaccard index (PJI). More concretely, suppose we have two graphs G1 and G2 , with edge sets EG1 and EG2 , respectively. The PJI for these two graphs is defined as |EG1 ∩ EG2 | / |EG1 ∪ EG2 |. With 4 sub-datasets, for the same target edge number, we obtained 4 graphs and thus 6 PJIs. We report the mean and standard deviation of these PJIs in Figure 2 for GPT-2 XL on SVA. The x-axis shows the number of edges we admitted during circuit discovery. We separate experiments for different templates, as we will show in Section 4 that different templates activate distinct circuits in the model and thus their respective circuit discovery should be done separately. Due to space constraints, we only show the first 6 templates here. The results for other models, datasets, and templates are shown in Appendix F. Overall, we find that CEAP achieves PJI values that are higher than, or at least on par with, those of EAP-IG. We believe this is because CEAP can find important edges in a more principled manner as shown in Theorem 3.5, while EAP-IG can only be viewed as a noisy version of CEAP. Note that CEAP attains unfaithfulness comparable to EAP-IG, so its increased stability in circuit selection does not arise from choosing task-irrelevant circuits. Further details are given in Appendix F.

4

Template-induced Circuit Variance

[11] observed that performing circuit discovery with paraphrased clean-corrupted prompt pairs can yield dramatically different circuits, even when a human judges the paraphrased prompts as representing the same underlying task. We demonstrate that this occurs because the templates influence the circuits the models use to perform the same task. To show this, we take 1000 samples 3 [5] also used three other datasets. They have relatively few samples and are unsuitable for our purpose.

5

single_0 plural_0 single_1 plural_1 single_2 plural_2 single_3 plural_3 single_4 plural_4 single_5 plural_5 single_6 plural_6 single_7 plural_7 single_8 plural_8 single_9 plural_9 plural_10 single_11 plural_11 single_12 plural_12 single_13 plural_13 single_14 plural_14

6

4

2

0.9

0.8

plural_1 plural_5

0.7

plural_8

Pairwise Jaccard index

8

number of edges in the circuit for each sample 826 number of edges of the union of all circuits = 8735 total number of edges in the entire graph = 32491 avg. unfaithfulness = 0.195

plural_0

plural_9 plural_10 plural_11

Template

10

Absolute-score rank UMAP 2

single_2 plural_2 single_3 plural_3 single_4 plural_4 single_6 plural_6 single_7 plural_7

template_label

12

0.6

plural_12 plural_13 plural_14

0.5

single_0 single_1

0.4

single_5 single_8

0.3

single_9 single_11 single_12

0.2

single_13

(a) Absolute-score rank UMAP.

single_14

single_13

single_9

single_12

single_8

single_11

single_5

single_1

Template

single_0

plural_13 plural_14

plural_12

plural_8

20

plural_9 plural_10 plural_11

10

plural_5

0

plural_1

10

Absolute-score rank UMAP 1

single_2 plural_2 single_3 plural_3 single_4 plural_4 single_6 plural_6 single_7 plural_7

20

plural_0

single_14

0

(b) Pairwise Jaccard index.

Figure 3: Template-dependent sample circuits for GPT-2 small on SVA. The UMAP embeds samples by their absolute-score rank vectors, and the Pairwise Jaccard index matrix captures the overlap between the corresponding circuits. In the right panel, two clearly separated template groups emerge. Circuits within each group exhibit substantial overlap, while overlap between the two groups is almost zero. These two groups correspond to the two clusters in the left panel: one in the upper-left corner and the other containing the rest. The statistics in the upper-right of the right panel offer quantitative insight into how little overlap there is among the circuits for different templates.

from the same task but in different templates, apply CEAP to evaluate edge importance, and identify the critical circuit for each sample. Then, we illustrate the similarity of the scores across different samples, as well as the similarity of the graphs identified for these samples. For the similarity of the scores, we investigate the ranking of edges based on the absolute values of their scores.
More precisely, consider all edges in the model arranged in a single
vector e1 , e2 , · · · , e|E| . Their associated scores for a given sample are given by s1 , s2 , · · · , s|E|
, where 4 si = ICond (ei ). We then take the absolute values of these scores, |s1 |, |s2 |, · ·
· , |s|E| | , and sort them in descending order to obtain the absolute-score rank vector r1 , r2 , · · · , r|E| , where ri denotes the rank of |si |. We then applied UMAP to the absolute-score rank vectors for all 1000 samples to project them onto 2 dimensions. The resulting visualization for GPT-2 small on SVA is shown in Figure 3a. Additionally, we directly visualize the similarity between the graphs obtained for different samples. We select the graph size for which the mean unfaithfulness across all 1000 samples falls below 0.2, and then compute the PJI among all graphs found for each sample at that size; these results are shown in Figure 3b. We produced the same visualization for GPT-2 small and Pythia-160M on SVA, IOI, and greater-than. Across all figures, it is evident that the model relies on different circuits to process different templates. The complete set of visualizations is provided in Appendix G. Numerous prior studies [4, 5, 11, 13, 14, 20] have operated under the assumption that a single circuit within a given model is responsible for performing a specific task, and that, by evaluating edge importance via averaging over a sufficient number of samples, usually generated with multiple templates [5, 11], we can comprehensively pinpoint the circuit responsible for performing the task. However, we have shown that such an assumption is flawed. Different templates, in fact, trigger different circuits within the models. Since the space of possible templates is effectively unbounded, it follows that identifying a comprehensive circuit for a particular task may be an unattainable goal. This implication challenges one of the central aspirations of mechanistic interpretability: that, by adjusting the identified task circuits, we can reliably steer the model’s behavior on that task in our preferred direction [21], including boosting math capability [9], moderating refusal [10], editing knowledge [22–24], and many more [7, 13, 25]. Although previous works showed that interventions on circuits identified from available samples can be effective [7, 13] and may improve benchmark performance [9, 10], our observations here suggest that such a methodology does not guarantee reliable behavior in the wild, where the model may encounter unseen templates.5 In practice, the 4 This is to follow the convention of [5] that performed circuit discovery using the absolute score values. 5 In Appendix H, we present a case that, after encouraging plural verbs for a sample by patching a circuit obtained for this

purpose from another sample, the model actually encourages singular verbs.

6

degree of concentration among template circuits can serve as a proxy for our confidence in the effect of a model intervention. To deepen the discussion, we ask whether it is achievable or meaningful to pursue concentrated task circuits independent of templates, since "tasks" may seem an artificial concept, while templates are the concrete data the models process. It may be quite demanding to require models to develop taskfocused circuits rather than template-focused ones. Nevertheless, a task can be thought of informally as a collection of prompts that a human is able to address using a single, unified algorithm,6 which indicates that it should be feasible to implement it using just one circuit. The fact that current models resort to different circuits for different templates implies that the models leverage redundant algorithms.7 A well-known principle in AI holds that intelligence can be understood as a form of compression [26, 27]. In particular, from the perspective of Kolmogorov complexity [28], intelligence can be viewed as the ability to find short descriptions of data that enable accurate prediction. In this sense, models that rely on redundant algorithms to perform tasks have suboptimal intelligence, and the aim of training models that rely on a template-independent circuit for a given task aligns with the broader objective of discovering more compressive, thus intelligent, models. A recent effort [13] to train more interpretable transformers showed that imposing sparsity during training shrinks the task circuits. Prima facie, this seems like a potential solution to mitigate the unsteerability of models that we discussed above. Could it be that sparsity forces the model to merge the circuits for different templates, thereby causing the task circuits to shrink? Our experiments, based on the models from [13], suggest that the benefit from sparsity is limited. In our experiments, we adapted transformer-lens [29], the interpretability infrastructure that we used for CEAP, to the specialized architectures of the sparse models built by [13], which were customized for coding. We also created two datasets suitable for these models, single-double-quote and else-elif, summarized in Appendix D. Across these experiments, we do not find that sparsity causes the model to merge template circuits: sparse models still deploy different circuits for different templates. Full visual comparisons and summary statistics are presented in Appendices I and J.

Cross-template avg. PJI

In addition, we analyze how the mean PJI between circuit pairs obtained 0.45 from different templates varies with increasing sparsity, while keeping all other settings fixed. As shown in Figure 4, the overlap of different +23.7% 0.40 template circuits increases substantially when we allow fewer nonzero 0.35 parameters. Nevertheless, the average PJI for circuit pairs across dif+11.9% ferent tasks (cross-task) also increases to an extent comparable to the 0.30 mean cross-template PJI for single-double-quote. This suggests that 0.25 else-elif the level of polysemanticity increases with sparsity, which leads to the single-double-quote 0.20 cross-task possibility that the increasing overlap between cross-template circuits +11.9% for one task might be a result of the model using a larger portion of 0.15 0.9 1.9 3.7 7.4 14.8 polysemantic neurons to implement redundant algorithms, rather than Nonzero parameters (M) truly compressing for a single, unified algorithm for the task. This complicates task steering. Determining whether sparsity primarily en- Figure 4: Cross-template average hances polysemanticity or compression requires detailed model-level PJI increases with more sparsity, investigation, which lies beyond the scope of this variance-focused but so does cross-task average PJI. paper. We leave such an analysis for future work. Our current observations do not indicate that sparsity resolves the issue discussed in this section.

5

Variance of Unfaithfulness Across Samples

[14] noted that averaging edge scores over samples to identify circuits can yield large variance in per-sample unfaithfulness, even if the resulting circuit is faithful at the population level.8 One might think this is because the averaged scores better match some samples while fitting others less well. However, it turns out that this is not the primary source of variance. In our experiments, we observed that even if we perform circuit discovery with CEAP using a fixed circuit size for each sample separately, thereby eliminating influence from other samples, the resulting unfaithfulness still 6 For example, for IOI the algorithm is: find all names, remove duplicates, and output the rest [7]. 7 A redundant algorithm may look like: if handling template_A: use algorithm_A, elif handling template_B: use algorithm_B... A good example of this for IOI is shown in Figure 3 of [12]. 8 The notion of population-level faithfulness has varied in previous works [5, 7, 14], which we discuss in Appendix K.

7

ρ = -0.52, p = 0.003

20.6

ρ = 0.79, p = 2.0e-07

0.10

101 edges ≈ 826

101 0.08

100

Ū

0.536

U

Q3 0.33 median 0.199

10−1

Ū 0

0.957

0.06

100 0.04

Q1 0.087

0.02 10−1 10−2

0.00 0.0

0.1

0.2

0.3

0.4

|QG|

0.1

0.2

0.3

0.4

|QG|

(c) Ū ′ vs. |QG |.

(b) Ū vs. |QG |.

(a) U distribution.

0.0

Figure 6: For GPT-2 small on SVA template plural_1 with target edge number 826. Here, we perform circuit discovery for each sample separately. The corresponding U distribution for scores obtained via averaging within the template is in Figure 52. Samples with smaller |QG | tend to show larger pessimistic unfaithfulness Ū , while the pessimistic unnormalized unfaithfulness Ū ′ tends to increase with |QG |. Orange curves show rolling-median trends as visual guides only. Boxes report Spearman rank correlation ρ and its p-value.

fluctuates greatly across samples. Figure 6a shows such an example, which we use throughout this section. We see that the unfaithfulness U of a sample (sample 78) may go as high as 20, while most of the samples from the same template (plural_1) already achieve low unfaithfulness. In this section, we show that under a reasonable circuit discovery scheme, poor unfaithfulness scores typically stem from how unfaithfulness interacts with the distribution of edge scores, rather than from missing important edges. Comparing unfaithfulness values between samples does not reveal which sample’s circuit is more sufficient for recovering the full model’s behavior. 80

U

60 40 20 31 edges U=0.267

0 80 70

Ū

60 50 40 30 0

250

500

750

1000 1250 1500 1750 2000

Number of edges

Figure 5:

Normal and pessimistic unfaithfulness for sample 78 of SVA.

It is important to recognize that unfaithfulness measured on an individual sample is often hard to interpret, because it can vary in a highly nonmonotonic way as the circuit size increases. Figure 5 shows how U changes with the circuit size. It reaches 0.267 with just 31 edges but shoots up to around 80 when we increase the circuit size. Ideally, however, unfaithfulness should consistently decrease when we admit more edges into the circuit. Single-sample unfaithfulness fails to meet this because it evaluates the behavior of the model at a single point in the input space. We detail this in Appendix L. Highly fluctuating unfaithfulness can introduce noise to our analysis and lower statistical significance. To reduce this noise, we define pessimistic unfaithfulness Ū : Definition 5.1. Let U (m) denote the unfaithfulness of the circuit found by the algorithm with target edge number m. Pessimistic unfaithfulness is defined as Ū (m) = maxm≤m̂≤|E| U (m̂).

In experiments, since we only evaluate a subset of all possible target edge counts, we estimate Ū (m) by taking the maximum unfaithfulness observed over the evaluated counts. Next, we show that extremely poor U should not be interpreted as a dramatic discrepancy between the circuit and the model’s behavior. We define the unnormalized unfaithfulness U ′ := |QG′ (x0 , x1 ) − QG (x0 , x1 )| = |MG′ (x1 ) − MG (x1 )| to directly measure the behavior gap between the circuit and the full model. We also define the pessimistic unnormalized unfaithfulness Ū ′ by enforcing monotonicity. In our experiments, we found that extremely high Ū is usually associated with nearzero |QG |,9 as shown in Figure 6b, where we obtained a negative ρ for Spearman rank correlation (SRC). Since QG is the normalization for computing (un)faithfulness, it is natural to suspect whether the extremely small normalization values are what cause Ū to blow up for certain samples. This suspicion is corroborated by a positive SRC coefficient ρ between the unnormalized Ū ′ and |QG |, illustrated in Figure 6c. These observations suggest that the extremely large Ū does not mean the logits produced by the chosen circuit differ wildly from those of the full model; rather, it is merely a consequence of normalization. 9 In very rare cases, we find negative Q

G , meaning the full model is more likely to output the wrong answer.

8

ρ=-0.884 p=9.16e-11

3000

ρ=-0.975 p=7.48e-20

0.7

ρ=-0.536 p=2.25e-03

ρ=0.511 p=3.90e-03

101

2500

101

Ū

Ū

R

μ

0.6

2000

100

100

0.5

1500

10−1

10−1

0.4 0.0

0.1

0.2

0.3

0.4

1500

|QG|

(a) µ vs. |QG |.

2000

2500

3000

0.4

0.5

μ

0.6

0.7

1500

(b) R vs. µ.

2000

2500

3000

μ

R

(c) Ū vs. R.

(d) Ū vs. µ.

Figure 7: Spearman rank correlations that explain Figure 6b. The setup is exactly the same as in Figure 6. The last panel, though not discussed, adds supporting evidence: it suggests that the heavy-tailedness of sn is indicative of the unfaithfulness Ū , combining the last two correlations in Equation (6).

5.1

The Correlation Between Ū and |QG |

What remains unclear is why, given the same number of allowed edges, the circuits identified for samples with small |QG | achieve much worse Ū than the circuits identified for samples with large |QG |. Intuitively, this may be because as |QG | decreases, the relative significance of the edges left out of the circuit, compared with those that are included, becomes greater. The following analysis supports this explanation. Given a model with each edge scored on sample n, we arrange the absolute-valued scores of all P|E| its edges into a nonincreasing vector sn . Let S n := ℓ=1 snℓ , and un = sn /S n so that all un P|E| components sum up to 1. We define the normalized momentum to be µn := ℓ=1 unℓ × ℓ, which n quantifies how heavy-tailed s is. From here on, for clarity, we introduce a superscript of sample index n for QG . We found that the samples with lower |QnG | have larger µn (Figure 7a). Suppose we consider two samples such that |QnG1 | > |QnG2 |. Then un1 and un2 would appear as illustrated in the lower panel of Figure 8. Given an edge number m, our circuit discovery algorithm approximately P selects the edges with the highest absolute scores to form the circuit. Define C n (m) := ℓ∈L snℓ where L is the set of indices of sn that correspond to the edges admitted into the circuit. Then, C n (m)/S n := Rn (m) can be interpreted as the ratio of score mass included in the circuit. Naturally, µn and Rn (m) should have a negative correlation (Figure 7b). Moreover, as Rn (m) carries the meaning of the aggregate importance of edges included in the circuit relative to total importance of all edges, it should have a negative correlation with unfaithfulness Ū , which reflects the change of logit behavior caused by including only the edges in the chosen circuit relative to the original logit behavior caused by the full model (Figure 7c). Based on these correlations, we derive the following relationships that account for the trend observed in Figure 6b. |QnG | ↘ −→

µn ↗ −→

Rn (m) ↘ −→

Ū ↗

(6)

While the last two arrows in the above appear intuitive, the mechanism underlying the first one is unclear. We show that this is due to a neural mechanism that we term selective contribution scaling, which we cover in Section 5.2. We verify all the correlation trends in this section, including the two in Figure 6 and the four in Figure 7, for all templates of SVA/IOI/greater-than, for both GPT-2 small and Pythia-160M, and for all graph sizes that we swept. The trends hold up well. Moreover, we check these trends using U in place of Ū , in case our design of Ū introduces undesirable artifacts. U -based plots generally show the same trends, despite being noisier. Note that the discussion of this section so far is based on scoring samples separately. We also produce all the visualizations for scores obtained by averaging samples within their own templates, which is closer to common practice [5, 11, 14]. Those experiments are well captured by the conclusion drawn in this section, as the circuits found within a template are highly similar in most cases. All the results are presented in Appendix M. 5.2

A Mental Image for Selective Contribution Scaling

Our conductance score satisfies that, if a set of edges is necessary and sufficient to produce the output (Figure 1), their scores must sum up to QnG (known as partition consistency [17]). In this sense, 9

one can interpret the score as the contribution of the edge to the output. Suppose we have a pair of samples with |QnG1 | > |QnG2 |. It is reasonable to expect that the entries in sn1 are larger than those in sn2 , given partition consistency. Since we know in hindsight that un2 is more heavy-tailed than un1 (Figure 7a), we can infer that sn1 is considerably higher than sn2 at low indices, while the differences between them at high indices are mild. Namely, if we move the input from sample n2 to n1 to increase the output signal amplitude from |QnG2 | to |QnG1 |, the important edges located at low indices will contribute more than the less important ones at high indices.10 This is what we call selective contribution scaling, as illustrated in the upper panel of Figure 8. We verify the mental image of Figure 8 with a concrete sample pair in Appendix N. That there exist specialized parts of the model reacting to structurally specific stimuli (tasks expressed in templates) and other parts being indifferent to them aligns with the notion of sparse coding, a widely endorsed framework of information processing in neuroscience [30–34]. Nonetheless, in our case, it links unfaithfulness to the insignificant magnitude of QG , which weakens unfaithfulness as a definitive marker of performance recovery. By the same token, we also anticipate great cross-sample variance in faithfulness-style normalization-dependent metrics measured on systems that exhibit sparse coding. Figure 8: Mental image

6

of selective contribution scaling.

Conclusion and Discussion

In this paper, we systematically investigate the three types of variance in circuit discovery algorithms. While resampling variance can be mitigated with more principled scoring, and sample-wise variance of unfaithfulness is largely benign, rephrasing variance is hard to diminish and suggests the impossibility of finding comprehensive task circuits. In discussing the above, we develop methods and guidance that can inform future practice. First, conductance is a more principled way of scoring compared to IG, as it satisfies additive order preservation. Second, although evaluating (un)faithfulness is relatively straightforward, and the notion itself is intuitively plausible [5], there is a caveat: comparisons of this metric across different samples are subject to an intrinsic sample-wise variance induced by selective contribution scaling. We further discuss in Appendix K that population-level (un)faithfulness should explicitly account for this effect and be more lenient toward samples whose behavioral magnitude is small. Nonetheless, (un)faithfulness remains informative for tracking how circuit quality improves as we increase the permitted circuit size. In this regard, pessimism serves as a useful complement to reduce interpretational noise. Furthermore, our discussion on template-based rephrasing variance sheds new light on the generalizability of task circuits [35]. On the one hand, it suggests that statements involving task circuits should be carefully tested for robustness under changes in templates,11 and that template-agnostic benchmarking can be inconclusive. On the other hand, it opens up avenues for future exploration. Let us define a task as the collection of samples that engage the same circuit. This definition is useful because it is operational and may provide insight into circuit steering. At present, despite certain post hoc heuristics [12], we lack a principled way to determine which templates engage the same circuit and which rely on distinct circuits. As a result, the notion of a task remains elusive. To address this, we must systematically characterize how templates influence the algorithmic procedures that models use to process them. Put differently, it is important to formalize a model-centered notion of a task, in contrast to the current notion, in which humans judge whether different instances should be considered part of the same task [5, 7, 8, 15]. Furthermore, as the circuits currently appear to be template-dependent, an important question is whether models can be trained to use the same circuit for tasks defined in human terms, which may, in turn, lead to greater compression and potentially higher intelligence.

10 Note that we are implicitly assuming that the edge orderings in sn1 and sn2 coincide. While this is not strictly true, they should be very similar, because, as we show in Section 4, samples sharing the same template tend to exhibit similar scoring patterns. 11 Some previous studies have considered robustness to paraphrasing [22, 23]. However, it has not yet become a standard practice.

10

Acknowledgements We thank Dr. Paul Rolland for valuable discussions. This work was funded by the Swiss National Science Foundation (SNSF) under grant number 2000-1-240094. This work was also supported under project ID # 37 as part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure.

References [1] Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16318–16352, 2023. [2] Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024. [3] Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 407–416, 2024. [4] Neel Nanda. Attribution patching: Activation patching at industrial scale. urlhttps://www.neelnanda.io/mechanistic-interpretability/attribution-patching, February 2022. Last accessed July 25, 2025. [5] Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://openreview.net/forum?id=grXgesr5dT. [6] nostalgebraist. Interpreting gpt: The logit lens. https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens, 2020. LessWrong blog post, accessed 2026-05-07. [7] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https: //openreview.net/forum?id=NpsVSN6o4ul. [8] Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greaterthan?: Interpreting mathematical abilities in a pre-trained language model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview. net/forum?id=p4PckNQR8k. [9] Xu Wang, Yan Hu, Wenyu Du, Reynold Cheng, Benyou Wang, and Difan Zou. Towards understanding fine-tuning mechanisms of LLMs via circuit analysis. In ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025. URL https://openreview. net/forum?id=Z9qzta1yiK. [10] Stephen Cheng, Sarah Wiegreffe, and Dinesh Manocha. What drives representation steering? a mechanistic case study on steering refusal. arXiv preprint arXiv:2604.08524, 2026. [11] Maxime Méloux, François Portet, and Maxime Peyrard. Mechanistic interpretability as statistical estimation: A variance analysis of eap-ig. arXiv preprint arXiv:2510.00845, 2025. [12] Gabriel Franco, Lucas M. Tassis, Azalea Rohr, and Mark Crovella. Finding highly interpretable prompt-specific circuits in language models, 2026. URL https://arxiv.org/abs/2602. 13483. [13] Leo Gao, Achyuta Rajaram, Jacob Coxon, Soham V Govande, Bowen Baker, and Dan Mossing. Weight-sparse transformers have interpretable circuits. arXiv preprint arXiv:2511.13653, 2025. 11

[14] Joseph Miller, Bilal Chughtai, and William Saunders. Transformer circuit evaluation metrics are not robust. In First Conference on Language Modeling, 2024. URL https://openreview. net/forum?id=zSf8PJyQb2. [15] Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. Refining targeted syntactic evaluation of language models. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3710– 3723, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.290. URL https://aclanthology.org/2021.naacl-main.290/. [16] Lin Zhang, Wenshuo Dong, Zhuoran Zhang, Shu Yang, Lijie Hu, Ninghao Liu, Pan Zhou, and Di Wang. EAP-GP: Mitigating saturation effect in gradient-based automated circuit identification. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=lGyXq0LOeQ. [17] Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is a neuron. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=SylKoo0cKm. [18] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/ sundararajan17a.html. [19] Avanti Shrikumar, Jocelin Su, and Anshul Kundaje. Computationally efficient measures of internal neuron importance, 2018. URL https://arxiv.org/abs/1807.09946. [20] Yifan Wang, Yifei Liu, Yingdong Shi, Changming Li, Anqi Pang, Sibei Yang, Jingyi Yu, and Kan Ren. Discovering influential neuron path in vision transformers. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview. net/forum?id=WQQyJbr5Lh. [21] Hengyuan Zhang, Zhihao Zhang, Mingyang Wang, Zunhai Su, Yiwei Wang, Qianli Wang, Shuzhou Yuan, Ercong Nie, Xufeng Duan, Feijiang Han, Qibo Xue, Zeping Yu, Chenming Shang, Xiao Liang, Jing Xiong, Hui Shen, Chaofan Tao, Zhengwu Liu, Senjie Jin, Zhiheng Xi, Dongdong Zhang, Sophia Ananiadou, Tao Gui, Ruobing Xie, Hayden Kwok-Hay So, Hinrich Schütze, Xuanjing Huang, Qi Zhang, and Ngai Wong. Locate, steer, and improve: A practical survey of actionable mechanistic interpretability in large language models, 2026. URL https://arxiv.org/abs/2601.14004. [22] Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https:// openreview.net/forum?id=-h6WAS6eE4. [23] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Massediting memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=MkbcAHIYgyS. [24] Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. Knowledge circuits in pretrained transformers. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 118571–118602. Curran Associates, Inc., 2024. doi: 10.52202/079017-3765. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/file/d6df31b1be98e04be48af8bedb95b499-Paper-Conference.pdf. [25] Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas 12

Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of a large language model. Transformer Circuits Thread, 2025. URL https: //transformer-circuits.pub/2025/attribution-graphs/biology.html. [26] J Gerard Wolff. Computing as compression: the sp theory of intelligence, 2013. URL https: //arxiv.org/abs/1303.2013. [27] Jürgen Schmidhuber. Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. In Workshop on anticipatory behavior in adaptive learning systems, pages 48–76. Springer, 2008. [28] Marcus Hutter. A theory of universal artificial intelligence based on algorithmic complexity, 2000. URL https://arxiv.org/abs/cs/0004001. Transformerlens. [29] Neel Nanda and Joseph Bloom. TransformerLensOrg/TransformerLens, 2022.

https://github.com/

[30] Horace Barlow. Possible principles underlying the transformations of sensory messages. Sensory Communication, 1, 01 1961. doi: 10.7551/mitpress/9780262518420.003.0013. [31] Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607–609, 1996. URL https: //api.semanticscholar.org/CorpusID:4358477. [32] Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research, 37(23):3311–3325, 1997. ISSN 0042-6989. doi: https://doi.org/10.1016/S0042-6989(97)00169-7. URL https://www.sciencedirect.com/ science/article/pii/S0042698997001697. [33] William Vinje and Jack Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. Science (New York, N.Y.), 287:1273–6, 03 2000. doi: 10.1126/science.287.5456. 1273. [34] Ben Lonnqvist, Zhengqing Wu, and Michael H Herzog. Latent noise segmentation: How neural noise leads to the emergence of segmentation and grouping. arXiv preprint arXiv:2309.16515, 2023. [35] Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeffrey Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Isaac Bloom, Stella Biderman, Adrià Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Mary Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, William Saunders, Eric J Michaud, Stephen Casper, Max Tegmark, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Thomas McGrath. Open problems in mechanistic interpretability. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/ forum?id=91H76m9Z94. Survey Certification. [36] Niladri S. Chatterji, Philip M. Long, and Peter L. Bartlett. When does gradient descent with logistic loss find interpolating two-layer networks?, 2021. URL https://arxiv.org/abs/ 2012.02409. [37] Christian H. Weiß, Fukang Zhu, and Aisouda Hoshiyar. Softplus ingarch models. Statistica Sinica, 32(2):pp. 1099–1120, 2022. ISSN 10170405, 19968507. URL https://www.jstor. org/stable/27118810. [38] David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. The MIT Press, 07 2010. ISBN 9780262514620. doi: 10.7551/mitpress/9780262514620.001.0001. URL https://doi.org/10.7551/mitpress/ 9780262514620.001.0001.

13

A

Experimental Details

Following previous conventions [4, 5], we used logit_diff for tasks whose correct and incorrect answers are single logits (IOI, single-double-quote). For other tasks, we used prob_diff. For GPT-2 and Pythia experiments, we used integration step K = 200 for both EAP-IG and CEAP. For sparse-transformer experiments [13], we used 150.

B

Code and Data Availability

A ready-to-use implementation of CEAP, along with code to reproduce the figures presented in this paper, is available at https://github.com/LIONS-EPFL/Circuit-Variance.

C

Numerical Issue Caused by KL Divergence

[1, 5] also recommended using the KL divergence as a metric, as it is in principle applicable to all tasks. In practice, however, we found KL divergence to be restrictive: it cannot be used for models that assign zero probability to some tokens, since this leads to an infinite KL value, which we observe e.g., for Pythia-160M. While Softmax(·) should theoretically ensure that every token has nonzero probability, numerical underflow (which occurs for both fp32 and fp64) frequently produces exact zeros in the probabilities. To keep the exposition uncluttered, we therefore report only the logit/prob diff metrics in the main paper. The experiments that we carried out for KL divergence do not show qualitatively different results from the conclusions we present here.

D

Dataset Templates

We use template-generated datasets so that clean and corrupted prompts differ in a controlled way. Tables 1 to 5 summarize the templates used in our experiments. Placeholders in square brackets are filled with names, nouns, variable names, or code literals during dataset construction. For the tasks customized for sparse-transformers [13], each base template generates a paired contrast. For single_double_quote, we show both paired directions explicitly using the _s and _d suffixes. For else_elif, we show one representative direction in the table and omit the _else/_elif suffixes from the displayed template names for space; the generated data and visualizations keep those suffixes.

14

Table 1: SVA dataset templates. Each row shows a prompt contrast where the correct answer is either a singular or plural verb form. In the placeholders, sg and pl denote singular and plural forms, respectively. Template single_0 plural_0 single_1 plural_1 single_2 plural_2 single_3 plural_3 single_4 plural_4 single_5 plural_5 single_6 plural_6 single_7 plural_7 single_8 plural_8 single_9 plural_9 single_10 plural_10 single_11 plural_11 single_12 plural_12 single_13 plural_13 single_14 plural_14

Clean prompt template example The [Nounsg ] that [Verbsg ] the [Noun] The [Nounpl ] that [Verbpl ] the [Noun] The [Nounsg ] from the [Nounpl ] The [Nounpl ] from the [Nounpl ] The [Nounpl ] that the [Nounsg ] The [Nounpl ] that the [Nounpl ] The [Nounpl ] the [Nounsg ] The [Nounpl ] the [Nounpl ] The [Nounpl ] said the [Nounsg ] The [Nounpl ] said the [Nounpl ] The [Nounsg ] the [Nounpl ] [Verb] The [Nounpl ] the [Nounpl ] [Verb] The [Nounpl ] that the [Nounsg ] The [Nounpl ] that the [Nounpl ] The [Nounpl ] the [Nounsg ] The [Nounpl ] the [Nounpl ] The [Nounsg ] that the [Nounpl ] [Verb] The [Nounpl ] that the [Nounpl ] [Verb] The [Nounsg ] that the [Nounpl ] [Verb] The [Nounpl ] that the [Nounpl ] [Verb] The [Nounsg ] The [Nounpl ] The [Nounsg ] [Verbsg ] [VP] and The [Nounpl ] [Verbpl ] [VP] and The [Nounsg ] next to the [Nounpl ] The [Nounpl ] next to the [Nounpl ] The [Nounsg ] [Verbsg ] and The [Nounpl ] [Verbpl ] and The [Nounsg ] the [Nounpl ] [Verb] The [Nounpl ] the [Nounpl ] [Verb]

Corrupted prompt template example The [Nounpl ] that [Verbpl ] the [Noun] The [Nounsg ] that [Verbsg ] the [Noun] The [Nounpl ] from the [Nounpl ] The [Nounsg ] from the [Nounpl ] The [Nounpl ] that the [Nounpl ] The [Nounpl ] that the [Nounsg ] The [Nounpl ] the [Nounpl ] The [Nounpl ] the [Nounsg ] The [Nounpl ] said the [Nounpl ] The [Nounpl ] said the [Nounsg ] The [Nounpl ] the [Nounpl ] [Verb] The [Nounsg ] the [Nounpl ] [Verb] The [Nounpl ] that the [Nounpl ] The [Nounpl ] that the [Nounsg ] The [Nounpl ] the [Nounpl ] The [Nounpl ] the [Nounsg ] The [Nounpl ] that the [Nounpl ] [Verb] The [Nounsg ] that the [Nounpl ] [Verb] The [Nounpl ] that the [Nounpl ] [Verb] The [Nounsg ] that the [Nounpl ] [Verb] The [Nounpl ] The [Nounsg ] The [Nounpl ] [Verbpl ] [VP] and The [Nounsg ] [Verbsg ] [VP] and The [Nounpl ] next to the [Nounpl ] The [Nounsg ] next to the [Nounpl ] The [Nounpl ] [Verbpl ] and The [Nounsg ] [Verbsg ] and The [Nounpl ] the [Nounpl ] [Verb] The [Nounsg ] the [Nounpl ] [Verb]

Correct singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb

Incorrect plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb plural verb singular verb

Table 2: IOI dataset templates. Each clean prompt is truncated immediately before the indirect-object answer; the corrupted prompt replaces the cueing name with a distractor [C]. Template ABBA_00 ABBA_01 ABBA_02 ABBA_03 ABBA_04 ABBA_05 ABBA_06 BABA_00 BABA_01 BABA_02 BABA_03 BABA_04 BABA_05 BABA_06

Clean prompt template example Corrupted prompt template example Correct Then, [A] and [B] went to the [PLACE]. [B] Then, [A] and [B] went to the [PLACE]. [C] [A] gave a [OBJECT] to gave a [OBJECT] to Then, [A] and [B] had a lot of fun at the Then, [A] and [B] had a lot of fun at the [A] [PLACE]. [B] gave a [OBJECT] to [PLACE]. [C] gave a [OBJECT] to Then, [A] and [B] were working at the Then, [A] and [B] were working at the [A] [PLACE]. [B] decided to give a [OBJECT] [PLACE]. [C] decided to give a [OBJECT] to to Then, [A] and [B] were thinking about going Then, [A] and [B] were thinking about going [A] to the [PLACE]. [B] wanted to give a [OB- to the [PLACE]. [C] wanted to give a [OBJECT] to JECT] to Then, [A] and [B] had a long argument, and Then, [A] and [B] had a long argument, and [A] afterwards [B] said to afterwards [C] said to After [A] and [B] went to the [PLACE], [B] After [A] and [B] went to the [PLACE], [C] [A] gave a [OBJECT] to gave a [OBJECT] to When [A] and [B] got a [OBJECT] at the When [A] and [B] got a [OBJECT] at the [A] [PLACE], [B] decided to give it to [PLACE], [C] decided to give it to Then, [B] and [A] went to the [PLACE]. [B] Then, [B] and [A] went to the [PLACE]. [C] [A] gave a [OBJECT] to gave a [OBJECT] to Then, [B] and [A] had a lot of fun at the Then, [B] and [A] had a lot of fun at the [A] [PLACE]. [B] gave a [OBJECT] to [PLACE]. [C] gave a [OBJECT] to Then, [B] and [A] were working at the Then, [B] and [A] were working at the [A] [PLACE]. [B] decided to give a [OBJECT] [PLACE]. [C] decided to give a [OBJECT] to to Then, [B] and [A] were thinking about going Then, [B] and [A] were thinking about going [A] to the [PLACE]. [B] wanted to give a [OB- to the [PLACE]. [C] wanted to give a [OBJECT] to JECT] to Then, [B] and [A] had a long argument, and Then, [B] and [A] had a long argument, and [A] afterwards [B] said to afterwards [C] said to After [B] and [A] went to the [PLACE], [B] After [B] and [A] went to the [PLACE], [C] [A] gave a [OBJECT] to gave a [OBJECT] to When [B] and [A] got a [OBJECT] at the When [B] and [A] got a [OBJECT] at the [A] [PLACE], [B] decided to give it to [PLACE], [C] decided to give it to

15

Incorrect [B] [B] [B]

[B]

[B] [B] [B] [B] [B] [B]

[B]

[B] [B] [B]

Table 3: greater-than dataset template example. The task asks whether the next two-digit year suffix is greater than the suffix xy implied by the clean prompt. Template x

Clean prompt template example Corrupted prompt template example Correct The [NOUN] lasted from the year 13xy to The [NOUN] lasted from the year 1301 to suffix > xy the year 13 the year 13

Incorrect suffix ≤ xy

Table 4: single-double-quote dataset templates. Each base template is shown in both directions: _s expects the closing single-quote-parenthesis token, and _d expects the closing double-quoteparenthesis token. Template append_call_s

Clean prompt template example if [date bounds fail]: [RESULT_LIST].append('Invalid Input append_call_d if [date bounds fail]: [RESULT_LIST].append("Invalid Input constructor_call_s [OBJECT_VAR] = Project( '[PROJECT_NAME] constructor_call_d [OBJECT_VAR] = Project( "[PROJECT_NAME] for_loop_s for [ITEM_VAR] in [ITEM_SET]: if len([ITEM_VAR]) == 0: print('invalid for_loop_d for [ITEM_VAR] in [ITEM_SET]: if len([ITEM_VAR]) == 0: print("invalid function_call_s [FUNC_NAME]('[FILE_NAME] function_call_d [FUNC_NAME]("[FILE_NAME] if_clause_1_s if [COND]: print('There is no topological sort, the graph has a cycle if_clause_1_d if [COND]: print("There is no topological sort, the graph has a cycle if_clause_2_s if degree not in range(3, [LIMIT_VAR]-1): print('Degree should be in the range [3, [LIMIT_VAR]-1] if_clause_2_d if degree not in range(3, [LIMIT_VAR]-1): print("Degree should be in the range [3, [LIMIT_VAR]-1] if_clause_3_s if [ALT_VAR] < [DIST_VAR].get( [NODE_VAR], float('inf if_clause_3_d if [ALT_VAR] < [DIST_VAR].get( [NODE_VAR], float("inf method_call_s [GRAPH_VAR].read_data( '[GRAPH_FILE] method_call_d [GRAPH_VAR].read_data( "[GRAPH_FILE] nested_loop_s for [OUTER] in [SET]: for [INNER] in [OUTER]: if ([INNER] == ()): print('No [INNER] at the current iteration nested_loop_d for [OUTER] in [SET]: for [INNER] in [OUTER]: if ([INNER] == ()): print("No [INNER] at the current iteration print_stmt_1_s print('[PRINT_TEXT] print_stmt_1_d print("[PRINT_TEXT] print_stmt_2_s print('[USAGE_TEXT] print_stmt_2_d print("[USAGE_TEXT] print_stmt_3_s print('[QUESTION_TEXT] print_stmt_3_d print("[QUESTION_TEXT] print_stmt_4_s print('[STATUS_TEXT] print_stmt_4_d print("[STATUS_TEXT] raise_error_s raise ValueError('[ERROR_TEXT] raise_error_d raise ValueError("[ERROR_TEXT] string_split_s def [FUNC]([ARG]): [LEFT], [RIGHT] = [ARG].split('string_split_d def [FUNC]([ARG]): [LEFT], [RIGHT] = [ARG].split("strftime_call_s [TIME_VAR] = datetime.datetime.now() .strftime('%Y%m%d_%H%M%S strftime_call_d [TIME_VAR] = datetime.datetime.now() .strftime("%Y%m%d_%H%M%S

Corrupted prompt template example if [date bounds fail]: [RESULT_LIST].append("Invalid Input if [date bounds fail]: [RESULT_LIST].append('Invalid Input [OBJECT_VAR] = Project( "[PROJECT_NAME] [OBJECT_VAR] = Project( '[PROJECT_NAME] for [ITEM_VAR] in [ITEM_SET]: if len([ITEM_VAR]) == 0: print("invalid for [ITEM_VAR] in [ITEM_SET]: if len([ITEM_VAR]) == 0: print('invalid [FUNC_NAME]("[FILE_NAME] [FUNC_NAME]('[FILE_NAME] if [COND]: print("There is no topological sort, the graph has a cycle if [COND]: print('There is no topological sort, the graph has a cycle if degree not in range(3, [LIMIT_VAR]-1): print("Degree should be in the range [3, [LIMIT_VAR]-1] if degree not in range(3, [LIMIT_VAR]-1): print('Degree should be in the range [3, [LIMIT_VAR]-1] if [ALT_VAR] < [DIST_VAR].get( [NODE_VAR], float("inf if [ALT_VAR] < [DIST_VAR].get( [NODE_VAR], float('inf [GRAPH_VAR].read_data( "[GRAPH_FILE] [GRAPH_VAR].read_data( '[GRAPH_FILE] for [OUTER] in [SET]: for [INNER] in [OUTER]: if ([INNER] == ()): print("No [INNER] at the current iteration for [OUTER] in [SET]: for [INNER] in [OUTER]: if ([INNER] == ()): print('No [INNER] at the current iteration print("[PRINT_TEXT] print('[PRINT_TEXT] print("[USAGE_TEXT] print('[USAGE_TEXT] print("[QUESTION_TEXT] print('[QUESTION_TEXT] print("[STATUS_TEXT] print('[STATUS_TEXT] raise ValueError("[ERROR_TEXT] raise ValueError('[ERROR_TEXT] def [FUNC]([ARG]): [LEFT], [RIGHT] = [ARG].split("def [FUNC]([ARG]): [LEFT], [RIGHT] = [ARG].split('[TIME_VAR] = datetime.datetime.now() .strftime("%Y%m%d_%H%M%S [TIME_VAR] = datetime.datetime.now() .strftime('%Y%m%d_%H%M%S

16

Correct ')

Incorrect ")

")

')

')

")

")

')

')

")

")

')

') ") ')

") ') ")

")

')

')

")

")

')

')

")

")

')

')

")

")

')

')

")

")

')

') ") ') ") ') ") ') ") ') ") ')

") ') ") ') ") ') ") ') ") ') ")

")

')

')

")

")

')

Table 5: else-elif dataset templates. The task contrasts Python branches where else should be followed by :\n, while elif should be followed by any other token. In visualizations, template labels additionally carry _else or _elif suffixes for the two directions; these suffixes are omitted here for space, and the prompt examples show one representative direction. Template return_result

answer_check

append_guard

program_dir_chain

schedule_branch

strategy_branch

equality_case

stats_update

inline_threshold

string_align

binary_search

truthy_call

compare_objects

cell_toggle

random_split

none_default

Clean prompt template example if [RESULT]: [CACHE] = [RESULT] return [RESULT] else if eval([PROBLEM]) == [ANSWER]: print([MSG]) elif if [COUNT_MAP][[NODE].topic] >= [TARGET]: [LIST].append([NODE]) else if [DIR] == 0: [DIR] = [NEG] elif [DIR] == 1: [DIR] = [POS] elif if [TIME] > [LAST]: [START] = [LAST] + 1 else if [STRATEGY] == ’bfs’: return bfs(...) elif [STRATEGY] == ’dfs’: return dfs(...) elif if [CONSTRAINT].is_equality(): return [Z3] == -[COEFF][’1’] else if [ROOT] not in [STATS]: [STATS][[ROOT]] = ([AREA], [COUNT]) elif if ([NUM] <= [LIMIT]): return [FUNC_CALL] else if [CHAR] in [STRING]: [LEFT], [RIGHT] = [STRING].split([CHAR], 1) elif if [DATA][[INDEX]] < [VALUE]: [RESULT] = [INDEX] [LOW] = [INDEX] + 1 else if [ROOM]: set_parameter([BOARD], [PARAM], [ROOM_NAME] + ’ - ’ + [ROOM]) elif if isinstance([LEFT], (list, tuple)): if isinstance([RIGHT], (list, tuple)): compare_objects(...) else if [BOARD][[ROW]][[COL]] == [OFF]: [BOARD][[ROW]][[COL]] = 1 elif if random.random() < 0.5: [OUT].append( f"ML,{[PEER]},{[ELEMENT]}") else if [END] is None: [GOAL] = ([ROWS]-1, [COLS]-1) elif

Corrupted prompt template example if [RESULT]: [CACHE] = [RESULT] return [RESULT] elif if eval([PROBLEM]) == [ANSWER]: print([MSG]) else if [COUNT_MAP][[NODE].topic] >= [TARGET]: [LIST].append([NODE]) elif if [DIR] == 0: [DIR] = [NEG] elif [DIR] == 1: [DIR] = [POS] else if [TIME] > [LAST]: [START] = [LAST] + 1 elif if [STRATEGY] == ’bfs’: return bfs(...) elif [STRATEGY] == ’dfs’: return dfs(...) else if [CONSTRAINT].is_equality(): return [Z3] == -[COEFF][’1’] elif if [ROOT] not in [STATS]: [STATS][[ROOT]] = ([AREA], [COUNT]) else if ([NUM] <= [LIMIT]): return [FUNC_CALL] elif if [CHAR] in [STRING]: [LEFT], [RIGHT] = [STRING].split([CHAR], 1) else if [DATA][[INDEX]] < [VALUE]: [RESULT] = [INDEX] [LOW] = [INDEX] + 1 elif if [ROOM]: set_parameter([BOARD], [PARAM], [ROOM_NAME] + ’ - ’ + [ROOM]) else if isinstance([LEFT], (list, tuple)): if isinstance([RIGHT], (list, tuple)): compare_objects(...) elif if [BOARD][[ROW]][[COL]] == [OFF]: [BOARD][[ROW]][[COL]] = 1 else if random.random() < 0.5: [OUT].append( f"ML,{[PEER]},{[ELEMENT]}") elif if [END] is None: [GOAL] = ([ROWS]-1, [COLS]-1) else

17

Correct :\n

Incorrect other tokens

other tokens :\n

:\n

other tokens

other tokens :\n

:\n

other tokens

other tokens :\n

:\n

other tokens

other tokens :\n

:\n

other tokens

other tokens :\n

:\n

other tokens

other tokens :\n

:\n

other tokens

other tokens :\n

:\n

other tokens

other tokens :\n

E

Proof of Theorem 3.5

Proof. We first consider the CEAP scoring function. Given any branch b ∈ B, suppose the edge has size y ∈ Rdb , we have: Z 1 X X ∂F (x0 + α(x1 − x0 )) ∂yb i ICond (yb ) = (x1j − x0j ) dα ∂yb i ∂xj 0 i∈[db ] j∈[din ] X Z 1 ∂F (x0 + α(x1 − x0 )) X ∂yb i = dα(x1j − x0j ) ∂y ∂x bi j i∈[db ] 0 j∈[din ] {z } | (=dyb i )

=

X Z 1 ∂fb (yb (α)) i

i∈[db ]

0

∂yb i

Z dyb i (α) =

X ∂fb (yb ) i

∂yb i

γyb i∈[d ] b

dyb i

Z (∇yb fb ) · dr = fb (x1 ) − fb (x0 ) = ∆fb .

= γyb

In the above, γyb is the trajectory that yb follows when α goes from 0 to 1, and the second last equality is due to that the integrand is a gradient, and thus the result is path-independent. The above derivation shows that conductance satisfies additive order preservation, almost by design. On the other hand, we can come up with the following counterexample for IG. Suppose we are studying a two-branched function: F = g1 (y1 ) + g2 (y2 ) = g1 (f1 (x)) + g2 (f2 (x)), where   1 1     y1 , y1 < , x, x < 1,     3 3   f2 (x) = x,   2 1 1 4 g1 (y1 ) = f1 (x) = x − , 1 ≤ x < 2, , ≤ y1 < , 1   3 3 3 3 g2 (y2 ) = y2 .     3  1   y1 − 1, y1 ≥ 4 ,  x + 2 , x ≥ 2, 3 3 3 We study the process where the input x is moved from 0 to 3. In this case, there are two branches, denoted by set B = {1, 2}. We can compute that |∆f1 | = 32 , and |∆f2 | = 1. We can also compute: Z Z 5 3 ∂F 5 1 ∂F dα = dx (7) IIG (y1 ) = 3 0 ∂y1 x=3α 3 0 ∂y1 x Z 1  Z 2 Z 3 5 10 . (8) = dx + 0dx + dx = 3 3 0 1 2 Z 1 IIG (y2 ) = 3 0

∂F dα = 3 ∂y2 x=3α

Z 3 0

∂F dx = 3 ∂y2 x

Z 3 0

1 dx = 3. 3

Hence |∆f1 | < |∆f2 | and yet |IIG (y1 )| > |IIG (y2 )|, breaking additive order preservation. Remark E.1. One might notice that g1 (·) above is not differentiable, and thus we are implicitly using ∂F the one-sided derivative in place of ∂y . However, our statement should still hold for such non-smooth 1 functions. It is easy to see that non-differentiability is not the root cause of IG breaking additive order preservation. First, in the counterexample, the non-differentiability occurs on a Lebesgue-null set and hence does not affect the integral. Second, we can smoothen the kinks in g1 (·) using techniques like huberized ReLU [36] or softplus [37], where the kinks are replaced with smooth arcs whose span can be arbitrarily small. Thus, IIG (y1 ) for the smoothened g1 can be arbitrarily close to what we compute above. Constructing such g1 might involve tedious notation and blur the main message. We settle for the current counterexample for its simplicity and clarity: The reason why the magnitude of IIG (y1 ) relative to IIG (y2 ) seems to lose its meaning is that the integrals receives the same weight 5 3 in Equation (8), while, in fact, the weighting should reflect how much y1 moves when x moves through each of the three integration intervals.

18

Remark E.2. The fact that conductance satisfies additive order preservation may be understood as a result of linearity and partition consistency [17] of conductance. Nevertheless, these two properties are not necessary to satisfy additive order preservation. In fact, if we engineer a new scoring method, which simply assigns to each edge the squared conductance score, additive order preservation is still satisfied, while linearity and partition consistency will be breached.

F

Complete Results for Pairwise Jaccard Index and Unfaithfulness

In this section, we compare unfaithfulness and pairwise Jaccard index yielded by CEAP and EAP-IG. Overall, we found that while CEAP does not have a consistent advantage over EAP-IG in terms of unfaithfulness, it clearly outperforms the latter for pairwise Jaccard index, or at the very least matches it. Specifically, for unfaithfulness, the quicker it drops with the increase of the number of edges, the better. This is based on the principle of minimality [7]. For the pairwise Jaccard index, the higher, the better. A circuit-finding scheme could in principle game the PJI metric by always selecting the same circuit with no regard to the task. We check that this is not the case for CEAP: it achieves low unfaithfulness across the tasks shown below. Nonetheless, we do not find compelling evidence that CEAP consistently attains lower unfaithfulness than EAP-IG. For instance, on GPT-2 XL, CEAP achieves lower unfaithfulness with smaller circuits for nearly all SVA templates, but the outcomes for IOI and greater-than are inconclusive. This is understandable: both CEAP and EAP-IG are pathintegral methods that characterize trajectories in the activation space as inputs move from corrupted samples to their clean counterparts. However, once we patch the activations of the excluded edges, the resulting trajectory no longer aligns with the paths used to assign their scores. Compounded with the nonlinearity of the network, this introduces errors that are difficult to control or even quantify and can drown out the advantage of CEAP over EAP-IG. F.1

GPT-2 XL

F.1.1

SVA EAP-IG prob diff

CEAP prob diff

plural_0

plural_1

plural_2

plural_3

plural_4

plural_5

plural_6

plural_7

plural_8

plural_9

plural_10

plural_11

0.8 0.6 0.4 0.2

0.8

not enough data

0.6 0.4

Pairwise Jaccard index

0.2

plural_12

plural_13

plural_14

single_0

single_1

single_2

single_3

single_4

single_5

single_6

single_7

single_8

single_9

single_10

single_11

single_12

single_13

single_14

0.8 0.6 0.4 0.2

0.8 0.6 0.4 0.2

0.8 0.6 0.4 0.2

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

Number of edges

Figure 9: Pairwise Jaccard index vs. number of edges for GPT-2 XL on SVA.

19

16k

24k

EAP-IG prob diff

CEAP prob diff

plural_0

plural_1

plural_2

plural_3

plural_4

plural_5

plural_6

plural_7

plural_8

plural_9

plural_10

plural_11

plural_12

plural_13

plural_14

single_0

single_1

single_2

single_3

single_4

single_5

single_6

single_7

single_8

single_9

single_10

single_11

single_12

single_13

single_14

1.00 0.75 0.50 0.25

1.00 0.75 0.50

Unfaithfulness

0.25

1.00 0.75 0.50 0.25

1.00 0.75 0.50 0.25

1.00 0.75 0.50 0.25 0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

Number of edges

Figure 10: Unfaithfulness vs. number of edges for GPT-2 XL on SVA. F.1.2

IOI

0.88

ABBA_00

ABBA_01

EAP-IG logit diff ABBA_02

ABBA_06

BABA_00

BABA_01

BABA_05

BABA_06

CEAP logit diff ABBA_03

ABBA_04

ABBA_05

BABA_02

BABA_03

BABA_04

0.80 0.72

Pairwise Jaccard index

0.64

0.88 0.80 0.72 0.64

0 0.88

40k

80k

120k

160k

0

40k

80k

120k

160k

0

40k

80k

120k

160k

0

40k

0.80 0.72 0.64 0

40k

80k

120k

160k

0

40k

80k

120k

160k

Number of edges

Figure 11: Pairwise Jaccard index vs. number of edges for GPT-2 XL on IOI.

20

80k

120k

160k

ABBA_00

ABBA_01

EAP-IG logit diff ABBA_02

ABBA_06

BABA_00

BABA_01

BABA_05

BABA_06

CEAP logit diff ABBA_03

ABBA_04

ABBA_05

BABA_02

BABA_03

BABA_04

0.8 0.6 0.4

Unfaithfulness

0.2

0.8 0.6 0.4 0.2 0

40k

80k

120k

160k

0

40k

80k

120k

160k

0

40k

80k

120k

160k

0

40k

80k

120k

160k

15k

20k

15k

20k

0.8 0.6 0.4 0.2 0

40k

80k

120k

160k

0

40k

80k

120k

160k

Number of edges

Figure 12: Unfaithfulness vs. number of edges for GPT-2 XL on IOI. F.1.3

Greater-than 0

1

EAP-IG prob diff 2

6

7

8

CEAP prob diff 3

4

5

0.96 0.90

Pairwise Jaccard index

0.84 0.78 0.72 0

5k

10k

15k

20k

0

5k

10k

9

0.96 0.90 0.84 0.78 0.72 0

5k

10k

15k

20k

0

5k

10k

15k

20k

0

5k

10k

15k

20k

0

Number of edges

5k

10k

15k

20k

Figure 13: Pairwise Jaccard index vs. number of edges for GPT-2 XL on Greater-than.

0

1

EAP-IG prob diff 2

6

7

8

CEAP prob diff 3

4

5

0.8 0.6 0.4

Unfaithfulness

0.2 0

5k

10k

15k

20k

0

5k

9

0.8 0.6 0.4 0.2 0

5k

10k

15k

20k

0

5k

10k

15k

20k

0

5k

10k

15k

20k

0

Number of edges

5k

10k

15k

20k

Figure 14: Unfaithfulness vs. number of edges for GPT-2 XL on Greater-than.

21

10k

F.2

Pythia-2.8B

F.2.1

SVA EAP-IG prob diff

CEAP prob diff

plural_0

plural_1

plural_2

plural_3

plural_4

plural_5

plural_6

plural_7

plural_8

plural_9

plural_10

plural_11

0.8 0.6 0.4 0.2

0.8

not enough data

0.6 0.4

Pairwise Jaccard index

0.2

plural_12

plural_13

plural_14

single_0

single_1

single_2

single_3

single_4

single_5

single_6

single_7

single_8

single_9

single_10

single_11

single_12

single_13

single_14

0.8 0.6 0.4 0.2

0.8 0.6 0.4 0.2

0.8 0.6 0.4 0.2

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

Number of edges

Figure 15: Pairwise Jaccard index vs. number of edges for Pythia-2.8B on SVA.

EAP-IG prob diff

CEAP prob diff

plural_0

plural_1

plural_2

plural_3

plural_4

plural_5

plural_6

plural_7

plural_8

plural_9

plural_10

plural_11

plural_12

plural_13

plural_14

single_0

single_1

single_2

single_3

single_4

single_5

single_6

single_7

single_8

single_9

single_10

single_11

single_12

single_13

single_14

0.60 0.45 0.30 0.15

0.60 0.45 0.30

Unfaithfulness

0.15

0.60 0.45 0.30 0.15

0.60 0.45 0.30 0.15

0.60 0.45 0.30 0.15 0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

8k

16k

24k

0

Number of edges

Figure 16: Unfaithfulness vs. number of edges for Pythia-2.8B on SVA.

22

8k

16k

24k

F.2.2

IOI

0.88

ABBA_00

ABBA_01

EAP-IG logit diff ABBA_02

ABBA_06

BABA_00

BABA_01

BABA_05

BABA_06

CEAP logit diff ABBA_03

ABBA_04

ABBA_05

BABA_02

BABA_03

BABA_04

0.80 0.72

Pairwise Jaccard index

0.64

0.88 0.80 0.72 0.64

0 0.88

20k

40k

60k

80k

0

20k

40k

60k

80k

0

20k

40k

60k

80k

0

20k

40k

60k

80k

60k

80k

15k

20k

0.80 0.72 0.64 0

20k

40k

60k

80k

0

20k

40k

60k

80k

Number of edges

Figure 17: Pairwise Jaccard index vs. number of edges for Pythia-2.8B on IOI.

ABBA_00

ABBA_01

EAP-IG logit diff ABBA_02

ABBA_06

BABA_00

BABA_01

BABA_05

BABA_06

CEAP logit diff ABBA_03

ABBA_04

ABBA_05

BABA_02

BABA_03

BABA_04

0.8 0.6 0.4 0.2

Unfaithfulness

0.8 0.6 0.4 0.2 0

20k

40k

60k

80k

0

20k

40k

60k

80k

0

20k

40k

60k

80k

0

20k

40k

0.8 0.6 0.4 0.2 0

20k

40k

60k

80k

0

20k

40k

60k

80k

Number of edges

Figure 18: Unfaithfulness vs. number of edges for Pythia-2.8B on IOI. F.2.3

Greater-than 0

1

EAP-IG prob diff 2

6

7

8

CEAP prob diff 3

4

5

0.96 0.90

Pairwise Jaccard index

0.84 0.78 0.72 0

5k

10k

15k

20k

0

5k

10k

9

0.96 0.90 0.84 0.78 0.72 0

5k

10k

15k

20k

0

5k

10k

15k

20k

0

5k

10k

15k

20k

0

Number of edges

5k

10k

15k

20k

Figure 19: Pairwise Jaccard index vs. number of edges for Pythia-2.8B on Greater-than.

23

0.40

0

1

EAP-IG prob diff 2

6

7

8

CEAP prob diff 3

4

5

0.32 0.24

Unfaithfulness

0.16 0.08 0 0.40

5k

10k

15k

20k

0

5k

10k

15k

20k

1.5k

2k

9

0.32 0.24 0.16 0.08 0

5k

10k

15k

20k

0

5k

10k

15k

20k

0

5k

10k

15k

20k

0

Number of edges

5k

10k

15k

20k

Figure 20: Unfaithfulness vs. number of edges for Pythia-2.8B on Greater-than. F.3

GPT-2 small

F.3.1

SVA EAP-IG prob diff

CEAP prob diff

plural_0

plural_1

plural_2

plural_3

plural_4

plural_5

plural_6

plural_7

plural_8

plural_9

plural_10

plural_11

0.8 0.6 0.4 0.2

0.8

not enough data

0.6 0.4

Pairwise Jaccard index

0.2

plural_12

plural_13

plural_14

single_0

single_1

single_2

single_3

single_4

single_5

single_6

single_7

single_8

single_9

single_10

single_11

single_12

single_13

single_14

0.8 0.6 0.4 0.2

0.8 0.6 0.4 0.2

0.8 0.6 0.4 0.2

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

Number of edges

Figure 21: Pairwise Jaccard index vs. number of edges for GPT-2 small on SVA.

24

1k

EAP-IG prob diff

CEAP prob diff

plural_0

plural_1

plural_2

plural_3

plural_4

plural_5

plural_6

plural_7

plural_8

plural_9

plural_10

plural_11

plural_12

plural_13

plural_14

single_0

single_1

single_2

single_3

single_4

single_5

single_6

single_7

single_8

single_9

single_10

single_11

single_12

single_13

single_14

1.00 0.75 0.50 0.25

1.00 0.75 0.50

Unfaithfulness

0.25

1.00 0.75 0.50 0.25

1.00 0.75 0.50 0.25

1.00 0.75 0.50 0.25 0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

4.5k

6k

Number of edges

Figure 22: Unfaithfulness vs. number of edges for GPT-2 small on SVA. F.3.2

IOI

0.96

ABBA_00

ABBA_01

EAP-IG logit diff ABBA_02

ABBA_06

BABA_00

BABA_01

BABA_05

BABA_06

CEAP logit diff ABBA_03

ABBA_04

ABBA_05

BABA_02

BABA_03

BABA_04

0.88 0.80 0.72

Pairwise Jaccard index

0.64

0.96 0.88 0.80 0.72 0.64

0 0.96

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

0

1.5k

0.88 0.80 0.72 0.64 0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

Number of edges

Figure 23: Pairwise Jaccard index vs. number of edges for GPT-2 small on IOI.

25

3k

ABBA_00

ABBA_01

EAP-IG logit diff ABBA_02

ABBA_06

BABA_00

BABA_01

BABA_05

BABA_06

CEAP logit diff ABBA_03

ABBA_04

ABBA_05

BABA_02

BABA_03

BABA_04

1.00 0.75 0.50

Unfaithfulness

0.25

1.00 0.75 0.50 0.25 0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

750

1k

750

1k

1.00 0.75 0.50 0.25 0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

Number of edges

Figure 24: Unfaithfulness vs. number of edges for GPT-2 small on IOI. F.3.3

Greater-than 0

1

EAP-IG prob diff 2

6

7

8

CEAP prob diff 3

4

5

0.96 0.90

Pairwise Jaccard index

0.84 0.78 0.72 0

250

500

750

1k

0

250

500

9

0.96 0.90 0.84 0.78 0.72 0

250

500

750

1k

0

250

500

750

1k

0

250

500

750

1k

0

Number of edges

250

500

750

1k

Figure 25: Pairwise Jaccard index vs. number of edges for GPT-2 small on Greater-than.

0

1

EAP-IG prob diff 2

6

7

8

CEAP prob diff 3

4

5

1.00 0.75 0.50

Unfaithfulness

0.25 0

250

500

750

1k

0

250

9

1.00 0.75 0.50 0.25 0

250

500

750

1k

0

250

500

750

1k

0

250

500

750

1k

0

Number of edges

250

500

750

1k

Figure 26: Unfaithfulness vs. number of edges for GPT-2 small on Greater-than.

26

500

F.4

Pythia-160M

F.4.1

SVA EAP-IG prob diff

CEAP prob diff

plural_0

plural_1

plural_2

plural_3

plural_4

plural_5

plural_6

plural_7

plural_8

plural_9

plural_10

plural_11

0.8 0.6 0.4 0.2

Pairwise Jaccard index

0.8 0.6 0.4 0.2

not enough data

plural_12

plural_13

plural_14

single_0

single_1

single_2

single_3

single_4

single_5

single_6

single_7

single_8

single_9

single_10

single_11

single_12

single_13

single_14

0.8 0.6 0.4 0.2

0.8 0.6 0.4 0.2

0.8 0.6 0.4 0.2 0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

1.5k

2k

Number of edges

Figure 27: Pairwise Jaccard index vs. number of edges for Pythia-160M on SVA.

EAP-IG prob diff

CEAP prob diff

plural_0

plural_1

plural_2

plural_3

plural_4

plural_5

plural_6

plural_7

plural_8

plural_9

plural_10

plural_11

plural_12

plural_13

plural_14

single_0

single_1

single_2

single_3

single_4

single_5

single_6

single_7

single_8

single_9

single_10

single_11

single_12

single_13

single_14

1.00 0.75 0.50 0.25

1.00 0.75 0.50

Unfaithfulness

0.25

1.00 0.75 0.50 0.25

1.00 0.75 0.50 0.25

1.00 0.75 0.50 0.25 0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

500

1k

1.5k

2k

0

Number of edges

Figure 28: Unfaithfulness vs. number of edges for Pythia-160M on SVA.

27

500

1k

F.4.2

IOI ABBA_00

ABBA_01

EAP-IG logit diff ABBA_02

ABBA_06

BABA_00

BABA_01

BABA_05

BABA_06

CEAP logit diff ABBA_03

ABBA_04

ABBA_05

BABA_02

BABA_03

BABA_04

0.9 0.8 0.7 0.6

Pairwise Jaccard index

0.5

0.9 0.8 0.7 0.6 0.5

0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

4.5k

6k

750

1k

0.9 0.8 0.7 0.6 0.5

0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

Number of edges

Figure 29: Pairwise Jaccard index vs. number of edges for Pythia-160M on IOI.

ABBA_00

ABBA_01

EAP-IG logit diff ABBA_02

ABBA_06

BABA_00

BABA_01

BABA_05

BABA_06

CEAP logit diff ABBA_03

ABBA_04

ABBA_05

BABA_02

BABA_03

BABA_04

0.8 0.6 0.4

Unfaithfulness

0.2

0.8 0.6 0.4 0.2 0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

0

1.5k

3k

0.8 0.6 0.4 0.2 0

1.5k

3k

4.5k

6k

0

1.5k

3k

4.5k

6k

Number of edges

Figure 30: Unfaithfulness vs. number of edges for Pythia-160M on IOI. F.4.3

Greater-than 0

1

EAP-IG prob diff 2

6

7

8

CEAP prob diff 3

4

5

0.9 0.8

Pairwise Jaccard index

0.7 0.6 0.5

0

250

500

750

1k

0

250

500

9

0.9 0.8 0.7 0.6 0.5

0

250

500

750

1k

0

250

500

750

1k

0

250

500

750

1k

0

Number of edges

250

500

750

1k

Figure 31: Pairwise Jaccard index vs. number of edges for Pythia-160M on Greater-than.

28

0

1

EAP-IG prob diff 2

6

7

8

CEAP prob diff 3

4

5

0.8 0.6

Unfaithfulness

0.4 0.2 0

250

500

750

1k

0

250

500

750

1k

9

0.8 0.6 0.4 0.2 0

250

500

750

1k

0

250

500

750

1k

0

250

500

750

1k

0

Number of edges

250

500

750

1k

Figure 32: Unfaithfulness vs. number of edges for Pythia-160M on Greater-than.

G Template Absolute-Score Rank UMAP and Sample-Level Circuit Overlap Visualizations This section pairs absolute-score rank vector UMAP embeddings of sample-level edge scores with sample-by-sample pairwise Jaccard index matrices for the greedily selected circuits. We include GPT-2 small and Pythia-160M on SVA, IOI, and greater-than. For the PJI matrix visualizations of GPT-2 small and Pythia-160M, we choose graph sizes large enough so that the average unfaithfulness across all samples falls below 0.2. The only exception is GPT-2 small on IOI, where we fix the graph size to 6000 edges (about 20% of all possible edges), which is the maximum graph size we experimented with; even at this size, the unfaithfulness remains above 0.2. GPT-2 small SVA single_0 plural_0 single_1 plural_1 single_2 plural_2 single_3 plural_3 single_4 plural_4 single_5 plural_5 single_6 plural_6 single_7 plural_7 single_8 plural_8 single_9 plural_9 plural_10 single_11 plural_11 single_12 plural_12 single_13 plural_13 single_14 plural_14

10

8

6

4

2

number of edges in the circuit for each sample 826 number of edges of the union of all circuits = 8735 total number of edges in the entire graph = 32491 avg. unfaithfulness = 0.195

0.8

plural_1 plural_5

0.7

plural_8 plural_9 plural_10 plural_11

0.6

plural_12 plural_13 plural_14

0.5

single_0 single_1

0.4

single_5 single_8

0.3

single_9 single_11 single_12

0.2

single_13

(a) Absolute-score rank UMAP.

Figure 33: Template-induced circuit difference for GPT-2 small on SVA.

29

single_14

single_13

single_9

(b) Pairwise Jaccard index.

single_12

single_11

single_8

single_5

single_1

Template

single_0

plural_13 plural_14

plural_12

20

plural_9 plural_10 plural_11

10

plural_8

0

plural_5

10

Absolute-score rank UMAP 1

plural_1

20

plural_0

single_14

0

0.9

plural_0

single_2 plural_2 single_3 plural_3 single_4 plural_4 single_6 plural_6 single_7 plural_7

Absolute-score rank UMAP 2

single_2 plural_2 single_3 plural_3 single_4 plural_4 single_6 plural_6 single_7 plural_7

template_label

12

Pairwise Jaccard index

G.1.1

Template

G.1

G.1.2

IOI template_label

0

5

ABBA_01

0.55

0.50

ABBA_02 ABBA_03

Pairwise Jaccard index

Absolute-score rank UMAP 2

5

number of edges in the circuit for each sample 6000 number of edges of the union of all circuits = 24711 total number of edges in the entire graph = 32491 avg. unfaithfulness = 0.240

ABBA_00

ABBA_04 ABBA_05

Template

ABBA_00 ABBA_01 ABBA_02 ABBA_03 ABBA_04 ABBA_05 ABBA_06 BABA_00 BABA_01 BABA_02 BABA_03 BABA_04 BABA_05 BABA_06

10

ABBA_06

0.45

BABA_00 BABA_01 BABA_02 BABA_03

10

0.40

BABA_04 BABA_05

15

(a) Absolute-score rank UMAP.

BABA_06

BABA_05

BABA_04

BABA_03

BABA_02

Template

BABA_01

BABA_00

ABBA_06

ABBA_05

5

ABBA_04

0

ABBA_03

5

ABBA_02

10

Absolute-score rank UMAP 1

ABBA_01

15

ABBA_00

BABA_06

0.35 0.30

(b) Pairwise Jaccard index.

Figure 34: Template-induced circuit difference for GPT-2 small on IOI. Greater-than template_label

10

1 2

10

20

30

4

0.8

0.6

5 6 7

40

1.0 0.9

0.7

3

Template

0.5

8 0.4

9

(a) Absolute-score rank UMAP.

9

0.3

Template

8

10

7

5

6

0

5

5

4

10

Absolute-score rank UMAP 1

3

15

2

20

1

50

0

Absolute-score rank UMAP 2

0

number of edges in the circuit for each sample 162 number of edges of the union of all circuits = 943 total number of edges in the entire graph = 32491 avg. unfaithfulness = 0.180

0

0 1 2 3 4 5 6 7 8 9

Pairwise Jaccard index

G.1.3

(b) Pairwise Jaccard index.

Figure 35: Template-induced circuit difference for GPT-2 small on greater-than.

30

Pythia-160M

G.2.1

SVA single_2 plural_2 single_3 plural_3 single_4 plural_4 single_6 plural_6 single_7 plural_7

template_label

14

Absolute-score rank UMAP 2

13

12

11

10

number of edges in the circuit for each sample 1438 number of edges of the union of all circuits = 14396 total number of edges in the entire graph = 32347 avg. unfaithfulness = 0.175 0.9

plural_0

0.8

plural_1 plural_5 plural_8

0.7

plural_9 plural_10 plural_11

Template

single_0 plural_0 single_1 plural_1 single_2 plural_2 single_3 plural_3 single_4 plural_4 single_5 plural_5 single_6 plural_6 single_7 plural_7 single_8 plural_8 single_9 plural_9 plural_10 single_11 plural_11 single_12 plural_12 single_13 plural_13 single_14 plural_14

Pairwise Jaccard index

G.2

0.6

plural_12 plural_13 plural_14

0.5

single_0 single_1 single_5

0.4

single_8 single_9

0.3

single_11 single_12

9

single_13

0.2

(a) Absolute-score rank UMAP.

single_14

single_13

single_9

single_12

single_11

single_8

single_5

single_1

Template

single_0

plural_13 plural_14

20

plural_12

15

plural_9 plural_10 plural_11

10

plural_8

5

plural_5

0

plural_1

5

Absolute-score rank UMAP 1

single_2 plural_2 single_3 plural_3 single_4 plural_4 single_6 plural_6 single_7 plural_7

10

plural_0

single_14

(b) Pairwise Jaccard index.

Figure 36: Template-induced circuit difference for Pythia-160M on SVA. IOI template_label

40

ABBA_01

ABBA_03 ABBA_04 ABBA_05

0.40

ABBA_06 BABA_00 BABA_01 BABA_02

60

0.50 0.45

ABBA_02

0.35

BABA_03 BABA_04

(a) Absolute-score rank UMAP.

(b) Pairwise Jaccard index.

Figure 37: Template-induced circuit difference for Pythia-160M on IOI.

31

BABA_06

BABA_05

BABA_04

BABA_03

BABA_02

Template

BABA_01

BABA_00

Absolute-score rank UMAP 1

0.25

0

ABBA_06

10

ABBA_05

20

ABBA_04

30

ABBA_03

40

0.30

BABA_06 ABBA_02

50

BABA_05

ABBA_01

80

ABBA_00

Absolute-score rank UMAP 2

20

number of edges in the circuit for each sample 2313 number of edges of the union of all circuits = 17666 total number of edges in the entire graph = 32347 avg. unfaithfulness = 0.200

ABBA_00

Template

ABBA_00 ABBA_01 ABBA_02 ABBA_03 ABBA_04 ABBA_05 ABBA_06 BABA_00 BABA_01 BABA_02 BABA_03 BABA_04 BABA_05 BABA_06

0

Pairwise Jaccard index

G.2.2

Greater-than

120

template_label

0.7

3

0.6

4 5

0.5

6 7

40

0.4

8

0.3

9

(a) Absolute-score rank UMAP.

9

8

0.2

Template

7

4

6

6

5

8

4

10

3

12

2

14

Absolute-score rank UMAP 1

1

16

0

20

0.8

2

80

60

1.0 0.9

1

Template

Absolute-score rank UMAP 2

100

number of edges in the circuit for each sample 60 number of edges of the union of all circuits = 325 total number of edges in the entire graph = 32347 avg. unfaithfulness = 0.176

0

0 1 2 3 4 5 6 7 8 9

Pairwise Jaccard index

G.2.3

(b) Pairwise Jaccard index.

Figure 38: Template-induced circuit difference for Pythia-160M on greater-than.

H Patching Circuits Found for a Sample Might Act Opposite to Intention on Another Sample Quantity

Sample 93

Sample 858

Clean prompt

The senators like to watch television shows and The senator likes to watch television shows and

The teachers the parent

Corrupted prompt Clean prob_diff Own-circuit prob_diff Cross-circuit prob_diff

0.352503 0.336449 0.256708

The teachers the parents 0.119733 0.072026 −0.223778

Table 6: Cross-sample circuit patching between samples 93 and 858 of IOI. The own-circuit rows evaluate each sample using the circuit discovered on that sample, whereas the cross-circuit rows evaluate each sample using the circuit discovered on the other sample.

We use the saved per-sample greedy circuits for GPT-2 on the SVA task, with the prob_diff metric and target circuit size 826. For a given sample here, the standard evaluation patches activations in the circuit so as to recover the movement from the pro-singular corrupted prompt toward the pro-plural prompt. The cross-sample condition keeps the input pair fixed but replaces the circuit with the circuit discovered on the other sample, thereby testing whether the learned pathway transfers across examples. Although the two circuits are comparable in size after pruning,12 with 712 edges for sample 93 and 754 edges for sample 858, their edge-level overlap is small (IoU = 0.125960). This low overlap is accompanied by a qualitative failure of transfer: when the circuit from sample 93 is used to patch sample 858, the prob_diff moves from 0.119733 under the clean full-model evaluation to −0.223778 under cross-circuit evaluation. Thus the transferred circuit does not merely recover less of the intended effect; it drives the logit movement in the opposite direction from the intended pro-plural intervention. 12 A procedure implemented by [5] to remove headless and tailless branches.

32

I

Template Absolute-Score Rank UMAP and Sample-Level Circuit Overlap Visualizations for Models by [13]

We repeat the visualization for three models open-sourced by [13]: • dense1_4x, • csp_sweep1_4x_3.7Mnonzero_afrac1.000, • csp_sweep1_4x_3.7Mnonzero_afrac0.500. Among these, dense1_4x and csp_sweep1_4x_3.7Mnonzero_afrac1.000 form the closest dense-sparse pair by test loss; the latter has 3.7M nonzero parameters out of 110M total parameters. The difference between the last two models (as indicated by name: afrac1.000 versus afrac0.500) is that the latter imposed activation sparsity on top of weight sparsity. Since the unfaithfulness of all models on both tasks is very low (see Tables 7 and 8) but can still differ by two orders of magnitude, we use circuits containing roughly 5.2% of all graph edges for the PJI plots, rather than choosing an edge count based on unfaithfulness. Tables 7 and 8 shows that cross-template overlap remains low, and the visualizations below show that sparse models still deploy different circuits for different templates. single-double-quote dense1_4x template_label

Pairwise Jaccard index

0.5

print_stmt_3_s print_stmt_4_d print_stmt_4_s raise_error_d

0.4

raise_error_s strftime_call_d strftime_call_s string_split_d

(a) Absolute-score rank UMAP.

string_split_s

string_split_d

raise_error_s

raise_error_d

strftime_call_s

strftime_call_d

print_stmt_4_s

print_stmt_3_s

print_stmt_2_s

print_stmt_4_d

print_stmt_3_d

print_stmt_2_d

nested_loop_s

nested_loop_d

print_stmt_1_s

20

print_stmt_1_d

string_split_s if_clause_3_s

15

print_stmt_2_s print_stmt_3_d

method_call_s

10

print_stmt_2_d

if_clause_2_s

5

0.6

print_stmt_1_s

if_clause_3_d

0

nested_loop_s print_stmt_1_d

method_call_d

5

nested_loop_d

if_clause_1_s

10

Absolute-score rank UMAP 1

0.7

if_clause_3_s method_call_s

if_clause_2_d

15

if_clause_3_d method_call_d

if_clause_1_d

15

if_clause_2_s

for_loop_s

10

0.8

if_clause_1_s if_clause_2_d

function_call_s

5

if_clause_1_d

for_loop_d

0

for_loop_s function_call_s

function_call_d

5

0.9

for_loop_d function_call_d

append_call_s

10

number of edges in the circuit for each sample 4029 number of edges of the union of all circuits = 23781 total number of edges in the entire graph = 76559 avg. unfaithfulness = 0.003

append_call_s constructor_call_s

append_call_d

Absolute-score rank UMAP 2

15

append_call_d constructor_call_d

Template

append_call_s append_call_d constructor_call_s constructor_call_d for_loop_s for_loop_d function_call_s function_call_d if_clause_1_s if_clause_1_d if_clause_2_s if_clause_2_d if_clause_3_s if_clause_3_d method_call_s method_call_d nested_loop_s nested_loop_d print_stmt_1_s print_stmt_1_d print_stmt_2_s print_stmt_2_d print_stmt_3_s print_stmt_3_d print_stmt_4_s print_stmt_4_d raise_error_s raise_error_d strftime_call_s strftime_call_d string_split_s string_split_d

20

constructor_call_s

I.1.1

constructor_call_d

I.1

Template

(b) Pairwise Jaccard index.

Figure 39: Template-induced circuit difference for dense1_4x on single-double-quote.

33

csp_sweep1_4x_3.7Mnonzero_afrac1.000 template_label

Pairwise Jaccard index

print_stmt_1_s print_stmt_2_d

0.5

print_stmt_2_s print_stmt_3_d print_stmt_3_s print_stmt_4_d

0.4

print_stmt_4_s raise_error_d raise_error_s strftime_call_d

0.3

strftime_call_s string_split_d string_split_s

string_split_d

raise_error_s

raise_error_d

strftime_call_s

strftime_call_d

print_stmt_4_s

print_stmt_3_s

print_stmt_2_s

print_stmt_4_d

print_stmt_3_d

print_stmt_2_d

nested_loop_s

nested_loop_d

print_stmt_1_s

print_stmt_1_d

if_clause_3_s

method_call_s

if_clause_2_s

20

if_clause_3_d

string_split_s method_call_d

10

0.6

nested_loop_s

if_clause_1_s

0

method_call_s nested_loop_d print_stmt_1_d

if_clause_2_d

10

Absolute-score rank UMAP 1

0.7

if_clause_3_s method_call_d

if_clause_1_d

20

if_clause_2_s if_clause_3_d

for_loop_s

15

if_clause_1_s if_clause_2_d

function_call_s

10

0.8

if_clause_1_d

for_loop_d

5

function_call_s

function_call_d

0

for_loop_s

append_call_s

5

0.9

for_loop_d function_call_d

constructor_call_s

10

number of edges in the circuit for each sample 18623 number of edges of the union of all circuits = 106191 total number of edges in the entire graph = 353837 avg. unfaithfulness = 0.003

append_call_s constructor_call_s

append_call_d

Absolute-score rank UMAP 2

15

append_call_d constructor_call_d

Template

append_call_s append_call_d constructor_call_s constructor_call_d for_loop_s for_loop_d function_call_s function_call_d if_clause_1_s if_clause_1_d if_clause_2_s if_clause_2_d if_clause_3_s if_clause_3_d method_call_s method_call_d nested_loop_s nested_loop_d print_stmt_1_s print_stmt_1_d print_stmt_2_s print_stmt_2_d print_stmt_3_s print_stmt_3_d print_stmt_4_s print_stmt_4_d raise_error_s raise_error_d strftime_call_s strftime_call_d string_split_s string_split_d

20

constructor_call_d

I.1.2

Template

(a) Absolute-score rank UMAP.

(b) Pairwise Jaccard index.

Figure 40: Template-induced circuit difference for csp_sweep1_4x_3.7Mnonzero_afrac1.000 on single-double-quote. csp_sweep1_4x_3.7Mnonzero_afrac0.500 template_label

Pairwise Jaccard index

nested_loop_s

0.4

print_stmt_1_s print_stmt_2_d print_stmt_2_s print_stmt_3_d print_stmt_3_s print_stmt_4_d

0.3

print_stmt_4_s raise_error_d raise_error_s strftime_call_d strftime_call_s string_split_d

0.2

(a) Absolute-score rank UMAP.

string_split_s

string_split_d

strftime_call_s

raise_error_s

raise_error_d

strftime_call_d

print_stmt_4_s

print_stmt_4_d

print_stmt_3_s

print_stmt_2_s

print_stmt_3_d

print_stmt_2_d

nested_loop_s

nested_loop_d

print_stmt_1_s

print_stmt_1_d

method_call_s

if_clause_3_s

if_clause_3_d

20

Absolute-score rank UMAP 1

method_call_d

string_split_s if_clause_2_s

10

method_call_s nested_loop_d print_stmt_1_d

if_clause_1_s

0

0.5

if_clause_3_s

if_clause_2_d

10

if_clause_2_s if_clause_3_d method_call_d

if_clause_1_d

10

if_clause_2_d

function_call_s

5

if_clause_1_s

for_loop_s

0

0.6

if_clause_1_d

for_loop_d

5

for_loop_s function_call_s

function_call_d

10

0.7

for_loop_d function_call_d

constructor_call_s

15

number of edges in the circuit for each sample 18623 number of edges of the union of all circuits = 168764 total number of edges in the entire graph = 353837 avg. unfaithfulness = 0.016

append_call_s constructor_call_s

append_call_s

Absolute-score rank UMAP 2

20

append_call_d constructor_call_d

append_call_d

append_call_s append_call_d constructor_call_s constructor_call_d for_loop_s for_loop_d function_call_s function_call_d if_clause_1_s if_clause_1_d if_clause_2_s if_clause_2_d if_clause_3_s if_clause_3_d method_call_s method_call_d nested_loop_s nested_loop_d print_stmt_1_s print_stmt_1_d print_stmt_2_s print_stmt_2_d print_stmt_3_s print_stmt_3_d print_stmt_4_s print_stmt_4_d raise_error_s raise_error_d strftime_call_s strftime_call_d string_split_s string_split_d

constructor_call_d

25

Template

I.1.3

Template

(b) Pairwise Jaccard index.

Figure 41: Template-induced circuit difference for csp_sweep1_4x_3.7Mnonzero_afrac0.500 on single-double-quote.

34

else-elif dense1_4x template_label

Pairwise Jaccard index

0.6

random_split_elif random_split_else return_result_elif return_result_else

0.5

schedule_branch_elif schedule_branch_else stats_update_elif stats_update_else strategy_branch_elif

0.4

strategy_branch_else string_align_elif string_align_else truthy_call_elif

(a) Absolute-score rank UMAP.

truthy_call_elif

truthy_call_else

string_align_elif

string_align_else

stats_update_elif

stats_update_else

strategy_branch_elif

strategy_branch_else

Template

schedule_branch_else

return_result_elif

schedule_branch_elif

random_split_elif

return_result_else

random_split_else

none_default_elif

none_default_else

program_dir_chain_else

inline_threshold_else

program_dir_chain_elif

20

Absolute-score rank UMAP 1

equality_case_elif

truthy_call_else equality_case_else

10

none_default_else program_dir_chain_elif program_dir_chain_else

inline_threshold_elif

0

0.7

none_default_elif

cell_toggle_elif

10

equality_case_elif equality_case_else inline_threshold_elif inline_threshold_else

cell_toggle_else

15

0.8

compare_objects_else

compare_objects_elif

10

cell_toggle_elif cell_toggle_else compare_objects_elif

compare_objects_else

5

binary_search_else

binary_search_elif

0

0.9

binary_search_elif

binary_search_else

5

append_guard_else

answer_check_elif

10

number of edges in the circuit for each sample 4029 number of edges of the union of all circuits = 19695 total number of edges in the entire graph = 76559 avg. unfaithfulness = 0.001

append_guard_elif

append_guard_elif

Absolute-score rank UMAP 2

15

answer_check_elif answer_check_else

Template

answer_check_else answer_check_elif append_guard_else append_guard_elif binary_search_else binary_search_elif cell_toggle_else cell_toggle_elif compare_objects_else compare_objects_elif equality_case_else equality_case_elif inline_threshold_else inline_threshold_elif none_default_else none_default_elif program_dir_chain_else program_dir_chain_elif random_split_else random_split_elif return_result_else return_result_elif schedule_branch_else schedule_branch_elif stats_update_else stats_update_elif strategy_branch_else strategy_branch_elif string_align_else string_align_elif truthy_call_else truthy_call_elif

answer_check_else

I.2.1

append_guard_else

I.2

(b) Pairwise Jaccard index.

Figure 42: Template-induced circuit difference for dense1_4x on else-elif. csp_sweep1_4x_3.7Mnonzero_afrac1.000 template_label

Pairwise Jaccard index

none_default_elif none_default_else

0.6

program_dir_chain_elif program_dir_chain_else random_split_elif random_split_else return_result_elif

0.5

return_result_else schedule_branch_elif schedule_branch_else stats_update_elif stats_update_else

0.4

strategy_branch_elif strategy_branch_else string_align_elif string_align_else truthy_call_elif

0.3

(a) Absolute-score rank UMAP.

truthy_call_else

truthy_call_elif

string_align_elif

string_align_else

strategy_branch_else

stats_update_elif

stats_update_else

strategy_branch_elif

schedule_branch_else

schedule_branch_elif

return_result_elif

return_result_else

random_split_elif

Template

random_split_else

program_dir_chain_else

none_default_elif

none_default_else

inline_threshold_else

program_dir_chain_elif

equality_case_elif

equality_case_else

inline_threshold_elif

20

Absolute-score rank UMAP 1

cell_toggle_else

truthy_call_else compare_objects_elif

10

0.7

inline_threshold_else

compare_objects_else

0

equality_case_elif equality_case_else inline_threshold_elif

cell_toggle_elif

10

0.8

compare_objects_else

binary_search_elif

20

cell_toggle_elif cell_toggle_else compare_objects_elif

binary_search_else

10

0.9

binary_search_elif binary_search_else

answer_check_elif

0

append_guard_else

append_guard_elif

10

number of edges in the circuit for each sample 18623 number of edges of the union of all circuits = 85257 total number of edges in the entire graph = 353837 avg. unfaithfulness = 0.003

append_guard_elif

answer_check_else

Absolute-score rank UMAP 2

20

answer_check_elif answer_check_else

Template

answer_check_else answer_check_elif append_guard_else append_guard_elif binary_search_else binary_search_elif cell_toggle_else cell_toggle_elif compare_objects_else compare_objects_elif equality_case_else equality_case_elif inline_threshold_else inline_threshold_elif none_default_else none_default_elif program_dir_chain_else program_dir_chain_elif random_split_else random_split_elif return_result_else return_result_elif schedule_branch_else schedule_branch_elif stats_update_else stats_update_elif strategy_branch_else strategy_branch_elif string_align_else string_align_elif truthy_call_else truthy_call_elif

append_guard_else

I.2.2

(b) Pairwise Jaccard index.

Figure 43: Template-induced circuit difference for csp_sweep1_4x_3.7Mnonzero_afrac1.000 on else-elif.

35

csp_sweep1_4x_3.7Mnonzero_afrac0.500 template_label

Pairwise Jaccard index

0.4

schedule_branch_elif schedule_branch_else stats_update_elif stats_update_else strategy_branch_elif strategy_branch_else

0.3

string_align_elif string_align_else truthy_call_elif

(a) Absolute-score rank UMAP.

truthy_call_elif

truthy_call_else

string_align_elif

string_align_else

stats_update_elif

stats_update_else

strategy_branch_elif

strategy_branch_else

schedule_branch_else

return_result_elif

Template

schedule_branch_elif

random_split_elif

return_result_else

random_split_else

none_default_elif

none_default_else

program_dir_chain_else

inline_threshold_else

20

Absolute-score rank UMAP 1

program_dir_chain_elif

truthy_call_else equality_case_elif

10

return_result_elif return_result_else

equality_case_else

0

random_split_elif random_split_else

inline_threshold_elif

10

0.5

cell_toggle_elif

15

none_default_elif none_default_else program_dir_chain_elif program_dir_chain_else

cell_toggle_else

10

inline_threshold_elif inline_threshold_else

compare_objects_elif

5

0.6

equality_case_elif equality_case_else

compare_objects_else

0

cell_toggle_elif cell_toggle_else compare_objects_elif compare_objects_else

binary_search_elif

5

binary_search_else

binary_search_else

10

0.7

binary_search_elif

answer_check_elif

Absolute-score rank UMAP 2

15

number of edges in the circuit for each sample 18623 number of edges of the union of all circuits = 126246 total number of edges in the entire graph = 353837 avg. unfaithfulness = 0.001

append_guard_elif

append_guard_elif

20

answer_check_elif answer_check_else append_guard_else

answer_check_else

answer_check_else answer_check_elif append_guard_else append_guard_elif binary_search_else binary_search_elif cell_toggle_else cell_toggle_elif compare_objects_else compare_objects_elif equality_case_else equality_case_elif inline_threshold_else inline_threshold_elif none_default_else none_default_elif program_dir_chain_else program_dir_chain_elif random_split_else random_split_elif return_result_else return_result_elif schedule_branch_else schedule_branch_elif stats_update_else stats_update_elif strategy_branch_else strategy_branch_elif string_align_else string_align_elif truthy_call_else truthy_call_elif

append_guard_else

25

Template

I.2.3

(b) Pairwise Jaccard index.

Figure 44: Template-induced circuit difference for csp_sweep1_4x_3.7Mnonzero_afrac0.500 on else-elif.

J

Cross-Template Circuit Overlap Summary

Tables 7 and 8 summarize the cross-template sample-level circuit overlap for the dense and circuitsparsity models. For each model and task, we use circuits containing roughly 5.2% of all graph edges. The PJI average excludes not only the elementwise diagonal, but also pairs of samples from the same normalized template; for example, A_s and A_d are treated as the same single-double-quote template, and A_else and A_elif are treated as the same else-elif template. Note that for models with afrac less than 1, Gao et al. [13] use topk(·) to enforce activation sparsity, which introduces discontinuities into the network. In this setting, conductance can only capture the contribution of each edge along the continuous portions of the computation. Any effect an edge has by causing the output to jump across a discontinuity is not reflected in the conductance score. In principle, this could introduce inaccuracies. However, as demonstrated in the tables below, all circuits achieve low unfaithfulness, providing empirical evidence that conductance remains effective. Type

Model

Cross-template avg. PJI

Unfaithfulness

sparse

csp_sweep1_4x_0.9Mnonzero_afrac1.000 csp_sweep1_4x_1.9Mnonzero_afrac1.000 csp_sweep1_4x_3.7Mnonzero_afrac1.000 csp_sweep1_4x_7.4Mnonzero_afrac1.000 csp_sweep1_4x_14.8Mnonzero_afrac1.000 csp_sweep1_4x_3.7Mnonzero_afrac0.062 csp_sweep1_4x_3.7Mnonzero_afrac0.125 csp_sweep1_4x_3.7Mnonzero_afrac0.250 csp_sweep1_4x_3.7Mnonzero_afrac0.500 csp_bridges2

0.3424 0.3195 0.3116 0.3074 0.3059 0.2129 0.2436 0.2547 0.2565 0.2642

0.0022 0.0025 0.0025 0.0024 0.0034 0.0270 0.0151 0.0139 0.0157 0.0184

dense

dense1_1x dense1_4x

0.3752 0.3686

0.0090 0.0026

Table 7: Cross-template pairwise Jaccard index and unfaithfulness for single-double-quote. Values use circuits containing roughly 5.2% of all graph edges; sample pairs from the same normalized template are excluded from the PJI average.

36

Type

Model

Cross-template avg. PJI

Unfaithfulness

sparse

csp_sweep1_4x_0.9Mnonzero_afrac1.000 csp_sweep1_4x_1.9Mnonzero_afrac1.000 csp_sweep1_4x_3.7Mnonzero_afrac1.000 csp_sweep1_4x_7.4Mnonzero_afrac1.000 csp_sweep1_4x_14.8Mnonzero_afrac1.000 csp_sweep1_4x_3.7Mnonzero_afrac0.062 csp_sweep1_4x_3.7Mnonzero_afrac0.125 csp_sweep1_4x_3.7Mnonzero_afrac0.250 csp_sweep1_4x_3.7Mnonzero_afrac0.500 csp_bridges2

0.4496 0.4106 0.3915 0.3818 0.3634 0.3496 0.3206 0.3330 0.3292 0.3471

0.0004 0.0159 0.0034 0.0014 0.0008 0.0026 0.0101 0.0006 0.0009 0.0106

dense

dense1_1x dense1_4x

0.4161 0.3929

0.0320 0.0013

Table 8: Cross-template pairwise Jaccard index and unfaithfulness for else-elif. Values use circuits containing roughly 5.2% of all graph edges; sample pairs from the same normalized template are excluded from the PJI average.

K

Population-level (Un)faithfulness

Let N denote the set of samples and (xn0 , xn1 ) denote the corrupted-clean pair for sample n ∈ N . Simplifying the notation from Section 2, we write M n := M (x; xn0 , xn1 ), QnΓ := QΓ (xn0 , xn1 ) for Γ ∈ {G, G′ } and write the corresponding sample-level faithfulness and unfaithfulness as Φn := Φ(G′ , G; xn0 , xn1 ) = QnG′ /QnG and U n := U (G′ , G; xn0 , xn1 ) = |1 − Φn |. Following the two notions of population faithfulness discussed by [14], we note that there can be two types of population unfaithfulness: Outer expectation:

UON := |N1 |

P

n∈N U

n

. P

Qn

′ Pn∈N G Inner expectation: The population faithfulness used by [5, 7] is defined as ΦN n . I := n∈N QG Then the inner expectation form of unfaithfulness naturally follows P n n n∈N (QG − QG′ ) N N P := UI |1 − ΦI | = . n n∈N QG

At first glance, outer expectation appears to be the more reasonable option. Its interpretation as an “average level of unfaithfulness” is straightforward. Nevertheless, we advocate the inner expectation form as the more principled measure, for the following two reasons. First, the current circuit discovery algorithms score the edges by averaging (or, equivalently, summing) the scores across all the samples [5, 11, 12]. For gradient-based methods, by the linearity P of the gradient, this approach is equivalent to substituting the metric M in Equation (2) with n∈N M n . This implies P that the network behavior accounted for is the summation of behavior over all the samples n∈N Qn when the model is confronted with all corrupted-clean pairs in N , analogous to the sample-wise scoring of Equation (2) accounting for the behavior of QG . From this perspective, N ΦN I and UI are more apt metrics to measure the quality of circuit discovery, since they directly account P for thensummationnin their form. Admittedly, one can instead substitute M with the weighted sum n∈N M , where QG is treated as a constant, in Equation (2), so that the behavior accounted Qn G for by the scoring is more aligned with the outer expectation form. Nevertheless, we also have the following reason to further advocate the inner expectation form. The outer expectation treats the unfaithfulness of all samples equally. However, as we see in Section 5.2, the high unfaithfulness of samples with small |QnG | is an artifact of the metric itself and does not indicate major circuit defects. Hence, we should be more forgiving of the unfaithfulness of samples with small |QnG |, which the inner expectation form captures. The inaccuracy of each n caused by G′ is factored into this form through (QnG − QnG′ ): If two samples contribute the same 37

(QnG − QnG′ ) to the inner expectation, then the sample with smaller |QnG | would have much worse U n , which reflects a higher level of tolerance for it. In contrast, the outer expectation form might yield results that are drowned out by the extreme unfaithfulness values of samples with small |QnG |.

L

The Defect of Unfaithfulness for a Single Sample

Unfaithfulness is a metric intended to capture how differently the chosen circuit behaves from the whole model. Nonetheless, the difference could lie in multiple aspects. [38] proposed three levels of analysis: computational, algorithmic, and implementational. The implementational level concerns the hardware-specific mechanisms to realize AI models and is largely irrelevant to interpretability. The algorithmic level characterizes the program that the model uses to handle tasks [7, 12]. While it is most appealing to characterize unfaithfulness at this level, there is currently no unified framework for doing so. The closest proxy for this is completeness [7], but it is computationally intractable. Thus, we are left with characterizing unfaithfulness at the computational level—that is, how much the inputoutput function implemented by the circuit over the relevant region of the input space (which defines a given task) deviates from the corresponding function implemented by the full model. Achieving low computational unfaithfulness is a necessary condition for attaining low algorithmic unfaithfulness [38]. In this light, the form of unfaithfulness we investigate (computational unfaithfulness) should be viewed as only a crude surrogate for the genuine notion of unfaithfulness (algorithmic unfaithfulness). This provides us with a rationale to refine our notion of unfaithfulness. Within computational unfaithfulness, there is also the difference between single-sample computational unfaithfulness and functional computational unfaithfulness. In terms of a single sample, the agreement between the full model’s output for it and that of the selected circuit provides very limited information regarding how similar the underlying algorithms implemented by the full model and the circuit actually are. The circuit could instantiate a vastly different function from the full model, only coinciding in value at the point of the sample itself (see Figure 45). As we include more edges into the circuit, the circuit function evaluated at the sample could deviate from the full model, but conceptually, the underlying algorithm is more developed with more edges and thus the function as a whole should be closer to the full model function. Hence, it is reasonable to adopt the pessimistic unfaithfulness as a more canonical notion of single sample unfaithfulness to enforce monotonicity, which reflects the underlying improvement of algorithmic faithfulness as more edges are added. On the other hand, population unfaithfulness involves many points in the input space, so it is more informative as a measure of unfaithfulness at the function level and already better satisfies the desideratum of monotonicity, which can be observed in Appendix F. This explains the efficacy of (un)faithfulness in numerous previous works [5, 9, 11].

38

Figure 45: A schematic of why single sample faithfulness is highly non-monotonic. Suppose we apply circuit discovery to recover the true function (which is the prerequisite for achieving algorithmic faithfulness), while the sample of interest is at x0 . When we include 10 edges, the circuit function might be quite far away from the true function, but they coincide at x0 . With more edges included, the circuit function is closer to the true function as a whole, but it may deviate further from the true function at x0 .

M

Extended Correlation Diagnostics Across Templates, Models, and Circuit Sizes

We extend the six rank-correlation diagnostics shown in Figures 6 and 7 from one GPT-2 small SVA template and one circuit size to all templates of the SVA/IOI/greater-than tasks, GPT-2 small and Pythia-160M, and the full greedy edge-count sweep. Solid curve segments indicate statistically significant Spearman rank correlations at p < 0.05; dotted segments indicate non-significant correlations. We present the diagnostics in the same order as the corresponding panels in the main text. When the curves are mostly above zero, it indicates a positive correlation; otherwise, a negative correlation. There are many curve segments that are not statistically significant. This is because certain templates reach low unfaithfulness easily. Once all the samples in one template are highly faithful, the trends that involve unfaithfulness will disappear. For example, if we include all edges in the graph, then all unfaithfulness will be zero, and the heavy-tailedness of sn will have no effect on it. However, this does not nullify our explanation, which aims mainly at explaining cases where extremely poor unfaithfulness scores exist. Note that U ′ /U graphs are, in general, noisier and less statistically significant than Ū ′ /Ū . However, the general trend does not change.

39

M.1

Individual Scoring

M.1.1

Figure 7(b): Ū /U vs. |QG | Template-wise rank correlation with |Q| across greedy CEAP edge counts (individual scoring)

IOI 1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0

−1.0 500

1000

1500

2000

GPT-2 small

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

0

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0

−1.0 500

1000

1500

2000

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

2000

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

1000

2000

3000

4000

5000

6000

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

800

1000

−1.0 0

Number of target edges plural_0 plural_1 plural_2 plural_3

200

−1.0 0

1.0

−1.0

0

−1.0

2000

1.0

0

Pythia-160M

−1.0 0

1.0

−1.0

Spearman ρ(U, |QG|)

greater-than

1.0

0

Pythia-160M

Spearman ρ(U,̄ |QG|)

GPT-2 small

SVA 1.0

ABBA_00 ABBA_01

ABBA_02 ABBA_03

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

5

6

7

8

9

Figure 46: Extension of the Ū /U vs. |QG | diagnostic. All the curves that are statistically significant are below zero, except in a few cases where the circuit is too small for the unfaithfulness to be informative yet. Moreover, if we constrain the circuit to be of a small size, the circuit discovery algorithm needs to accommodate more for the connectivity of the subgraph, rather than picking the edges with higher score absolute values, which deviates from the mental image of Figure 8.

40

M.1.2

Figure 7(c): Ū ′ /U ′ vs. |QG |

Template-wise rank correlation with |Q| across greedy CEAP edge counts; default task subset uses lower 70\% of each template's |Q| range (individual scoring)

IOI 1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0

−1.0 500

1000

1500

2000

GPT-2 small

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

0

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0

−1.0 500

1000

1500

2000

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

2000

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

1000

2000

3000

4000

5000

6000

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

800

1000

−1.0 0

Number of target edges plural_0 plural_1 plural_2 plural_3

200

−1.0 0

1.0

−1.0

0

−1.0

2000

1.0

0

Pythia-160M

−1.0 0

1.0

−1.0

Spearman ρ(U 0, |QG|)

greater-than

1.0

0

Pythia-160M

Spearman ρ(Ū 0, |QG|)

GPT-2 small

SVA 1.0

ABBA_00 ABBA_01

ABBA_02 ABBA_03

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

5

6

7

8

9

Figure 47: Extension of the unnormalized Ū ′ /U ′ vs. |QG | diagnostic. It generally shows positive correlation, with a few exceptions, such as plural_3. There, we have only 11 samples, and the correlation readings may be unreliable. Greater-than may show negative correlations here, because we see that Ū ′ /U ′ normally first goes up with |QG |, then goes down. To account for this, for each template we remove all points beyond the first 70% of that template’s |QG | range. Note that this does not contradict our main point that points with small |QG | in general do not have large Ū ′ /U ′ .

41

M.1.3

Figure 9(a): µ vs. |QG | Template-wise rank correlation between first momentum and |Q| (individual scoring)

SVA

IOI

greater-than

1.00

GPT-2 small Spearman ρ(μ, |QG|)

0.75 0.50 0.25 0.00 −0.25 −0.50 −0.75

8

9 9

7

8

4 4

6

3 3

7

2 2

5

1 1

6

0 0

6

5

A_0

4

A_0

BAB

3

A_0

BAB

2

A_0

BAB

1

A_0

BAB

BAB

0

6

A_0 BAB

5

A_0 BAB

4

A_0 ABB

3

A_0 ABB

2

A_0 ABB

1

A_0 ABB

A_0

A_0

ABB

A_0 ABB ABB

ABB

plura l_ plura 0 l_ plura 1 l_ plura 2 l_ plura 3 l_ plura 4 l_ plura 5 l_ plura 6 l_ plura 7 l_ plura 8 plura l_9 l_ plura 10 l_ plura 11 l_ plura 12 l_ plura 13 l_ sing 14 le sing _0 le sing _1 le sing _2 le sing _3 le sing _4 le sing _5 le sing _6 le sing _7 le sing _8 le sing _9 le sing _11 le_1 2 sing le_1 3 sing le_1 4 plura l_ plura 0 l_ plura 1 l_ plura 2 l_ plura 3 l_ plura 4 l_ plura 5 l_ plura 6 l_ plura 7 l_ plura 8 plura l_9 l_ plura 10 l_ plura 11 l_ plura 12 l_ plura 13 l_ sing 14 le sing _0 le sing _1 le sing _2 le sing _3 le sing _4 le sing _5 le sing _6 le sing _7 le sing _8 le sing _9 le sing _11 le_1 2 sing le_1 3 sing le_1 4

0

−1.00

1.00

Pythia-160M Spearman ρ(μ, |QG|)

0.75 0.50 0.25 0.00 −0.25 −0.50 −0.75

5

6 A_0

5 A_0

BAB

4 A_0

BAB

3 A_0

BAB

2 A_0

BAB

BAB

1 BAB

A_0

0

6

A_0 BAB

5

A_0 ABB

4

A_0 ABB

3

A_0 ABB

2

A_0 ABB

ABB

A_0

1 A_0

A_0

ABB

0

−1.00

Figure 48: Extension of the µ vs. |QG | diagnostic. There is a clearly negative correlation in general. A darker color suggests the Spearman ρ is statistically significant, and a lighter color suggests otherwise. The x’s here indicate that there are too few samples for those templates in the original dataset and we did not get any data for them when sampling the dataset for our probing here. M.1.4

Figure 9(b): R vs. µ Template-wise rank correlation: included mass vs first momentum (individual scoring)

SVA

IOI

greater-than

GPT-2 small

1.0 0.5 0.0 −0.5 −1.0

Pythia-160M

Spearman ρ(R, μ)

0

250

500

750 1000 1250 1500 1750 2000

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

2000

3000

4000

5000

6000

0

200

400

600

800

1000

800

1000

1.0 0.5 0.0 −0.5 −1.0 0

250

0

1000

ABBA_00 ABBA_01

ABBA_02 ABBA_03

Number of target edges plural_0 plural_1 plural_2 plural_3

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

5

6

7

8

9

Figure 49: Extension of the R vs. µ diagnostic. There is a clearly negative correlation in general.

42

Figure 9(c): Ū /U vs. R

M.1.5

Template-wise rank correlation: unfaithfulness vs included mass (individual scoring)

SVA

IOI

greater-than

GPT-2 small

1.0

0.5

0.0

−1.0 0

250

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

0

250

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

0

250

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

500

750 1000 1250 1500 1750 2000

2000

3000

4000

5000

6000

0

200

800

1000

1.0

Pythia-160M

Spearman ρ(U,̄ R)

−0.5

0.5

0.0

−0.5

−1.0

GPT-2 small

1.0

0.5

0.0

−1.0 1.0

Pythia-160M

Spearman ρ(U, R)

−0.5

0.5

0.0

−0.5

−1.0 0

250

0

1000

ABBA_00 ABBA_01

ABBA_02 ABBA_03

Number of target edges plural_0 plural_1 plural_2 plural_3

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

5

6

7

8

9

Figure 50: Extension of the Ū /U vs. R diagnostic. There is a clearly negative correlation in general.

43

Figure 9(d): Ū /U vs. µ

M.1.6

Template-wise rank correlation: unfaithfulness vs first momentum (individual scoring)

SVA

IOI

greater-than

GPT-2 small

1.0

0.5

0.0

−1.0 0

250

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

0

250

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

0

250

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

500

750 1000 1250 1500 1750 2000

2000

3000

4000

5000

6000

0

200

800

1000

1.0

Pythia-160M

Spearman ρ(U,̄ μ)

−0.5

0.5

0.0

−0.5

−1.0

GPT-2 small

1.0

0.5

0.0

−1.0 1.0

Pythia-160M

Spearman ρ(U, μ)

−0.5

0.5

0.0

−0.5

−1.0 0

250

0

1000

ABBA_00 ABBA_01

ABBA_02 ABBA_03

Number of target edges plural_0 plural_1 plural_2 plural_3

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

5

6

7

8

9

Figure 51: Extension of the Ū /U vs. µ diagnostic. There is a clearly positive correlation in general. M.2

Template-averaged Scoring

All figures follow the trend in the last section, with a few exceptions for greater-than. We believe this is due to the diffuse nature of the greater-than circuits. As shown in Appendix G, the circuits underlying templates 1 through 9 form a smooth continuum rather than cleanly separated clusters, meaning that circuits within these templates are not particularly similar. Consequently, the results from the previous section provide only limited predictive power for this setup, because we require the greedy circuit discovery to approximately select the edges with the top absolute value scores for the mental image of Figure 8 to work. If the circuit is chosen via averaging across circuits that are not similar, then this assumption does not hold.

44

M.2.1

Figure 5(a): U distribution 59.8

edges ≈ 826 101

1.19

100

U

0.665 Q3 0.35 median 0.196

10−1

Q1 0.0709

10−2

Figure 52: Template-averaged scoring version of the U distribution diagnostic for GPT-2 small on SVA template plural_1. M.2.2

Figure 7(b): Ū /U vs. |QG | Template-wise rank correlation with |Q| across greedy CEAP edge counts (template-averaged scoring)

IOI 1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

2000

GPT-2 small

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

0

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

2000

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0

−1.0 500

1000

1500

2000

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

1000

2000

3000

4000

5000

6000

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

800

1000

−1.0 0

Number of target edges plural_0 plural_1 plural_2 plural_3

200

−1.0 0

1.0

0

0

−1.0

2000

1.0

−1.0

Pythia-160M

−1.0 0

1.0

−1.0

Spearman ρ(U, |QG|)

greater-than

1.0

−1.0

Pythia-160M

Spearman ρ(U,̄ |QG|)

GPT-2 small

SVA 1.0

ABBA_00 ABBA_01

ABBA_02 ABBA_03

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

5

6

Figure 53: Template-averaged scoring version of the Ū /U vs. |QG | diagnostic.

45

7

8

9

Figure 7(c): Ū ′ /U ′ vs. |QG |

M.2.3

Template-wise rank correlation with |Q| across greedy CEAP edge counts; default task subset uses lower 70\% of each template's |Q| range (template-averaged scoring)

Spearman ρ(Ū 0, |QG|)

GPT-2 small

SVA

IOI 1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0

−1.0

Pythia-160M

0

500

1000

1500

2000

GPT-2 small

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

0

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

2000

1000

2000

3000

4000

5000

6000

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

−0.5

−0.5

−0.5

−1.0 0

500

1000

1500

2000

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

800

1000

−1.0 0

Number of target edges plural_0 plural_1 plural_2 plural_3

200

−1.0 0

1.0

−1.0

0

−1.0

2000

1.0

−1.0

Pythia-160M

−1.0 0

1.0

−1.0

Spearman ρ(U 0, |QG|)

greater-than

1.0

1000

2000

3000

4000

5000

6000

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

ABBA_00 ABBA_01

ABBA_02 ABBA_03

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

5

6

7

8

9

Figure 54: Template-averaged scoring version of the unnormalized Ū ′ /U ′ vs. |QG | diagnostic. M.2.4

Figure 9(a): µ vs. |QG |

This plot is exactly the same as Figure 48, as it reflects a property of each sample and does not depend on whether we average over all samples. M.2.5

Figure 9(b): R vs. µ Template-wise rank correlation: included mass vs first momentum (template-averaged scoring)

SVA

IOI

greater-than

GPT-2 small

1.0 0.5 0.0 −0.5 −1.0

Pythia-160M

Spearman ρ(R, μ)

0

250

500

750 1000 1250 1500 1750 2000

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

2000

3000

4000

5000

6000

0

200

400

600

800

1000

800

1000

1.0 0.5 0.0 −0.5 −1.0 0

250

0

1000

ABBA_00 ABBA_01

ABBA_02 ABBA_03

Number of target edges plural_0 plural_1 plural_2 plural_3

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

Figure 55: Template-averaged scoring version of the R vs. µ diagnostic. 46

5

6

7

8

9

Figure 9(c): Ū /U vs. R

M.2.6

Template-wise rank correlation: unfaithfulness vs included mass (template-averaged scoring)

SVA

IOI

greater-than

GPT-2 small

1.0

0.5

0.0

−1.0 0

250

500

750 1000 1250 1500 1750 2000

0

250

500

750 1000 1250 1500 1750 2000

0

250

500

750 1000 1250 1500 1750 2000

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

2000

3000

4000

5000

6000

0

200

800

1000

1.0

Pythia-160M

Spearman ρ(U,̄ R)

−0.5

0.5

0.0

−0.5

−1.0 0

GPT-2 small

1.0

0.5

0.0

−1.0 0

1.0

Pythia-160M

Spearman ρ(U, R)

−0.5

0.5

0.0

−0.5

−1.0 0

250

0

1000

ABBA_00 ABBA_01

ABBA_02 ABBA_03

Number of target edges plural_0 plural_1 plural_2 plural_3

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

5

Figure 56: Template-averaged scoring version of the Ū /U vs. R diagnostic.

47

6

7

8

9

Figure 9(d): Ū /U vs. µ

M.2.7

Template-wise rank correlation: unfaithfulness vs first momentum (template-averaged scoring)

SVA

IOI

greater-than

GPT-2 small

1.0

0.5

0.0

−1.0 0

250

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

0

250

500

750 1000 1250 1500 1750 2000

0

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

0

250

500

750 1000 1250 1500 1750 2000

1000

2000

3000

4000

5000

6000

0

200

400

600

800

1000

500

750 1000 1250 1500 1750 2000

2000

3000

4000

5000

6000

0

200

800

1000

1.0

Pythia-160M

Spearman ρ(U,̄ μ)

−0.5

0.5

0.0

−0.5

−1.0

GPT-2 small

1.0

0.5

0.0

−1.0 0

1.0

Pythia-160M

Spearman ρ(U, μ)

−0.5

0.5

0.0

−0.5

−1.0 0

250

0

1000

ABBA_00 ABBA_01

ABBA_02 ABBA_03

Number of target edges plural_0 plural_1 plural_2 plural_3

plural_4 plural_5 plural_6 plural_7

plural_8 plural_9 plural_10 plural_11

plural_12 plural_13 plural_14 single_0

single_1 single_2 single_3 single_4

single_5 single_6 single_7

Number of target edges single_8 single_9 single_11

single_12 single_13 single_14

ABBA_04 ABBA_05

ABBA_06 BABA_00

BABA_01 BABA_02

400

600

Number of target edges BABA_03 BABA_04

BABA_05 BABA_06

0

1

2

3

4

5

6

Figure 57: Template-averaged scoring version of the Ū /U vs. µ diagnostic.

N

Substantiate the Mental Image of Figure 8 with Two Real Samples

48

7

8

9

sample 78: |QG|=0.173

s n (log scale)

10−1

sample 79: |QG|=3.9

10−3 10−5 10−7 10−9

u n (log scale)

10−3 10−5 10−7 10−9 10−11 100

101

102

103

104

Ranked edge index (log scale) Figure 58: Redraw Figure 8 with two real samples of the same template. Sample 78 is the one with the poorest U reading in Figure 6a, and sample 79 is a sample of the same template but much higher |QG |. The curves are shown on logarithmic axes to make both the high-score region and long tail visible.

49