arXiv CSopen access

Weight Patching: Toward Source-Level Mechanistic Localization in LLMs

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Weight Patching: Toward Source-Level Mechanistic Localization in LLMs Abstract—Mechanistic interpretability seeks to localize model behavior to the internal components that causally realize it. Prior work has advanced activation-space localization and causal tracing, but modules that appear important in activation space may merely aggregate or amplify upstream signals rather than encode the target capability in their own parameters. To address this gap, we propose Weight Patching, a parameter-space intervention method for source-oriented analysis in paired same-architecture models that differ in how strongly they express a target capability under the inputs of interest. Given a base model and a behavior-specialized counterpart, Weight Patching replaces selected module weights from the specialized model into the base model under a fixed input. We instantiate the method on instruction following and introduce a framework centered on a vector-anchor behavioral interface that provides a shared internal criterion for whether a task-relevant control state has been formed or recovered in open-ended generation. Under this framework, the analysis reveals a hierarchy from shallow candidate source-side carriers to aggregation and routing modules, and further to downstream execution circuits. The recovered component scores can also guide mechanism-aware model merging, improving selective fusion across the evaluated expert combinations and providing additional external validation. Index Terms—Mechanistic interpretability, large language models, instruction following, causal intervention, weight patching, model merging.

✦

Single Model

...

Blocking Patching

Base Neg

Pos

Parameters

Residual Stream

Our Proposed Weight Patching

Instruct

Single Input

Contrastive Model Pair

Replacing

Pos

Base +

• •

All authors are with the Key Laboratory of Ministry of Education for Brain Inspired Intelligent Perception and Cognition, University of Science and Technology of China, Hefei, Anhui 230026, China. Corresponding author: Xinmei Tian (e-mail: [email protected]). This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

Activation

Contrastive Input Pair

•

Standard Activation Patching

behavior to the internal components that causally realize it, rather than only describing correlations in activations or outputs [1], [2]. Previous work in large language models (LLMs) has made substantial progress in activationspace localization and causal tracing, including path-level analysis and circuit discovery [3]–[6]. Yet if the goal is to determine where a target capability is actually written in parameters, activation-space evidence alone is not enough. A module may appear critical because it aggregates or amplifies decisive upstream signals, even when the capability is not primarily implemented in that module’s own parameters [7]–[9]. This distinction between activation-side importance and parameter-side implementation matters not only for mechanistic explanation, but also for downstream parameter-space analyses and operations that depend on identifying capability-relevant components [10]–[16]. Framed this way, the problem becomes source-oriented: it shifts attention from inference-time signal flow to the parameter differences introduced by training or post-training. In same-architecture models, differences in how strongly a target capability is expressed are realized through such parameter differences, making paired checkpoints with different levels of capability expression a natural object of analysis. Here, source-level localization refers to identifying candidate source-side carriers under the paired-model set-

...

ECHANISTIC interpretability aims to localize model

...

I NTRODUCTION

...

arXiv:2604.13694v1 [cs.AI] 15 Apr 2026

Chenghao Sun, Chengsheng Zhang, Guanzheng Qin, Rui Dai, and Xinmei Tian

Figure 1. Weight Patching versus standard Activation Patching. Standard activation patching intervenes on activations under clean/corrupted inputs, whereas Weight Patching intervenes on parameters under a fixed input. In our instruction-following analyses, the activation-side companion intervention uses a fixed-input cross-model form with the same restoration logic.

ting and head-/neuron-level intervention granularity—that is, components whose specialized parameter slices help recover a target internal control state when transplanted into the base model. We therefore propose Weight Patching, a parameter-space intervention method for source-level mechanistic localization in LLMs. Given a base model and a behavior-specialized counterpart, it replaces selected module weights under a fixed input and measures whether the transplanted parameters recover a capability-relevant inter-

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Upstream

Downstream

nal state. Because exhaustive exact intervention is expensive Write a rap about a new smartphone . at fine granularity, we further a gradient-based <task: Englishdevelop Capital> Write it in all capital letters . a rap about a new fine-grained smartphone." first-order "Write approximation that makes screening practical at model scale. LLM on instruction followInstruct LLM We instantiate WeightInstruct Patching Intent Forming Intent Forming Intent Aggregation Intent Aggregation ing because it provides a natural setting for studying MLP Neurons MLP Neurons Attention Heads Attention Heads how post-training writes control into model parameters. In this setting, post-training ... substantially affects how ... ... natural... language instructions are converted into internal control Parallel Connection Parallel Connection Transmission Transmission and used to guide downstream generation, making it a useful testbed for asking which specialized parameters most Instruction Following Direction Instruction Following Direction directly support the resulting capability difference. Recent work has shown that alignment induces structured control Instruction Execution Instruction Execution directions, steerable behavioral signals, and systematic routInstruction-specific Instruction-specific ing changes [17]–[22], but these studies mainly characterize ... ... Unembed Unembed representations or inference-time activations rather than the parameter subsets that carry the resulting specialization. A Response Generation Response Generation further challenge is that the relevant control must be established and maintained across generations, while standard [Instruction [Instruction Following] token-level utilities often do notFollowing] provide a stable criterion "I’M A NEW SMARTPHONE AND I’M THE BEST..." "I’M A NEW SMARTPHONE AND I’M THE BEST..." for determining whether that state has been formed and sustained. To make this problem operational for generative behav- Figure 2. Mechanistic summary of instruction following under the proiors, we introduce a framework centered on a vector-anchor posed framework. The recovered hierarchy suggests that shallow combehavioral interface that provides a shared internal criterion ponents act as candidate source-side carriers of instruction-conditioned control features, mid-layer attention heads aggregate and route them Write it in allor capital letters. for whether a task-relevant control state has been formed into an instruction-following direction, and downstream instructionrecovered. Under task-aligned instruction-following inputs, specific modules help unfold the recovered control state into response this criterion lets us evaluate paired-model interventions generation. without requiring the paired models to differ only in the studied capability. Within this framework, Weight Patching tests candidate source-side carriers in parameter space, 2 P RELIMINARIES while activation-side analyses identify where the recovered 2.1 Transformer Components in a Residual-Stream Costate is aggregated, routed, and translated into output be- ordinate System havior. Together, these analyses reveal a hierarchy from shal- We study decoder-only transformers through a residuallow source-side carriers to mid-layer aggregation/routing stream view, in which modules write additively into a modules and downstream execution circuits, as summa- shared residual-stream space. This shared residual stream rized in Fig. 2. provides a common coordinate system for both activationA further benefit of this analysis is that the recov- space and parameter-space analysis; throughout the paper, ered component scores can be reused for mechanism-aware attention heads and individual MLP neurons serve as the model merging, providing structured priors for component- basic component-level intervention units. Let the input wise fusion rather than uniform averaging. Empirically, sequence be x = (x1 , . . . , xT ), with input embeddings this reuse yields the strongest overall average performance Z0 = Embed(x) ∈ RT ×dmodel . Under a pre-norm decoder, across the evaluated expert combinations. layer l ∈ {1, . . . , L} updates the residual stream as In summary, our contributions are four-fold as follows: Z′l = Zl−1 + Attnl Norm(Zl−1 ) , (1) • We propose Weight Patching, a parameter-space inter Zl = Z′l + MLPl Norm(Z′l ) , (2) vention method for source-level mechanistic localization in LLMs, together with a gradient-based approxiwhere Zl ∈ RT ×dmodel denotes the residual-state matrix after mation that makes fine-grained screening practical. (t) layer l, and zl ∈ Rdmodel denotes its row at position t. • We introduce a framework for mechanistic analysis of Within this shared residual-stream space, an attention generative behaviors, in which a vector-anchor behavhead and an MLP neuron can both be treated as decomioral interface provides a shared internal criterion for posable write-back units. For head H (l,h) , let O(l,h) denote open-ended generation. its head output; its contribution to the residual stream is • Under this framework, we recover a hierarchical mech(l,h) (l,h) (l,h) WO . For the SwiGLU MLP, neuron anistic account of instruction following, from shallow ∆Zattn = O (l,j) (l) (l,j) induces a write-back term ∆Zmlp = a(l,j) Wdown [j, :], source-side carriers to aggregation and routing mod- N (l,j) ules and downstream execution circuits. where a denotes the corresponding hidden activation. • We demonstrate that the recovered source-level scores Thus, both attention heads and MLP neurons appear as can be leveraged for mechanism-aware model merging, additive component-level updates in the same residualwhere they guide component-wise expert fusion to stream space. Let C denote the candidate component set yield superior overall performance across the evaluated consisting of attention heads and MLP neurons; these form expert combinations. the basic units for the activation-space and parameter-space

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

interventions introduced later. Detailed component interfaces and architecture-specific replaceable parameter slices are deferred to the supplementary material. 2.2 Standard Activation Patching as a Reference Template Activation patching provides a standard activation-space intervention template for testing whether a component’s clean-run activation can restore a target behavior or control signal in a corrupted run. Given a clean input xclean , a corrupted input xcorr , and a component c ∈ C , it replaces the corrupted activation of c with its clean counterpart and measures the resulting restoration effect ∆Fpatch (c) under a utility F . Activation patching is therefore useful for identifying where behavior-relevant signals become causally effective during inference, including restoration, routing, and aggregation sites. Here, however, we use the clean/corrupted formulation only as a canonical restoration template, not as the exact activation-side protocol in our main analyses. While F can often be defined directly in token space for narrowly defined prediction tasks, this is less suitable for capability-level generative behaviors such as instruction following: changing the instruction may alter task semantics, and token-level readouts are often too narrow to determine whether an instruction-conditioned control state has been formed and sustained. We therefore preserve the same restoration logic but instantiate it through the shared vector-anchor behavioral interface introduced later. In the paired-model setting studied here, the activation-side analysis takes a fixed-input cross-model form under this shared criterion; details are given in Sec. 3.5 and the supplementary material.

M ETHOD

Fig. 3 summarizes the framework. We first construct a vector-anchor behavioral interface as a shared internal criterion. We then use Weight Patching as the primary sourcelevel test and a first-order approximation for scalable screening. Finally, we combine parameter-space and activationspace evidence to organize localized components into a source–aggregation–execution hierarchy. 3.1

Paired-Model Setting

The primary setting is a paired-model setting consisting of a pretrained base model Mbase with parameters θbase and a behavior-specialized counterpart Msft with parameters θsft . This setting is natural for source-level analysis because the capability difference of interest is realized through posttraining parameter change under a shared architecture. The two models share the same architecture and differ only in parameter values, with post-training parameter change ∆θ = θsft − θbase . In the main empirical setting, the input takes the form x = [I; Cctx ], where I denotes the naturallanguage instruction and Cctx denotes the accompanying context. Throughout the method, the candidate component set C consists of attention heads and individual MLP neurons. We do not assume that Msft differs from Mbase only in the studied capability; rather, under task-aligned inputs, the analysis asks which specialized parameterized components

consistently help recover the capability-associated anchor state. Throughout, ‘source-level’ is interpreted relative to the paired-model difference and the intervention granularity used here. 3.2

Vector-Anchor Behavioral Interface

Direct text-level evaluation is often unstable for mechanistic localization in generative tasks such as instruction following, where control must persist across extended generation, and narrow token-level readouts are often insufficient to determine whether the relevant control state has been formed. To make source-level localization operational in this setting, we construct a vector-anchor behavioral interface as a shared internal criterion, using an existing task-vector extraction and steering procedure [19]. (i) (i) Let Dinst = {(xr , xcf )}N i=1 denote a paired dataset, (i)

(i)

where xr is an instructed input and xcf is its instructionremoved counterpart with matched context. For each candidate layer l ∈ L, an instruction direction is extracted by averaging residual-stream differences at a designated anchor position ta : N

v(l) =

i 1 X h (ta ) (i) (t ) (i) zl (xr | Msft ) − zl a (xcf | Msft ) . (3) N i=1

The anchor layer is then selected as la = arg max Recover v(l) , l∈L

(4)

where Recover(·) denotes the behavioral recovery score under steering. The final task vector is set to v = v(la ) . Based on this construction, the anchor utility is defined as Fa (M, x) = sim za (x | M ), v . (5) (t )

Here, za (x | M ) ≡ zlaa (x | M ) denotes the anchor representation. The specialized-to-base anchor gap is

G(x) = Fa (Msft , x) − Fa (Mbase , x).

(6)

This quantity serves as the normalization term for both exact and first-order restoration scores. The resulting interface provides the shared internal criterion under which parameter-space and activation-space evidence can be compared in the remainder of the method; full anchor construction, layer selection, and input-adaptive calibration are deferred to the supplementary material. 3.3 Weight Patching for Identifying Candidate SourceSide Carriers Weight Patching shifts the intervention target from activations to parameters. Under a fixed input, it tests whether transplanting a component’s specialized parameters into the base model recovers the target control representation at the anchor, thereby helping identify candidate sourceside carriers and separate them from downstream aggregation/routing and execution sites under the paired-model setting. For a component c ∈ C , let Θ(c) denote its replaceable parameter slice. The corresponding single-component parameter-patched model is defined as (c←sft)

Mbase

:= Replace(Mbase , Msft ; Θ(c) ),

(7)

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

...

Gate

V Group

Down

...

Gate ×

...

... Layer L

V Group

Replace

Down

...

Hidden States

Layer l+1

Base

V Group

Layer l

Layer 1

Gate ×

Layer l-1

Target Component (with Same Size)

...

Model

Pinpoint Neuron Weight

Pinpoint Head Weight Q

Instruct �풄

Separate Forward

Instruct

Weight Patching

Negative: [Query]

Backup

Down

...

Instruction Direction Extraction

Positive: [Instruction][Query]

...

Parameters Replacement Steering Vectors

푨�

−

Causal Localization of Instruction Components Tracing Upstream back

Single Input

Layer L

Complete Generation

= �up

푲

= �down

Forward Causal Effect

Causal Effect

Layer L

...

Layer l+1

Layer l-1

layer

Directtion

Logits Probability

Decoder Paramters

Attention Head

MLP Neuron

Dot Product

Vector

Token

Layer 1

...

Model

...

Score

Division

...

Behavior Evaluation

푪풐풔

...

Layer l

Positive: [Instruction][Query]

...

Model

...

Base

Layer 1

Intervention

Tracing Downstream back

Hadamard

Figure 3. Overview of the proposed framework. We first extract an instruction-direction anchor as a shared behavioral interface, then apply Weight Patching for source-level localization, and finally combine parameter-space and activation-space evidence to identify source-side carriers, aggregation/routing modules, and downstream execution circuits.

meaning that only the parameters in Θ(c) are taken from Msft , while all remaining parameters are kept from Mbase . When evaluating multi-component restoration or ablation, the same replacement operator is applied to a set of components: ! [ (S←sft) (c) := Replace Mbase , Msft ; Θ . (8) Mbase c∈S

The basic intervention unit throughout source localization, however, remains a single component. The parameter slice is defined at the same structural granularity as the mechanistic unit under study:

Θ(H Θ

(l,h)

)

(N (l,j) )

(l,h)

= {WQ

(l,h)

, WO

(l) (l) (l) = {Wgate [:, j], Wup [:, j], Wdown [j, :]}.

(9)

For attention heads, replacing only the query and output slices avoids ambiguity when key and value projections are shared across heads. For MLP neurons, the gate column, up-projection column, and down-projection row together preserve the full feature-detection, gating, and write-back interface of the neuron. Detailed architecture-specific implementations, including grouped-query attention, are deferred to the supplementary material. The source-level effect of component c is quantified by how much of the anchor gap is restored when only that component’s parameters are replaced: " # (c←sft) Fa (Mbase , x) − Fa (Mbase , x) Ew (c) = Ex∼Dinst . G(x) (10)

A large Ew (c) provides evidence that the component’s specialized parameters make a substantial contribution to recovering the target internal control representation. Such components are therefore supported as candidate sourceside parameter carriers, rather than merely downstream sites where behavior-relevant information becomes visible. 3.4 Efficient Screening via First-Order Weight Attribution Exact Weight Patching provides the most direct source-level causal test, but exhaustive evaluation over all heads and especially all MLP neurons is computationally prohibitive. We therefore introduce a first-order weight-attribution approximation not as a replacement for exact intervention, but as a scalable surrogate for full-model screening and neuron-scale localization. This design is conceptually analogous to attribution patching in activation space, which approximates exact patching by locally linearizing the restoration utility around a reference point and using a gradient–difference inner product as a fast screening score [23], [24]. Here, we apply the same approximation logic in parameter space, while retaining exact Weight Patching as the primary exact interventional test. For a component c, exact Weight Patching replaces the (c) (c) (c) base-model slice Θbase by Θsft = Θbase + ∆θ(c) , where ∆θ = θsft −θbase . Under a first-order Taylor expansion of the anchor utility around Mbase , the resulting change in anchor utility is approximated by X ∂Fa (Mbase , x) (c←sft) Fa (Mbase , x)−Fa (Mbase , x) ≈ ∆θp . ∂θp (c) p∈Θ

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Normalizing this local approximation by the same anchor gap G(x) used in exact Weight Patching yields the firstorder weight-attribution score: P  ∂Fa (Mbase ,x) p∈Θ(c) ∆θp ∂θp  . (11) Attrwt (c) = Ex∼Dinst  G(x)

Under the shared anchor utility, WP serves as the primary parameter-space sufficiency test for candidate source-side carriers, while AP provides complementary activation-space evidence for where the recovered state becomes effective during inference. Here AP is instantiated in the fixed-input cross-model form introduced in Sec. 2.2: for the same input x, we replace the activation of component c in the basemodel run with its counterpart from Msft and measure the extent to which the anchor-side state is recovered. Although this differs from the canonical clean/corrupted formulation, it preserves the same restoration logic under controlled component replacement. Relative to the anchor layer, components before the anchor are treated as potential upstream suppliers because they contribute to forming the recovered anchor-side control state, whereas components after the anchor are treated as downstream execution modules because they determine how that recovered state is translated into output behavior. For the downstream stage, we considered both vector-projection and KL-divergence readouts and found qualitatively similar patterns; in the main text, we report the KL-based results because executionstage modules are most directly characterized by changes in the next-token distribution. This yields a staged hierarchy rather than a single global ranking. Detailed activation-side intervention and hierarchy-recovery procedures, including the projection-based variant, are deferred to the supplementary material. f(c) denote the base-model For a component c ∈ C , let M base run in which the activation of c is replaced by its counterpart from Msft . Its activation-side restoration effect is " # f(c) , x) − Fa (Mbase , x) Fa (M base Ea (c) = Ex∼Dinst . (12) G(x) Large Ea (c) indicates that restoring the specialized activation of c helps recover the anchor-side control representation under the shared criterion. This supports c as

WP-Guided Multi-Expert Parameter Fusion Base �base Fine-tuning Instruct ��

Here, Θ(c) denotes the parameter slice of component c. The score measures how well the post-training parameter change on c aligns with the local sensitivity of the anchor utility, and thus serves as a first-order surrogate to the exact restoration effect rather than an independent heuristic. In practice, exact Weight Patching is used whenever exhaustive intervention remains feasible, especially for headlevel analyses and for validating top-ranked candidates. For neuron-scale localization, however, exhaustive exact patching is computationally prohibitive, so first-order weight attribution serves as the practical screening score that makes fine-grained localization operational at model scale. The fidelity of this approximation is evaluated separately in Sec. 4.7. 3.5 From Candidate Source Carriers to Aggregation/Routing and Execution Modules

Math ��

Code ��

...

Causal Localization of Task-Specific Parameters Causal Effect

Weighted Sum

Model Merging

� + � ∗ ��

�

+ � ∗�

Fusion

�

Instruct-following Math Calculation Code-writing

+ � � ∗ �� ...

...

Figure 4. Reusing WP-recovered component scores for mechanismaware model merging.

an activation-side restoration site or aggregation/routing module, but does not by itself show that the behavior difference is written in c’s own parameters. To trace candidate upstream suppliers for a chosen target component ctar , let

gneed (ctar ) = ∇Ψctar (Z) Fa (Mbase , x)

(13)

denote the need direction at the residual-stream input of ctar , where Ψctar (Z) extracts that input. For a candidate supplier component csup , define its weight-side support to ctar by (c

swt (ctar , csup ) =

X p∈Θ(csup )

∆θp

)

tar ∂Fneed (Mbase , x) , ∂θp

(14)

where (c

)

tar Fneed (M, x) = ⟨Ψctar (Z(M, x)), gneed (ctar )⟩ .

Their functional link strength is then swt (ctar , csup ) Elink (ctar , csup ) = Ex∼Dinst . G(x)

(15)

(16)

A large Elink (ctar , csup ) indicates that the post-training change on csup is well aligned with increasing the targetside need utility. In practice, target components for tracing may be nominated by activation-side evidence, parameterside evidence, or prior mechanistic hypotheses. Retrieved suppliers are retained as source candidates only when they also exhibit strong exact WP effects Ew (c), so that supplier tracing complements rather than replaces the primary parameter-side source criterion. 3.6 Reusing WP-Recovered Component Scores for Mechanism-Aware Model Merging WP-recovered component scores can be reused for component-wise model merging by weighting experts at

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

the component level rather than averaging expert deltas uniformly. Let Mbase denote a shared pretrained base model, and let {M (k) }K k=1 denote expert models obtained by finetuning the same base toward different capabilities. Their expert-specific parameter deltas are ∆θ(k) = θ(k) − θbase . Rather than averaging these deltas uniformly over the full parameter space, we propose a precise component-wise fusion over the same head- and neuron-level units used in Weight Patching. For each expert model M (k) , a component score s(k) (c) is computed for every component c ∈ C , using either the exact Weight Patching effect or its first-order approximation. Only positive evidence is retained, s̄(k) (c) = max(s(k) (c), 0), and the component-wise fusion weight of expert k on component c is defined as  PK s̄(k) (c)  (m)   PK , (c) > 0, m=1 s̄ (m) (k) s̄ (c) α (c) = (17) m=1  PK  (m) 1, s̄ (c) = 0. m=1 K The fused model Mfuse is then defined component-wise. For each component c, its fused parameter slice is (c)

(c)

θfuse = θbase +

K X

α(k) (c) ∆θ(k,c) ,

and end-to-end behavior by IFEval strict accuracy; a vectorprojection variant gives similar patterns and is deferred to the supplementary material. Unless otherwise noted, exact Weight Patching is used when exhaustive intervention is tractable, while the first-order score is used mainly to screen neuron-scale candidates before exact validation. Additional implementation details are provided in the supplementary material. For mechanism-aware model merging, we follow the Llama-2-13B expert-pool setup used in DARE, WIDEN, and AIM [15], [31], [32]. We use WizardLM-13B, WizardMath13B, and Llama-2-13B-Code-Alpaca as the instruction, math, and code experts [33]–[35], and study four combinations: Code + Instruction, Code + Math, Instruction + Math, and Code + Instruction + Math. We evaluate merged models on HumanEval [36], MBPP [37], MMLU [38], MATH [39], GSM8K [40], and IFEval [25], and compare against Avg Baseline [11], Task Arithmetic [41], TIES-Merging [13], DARE Task Arithmetic, DARE TIES [15], and WIDEN [31]. Except for WIDEN, the baselines are implemented with MergeKit [16]. Additional implementation details are provided in the supplementary material. 175

(18) Correction rate (%)

(c)

No Comma Title Multiple Sections Quotation Number Highlighted Sections English Capital

150

k=1

where ∆θ(k,c) = θ(k,c) − θbase . This yields a mechanismaware model-merging rule implemented through component-wise fusion, in which different experts dominate different components based on source-level support rather than a globally uniform parameter average. Detailed slice-level fusion and implementation are provided in the supplementary material.

125 100 75 50 25 0 0

E XPERIMENTS

4.1

Experimental Setup

We evaluate the proposed framework on instructionfollowing behaviors that require persistent control over generation. For mechanistic analysis, we use IFEval [25], whose constraints are explicit and automatically checkable. We study six representative tasks—No Comma, Title, Multiple Sections, Quotation, Number Highlighted Sections, and English Capital—using English Capital as the main running example. This choice is driven by interface reliability: using the existing task-vector extraction and steering procedure of [19], we obtain sufficiently recoverable anchor directions on these six tasks, whereas the remaining IFEval tasks yield directions that are too weak for reliable anchor-based causal analysis. Our interpretability experiments use three paired base/instruction-tuned Llama settings: Llama-3.2-3B, Llama-3.1-8B [26], and Llama-2-13B [27]. The main paper focuses on Llama-3.2-3B, with selected cross-scale results on 8B and 13B. To broaden architectural coverage beyond the Llama family, the supplementary material additionally reports selected results on paired Qwen2.5, Mistral-7B-v0.3, and Gemma 2 settings [28]–[30]. For IFEval data usage and steering-vector extraction, we follow [19]. We evaluate upstream localization by anchor similarity, downstream execution by KL divergence to the instruction-tuned model,

Layer

Figure 5. Layer-wise recovery under task-vector injection. Injecting the extracted task vector into different layers of Llama-3.2-3B-Base yields clear recovery peaks across six IFEval tasks, indicating that the anchorlevel control representation emerges primarily in middle-to-late layers.

4.2

Validating the Vector-Anchor Interface

Before comparing AP and WP under a shared internal criterion, we first verify that the extracted task vector serves as a usable behavioral interface for instruction following rather than a fragile text-level proxy. To this end, we inject each task vector into different layers of Llama-3.2-3B-Base and measure the correction rate, defined as the normalized fraction of the original base-to-instruct performance gap recovered by steering, R = (Accsteer − Accbase )/(Accinst − Accbase ). As shown in Fig. 5, recovery peaks consistently emerge in mid-to-late layers across all six IFEval tasks rather than in shallow lexical layers, and the selected anchor layers lie in the same range. A vocabulary-space projection of the best-layer vectors offers only limited interpretability: structurally explicit tasks show partially recognizable format-related cues, whereas other tasks remain diffuse and abstract. Together, these results suggest that the extracted vector is not merely a prompt-level lexical residue, but a relatively stable intermediate control representation. These checks are intended to validate the interface instantiation

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

used in this paper, rather than to claim a new task-vector extraction method. 0.10

0.05

Layer

0.00

0.05

0.10

25 0

Head

0.03 0.02 0.01 0.00 0.01 0.02 0.03

5 10

Layer

0.50.0 0.5

15 20 25 0

Neuron AVG

(a) Activation Patching

Head

0.2 0.1 0.0 0.1 0.2 0.50.0 0.5

Neuron Avg

(b) Weight Patching

Figure 6. Component-importance heatmaps under Activation Patching and Weight Patching on the English Capital task.

Activation Patching Weight Patching Shared

Count

3 2 1 0 0

Layer

(a) Key Head Distribution

30 25 20 15 10 5 0 0

Count

Activation Patching Weight Patching Shared

Layer

(b) Key Neuron Distribution

Figure 7. Layer distributions and overlaps of top-ranked heads and neurons identified by Activation Patching and Weight Patching on the English Capital task.

4.3 Weight Patching vs. Activation Patching: Source Carriers versus aggregation/Routing Modules With the vector anchor validated as a shared criterion, we compare Activation Patching (AP) and Weight Patching (WP) under the same task and restoration objective. Although both interventions are evaluated by anchor recovery, they probe different causal objects. AP is more sensitive to modules that become important once instructionconditioned information is already in motion, whereas WP more directly tests whether a module’s specialized parameters are sufficient to reintroduce the target control signal. Accordingly, limited overlap between AP- and WP-localized components should not be read as inconsistency; rather, it is expected if source-level carriers and downstream aggregation or routing modules are functionally distinct. The localization patterns differ systematically. On the English Capital task, Fig. 6 shows that AP concentrates on middle-layer heads and nearby neurons, consistent with modules involved in routing or consolidating instructionconditioned information during inference, whereas WP shifts more strongly toward shallow neurons, suggesting earlier and more neuron-centric source-level carriers. Fig. 7 shows that this difference is global rather than local: the layer distributions are misaligned, and the overlap between AP- and WP-ranked components remains limited. Thus, the two interventions do not simply provide two views of the same important modules, but instead capture different functional stages of the mechanism. These results are consistent with a source–aggregation separation in the present setting: activation patching more strongly surfaces aggregation and routing bottlenecks, whereas weight patching more directly highlights candidate source-side parameter carriers. At this stage, however, the evidence is still distributional. The next subsection, therefore, tests whether this separation corresponds to a

Table 1 Overlap between neurons localized directly by Weight Patching and neurons recovered by tracing upstream from the Top-20 AP-ranked heads. We report the maximum overlap (Max), the mean overlap over the Top-5 heads (Top-5 Avg), and the mean overlap over all 20 heads (Top-20 Avg). Task

Max Top-5 Avg Llama-3.2-3B English Capital 0.775 0.734 Multiple Sections 0.405 0.371 Number Highlighted Sections 0.485 0.442 Title 0.435 0.422 No Comma 0.445 0.363 Quotation 0.705 0.628 Avg 0.542 0.493 Llama-3.1-8B English Capital 0.760 0.654 Multiple Sections 0.620 0.519 Number Highlighted Sections 0.380 0.298 Title 0.495 0.458 No Comma 0.600 0.551 Quotation 0.575 0.492 Avg 0.572 0.495 Llama-2-13B English Capital 0.700 0.642 Multiple Sections 0.450 0.427 Number Highlighted Sections 0.400 0.390 Title 0.575 0.498 No Comma 0.530 0.507 Quotation 0.605 0.566 Avg 0.543 0.505

Top-20 Avg 0.507 0.264 0.305 0.321 0.246 0.476 0.353 0.422 0.348 0.205 0.313 0.387 0.375 0.342 0.411 0.342 0.287 0.393 0.355 0.405 0.365

genuine causal division of labor through targeted ablation, restoration, and upstream linkage analyses. 4.4

Causal Validation of the Recovered Hierarchy

The source–aggregation separation in Sec. 4.3 is still, by itself, a distributional observation. We therefore ask whether the modules highlighted by AP and WP also differ in their functional roles under intervention. Fig. 8 provides this test for upstream modules on the English Capital task. The contrast between heads and neurons is especially clear. Ablating a small number of AP-ranked upstream heads sharply degrades performance, yet restoring those heads alone yields only limited recovery. This pattern is consistent with aggregation or routing bottlenecks: these heads are important once the instruction-conditioned signal is already present, but are not by themselves sufficient to recreate it in the base model. In contrast, WP-ranked upstream neurons show both stronger degradation under ablation and substantially stronger recovery under restoration. This supports interpreting the shallow neuron subsets identified by WP as candidate source-side carriers of the control representation, rather than merely downstream sites where the signal becomes visible. A related but distinct pattern appears downstream. As shown in Fig. 8, downstream heads remain only weakly restorative, even when some of them are behaviorally important under ablation, suggesting a more conditional routing role. Downstream neurons, however, exhibit both strong necessity and substantial sufficiency: ablating them rapidly destroys task performance, whereas restoring them recovers a much larger fraction of the target behavior. This

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Figure 8. Causal validation of upstream (top row) and downstream (bottom row) modules on the English Capital task. The results support a source–convergence separation in upstream modules and a neuron-dominant pattern in downstream execution.

0.8 Individual Task

0.8

Individual Task

0.7

Completeness (%)

Intersection with Full-Unit ranking (%)

Figure 9. Fine-grained comparison of upstream and downstream neurons on the English Capital task. Left panels show ablation behavior, and right panels show output-space projections. Upstream neurons exhibit threshold-like collective effects and more abstract projections, whereas downstream neurons behave more additively and show more directly interpretable output-space features.

0.6 0.5 0.4

0.6

4.5

0.4

0.2

0.3 Gate

Down

Gate+Up

(a) Ablation of MLP for Neuron Localization

state into concrete output behavior. In other words, the downstream evidence extends the hierarchy beyond the upstream source–aggregation distinction and points to a later execution stage centered on neurons. Table 1 further connects these two views by tracing the upstream suppliers of the top AP-ranked heads. Across tasks, the neurons recovered by tracing from the top-20 AP heads show substantial overlap with the neurons directly identified by WP, under both signed and absolute-value variants. The overlap is stable not only at the best-matching head (Max), but also in the Top-5 Avg and Top-20 Avg, indicating that the traced neurons are not random upstream artifacts. Instead, they align systematically with the WPlocalized source neurons. This provides structural support for the interpretation that WP-recovered neurons are not merely another set of important modules; they are plausible upstream suppliers of the aggregation heads identified by AP. Taken together, these results are consistent with a hierarchical division of labor, with shallow neurons emerging as candidate source-side carriers, middle-layer heads as aggregation/routing interfaces, and later neurons as downstream execution units. The next subsection examines whether this hierarchy also remains visible in finer-grained component signatures.

Gate

Down

Gate+Up

(b) Ablation of MLP for Neuron Restoration

Figure 10. MLP submodule ablation for neuron localization. (a) Overlap between neurons localized from gate, up, down, or gate+up and the full-unit neuron ranking under WP. (b) Restoration completeness after globally restoring neurons selected under each setting, averaged over six IFEval tasks.

result suggests that, once an abstract control representation has been formed and routed forward, later neurons appear to play a major role in translating the recovered control

Fine-Grained Component Analysis

The hierarchy is most convincing if it remains visible not only in aggregate intervention scores, but also in the finegrained behavior of individual components. We therefore examine whether the recovered components exhibit distinct signatures consistent with the source–aggregation– execution organization, mainly through the English Capital case, with Fig. 10 providing a task-averaged MLP ablation across the six IFEval tasks. Fig. 9 shows a clear contrast between upstream and downstream neurons: upstream neurons exhibit threshold-like collective effects, where ablating a relatively small set of WP-ranked neurons can trigger an abrupt collapse even though no single neuron alone accounts for the full behavior, whereas downstream neurons behave more additively and align more closely with

Instruct (Top-K)

Base (Top-K)

Global Head Knockout

Instruct (Random)

Global Head Restore

1.0

0.8

Faithfulness

Completeness

0.6

0.4

0.0

No Comma

Title

Multiple Sections

Quotation

Number Highlighted Sections English Capital

Figure 11. Selective attention of critical heads to instruction tokens. Critical heads in the instruction-tuned model allocate substantially more attention to instruction regions than matched heads in the base model or random heads, supporting their routing role.

0.2

Top-K Global Neuron Knockout

1.0

0.8

0.6 0.4

4.6 Cross-Task Generalization and Shared-vs.-Specific Structure The analyses above establish a source–aggregation– execution hierarchy on the English Capital task. We next ask whether this organization reflects a broader pattern of instruction following rather than a property of a single case. Fig. 12 shows that, although absolute intervention effects vary across tasks, the overall division of labor remains stable: Attention heads are more strongly associated with aggregation/routing effects, whereas neurons are more strongly associated with upstream source-side storage and downstream execution. Thus, what generalizes across these tasks is not a single intervention profile, but a recurring hierarchical organization of how instruction-conditioned control is formed, routed, and enacted. At the same time, this shared organization does not imply a universal set of reused local components. Fig. 13 shows that the overlap among top-ranked heads and neurons is limited, and Fig. 14 further shows that this distinction is functional as well as structural: when the Top-

0.0

500

Top-K

1500

2000

Top-K Global Neuron Restore

0.6 0.4

0.2

output-space behavior. This supports a distinction between control formation and control execution. Fig. 10 further sharpens this picture within SwiGLU: gate-only and uponly localization each recover a substantial portion of the full-unit neuron ranking, while down-only localization is markedly weaker; restoration is strongest when gate and up are combined, with up alone outperforming gate alone. This indicates that neuron localization depends primarily on the activation-forming branches of the MLP rather than on the write-back branch alone. The head-level evidence is complementary. In Fig. 11, the critical heads identified in the instruction-tuned model allocate substantially more attention to instruction-bearing regions than matched heads in the base model or random heads in the same model, indicating a routing role rather than a storage role. Thus, the fine-grained evidence refines rather than replaces the hierarchy recovered in Sec. 4.4: Sparse neurons are more closely tied to upstream control formation and downstream execution, whereas critical heads are more closely tied to selective reading, routing, and aggregation. Having established these fine-grained signatures—primarily in the English Capital case, with complementary task-averaged evidence from Fig. 10—we next ask whether the recovered organization generalizes across instruction-following tasks.

0.0

Faithfulness

0.4

0.2

Completeness

Attention Ratio of Instruction Prompt (%)

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

0.2

500

1000

Top-K

No Comma Number Highlighted Sections Random Select (Single)

1500

2000

Title English Capital Activation Patching (Avg)

0.0

Multiple Sections Activation Patching (Single) Weight Patching (Avg)

1000

Quotation Weight Patching (Single) Random Select (Avg)

Figure 12. Cross-task ablation and restoration across six IFEval tasks. The plots compare head-level and neuron-level knockout/restoration under Activation Patching, Weight Patching, and random selection.

Figure 13. Overlap of critical modules across tasks. Intersection counts are shown for top-ranked heads and neurons across six IFEval tasks.

2000 upstream neurons localized for one task are globally restored and tested across all six IFEval tasks, the strongest recovery typically remains on the task from which they were localized, especially under WP. The remaining off-diagonal responses indicate partial sharing rather than fully isolated circuits. Together, these results suggest that instructionfollowing tasks share a common mechanistic template while instantiating it through partially task-specific local components. Fig. 15 then provides a complementary static view from parameter space. Across layers, MLP modules exhibit larger average parameter changes than attention modules, with especially pronounced drift in gating-related components. While this result is not itself a causal localization test, it is consistent with the WP-based conclusion that post-training writes instruction-following behavior more strongly into sparse neuron-level parameter subsets than into attention parameters alone. 4.7

Fidelity and Efficiency of Gradient Approximation

Because the gradient-based score introduced in Sec. 3.4 is only a first-order surrogate to exact Weight Patching (WP), we evaluate it from two perspectives: fidelity to exact WP and computational efficiency. Table 2 shows that the approximation preserves a substantial fraction of the most important components identified by exact WP at both granularities. At the head level, the overlap of top-ranked

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

0.00

100.00

0.00

17.31

10.79

0.00

6.81

Title

93.86

69.40

13.20

30.25

16.81

3.93

95.39

100.00

27.28

35.82

22.01

24.92

Multiple Sections 100.00

78.33

19.45

25.00

22.76

5.67

82.26

62.98

100.00

0.00

54.79

0.00

Quotation

0.00

40.97

100.00

61.12

59.47

50.97

0.00

63.30

49.48

100.00

36.72

58.25

Number Highlighted Sections

17.88

100.00

61.50

79.76

100.00

52.48

41.05

16.76

30.54

9.39

100.00

26.29

43.03

98.13

93.45

100.00

75.44

100.00

53.65

63.62

0.00

83.48

48.29

100.00

Relative Change

0.00

English Capital

Attention Submodules Decomposition

100

86.58

Multiple Quotation Number English Sections Highlighted Capital Sections

No Comma Title

Test Task (Eval)

Multiple Quotation Number English Sections Highlighted Capital Sections

Test Task (Eval)

Figure 14. Cross-task specificity of restored upstream neurons. Rows denote the restored task and columns the evaluation task; values are normalized restoration completeness. Left: AP-traced upstream neurons. Right: WP-localized upstream neurons. Table 2 Comparison between exact Weight Patching and its gradient-based approximation, averaged over six tasks, including the overlap of Top-20 heads and Top-1000 neurons, total analysis time, and peak VRAM on three RTX 4090 GPUs. Model

LLama-3.2-3B LLama-3.1-8B LLama-2-13B

Gradient Attribution ✘ ✔ ✘ ✔ ✘ ✔

Intersection (%) Head-level 66.00 67.50 66.00

Total Time (Second)

Peak VRAM (MiB)

384.85 4.03 1099.68 4.58 3097.15 15.54

13150 15760 34458 37747 51921 55699

LLama-3.2-3B LLama-3.1-8B LLama-2-13B

80.14

90829.54 5.89 484704.23 5.19 1766160.89 28.01

13304 16096 35317 40991 52491 67831

53.33 48.97

q_proj k_proj

v_proj o_proj

Attention Overall

MLP Submodules Decomposition

0.150 0.125 0.100 0.075 0.050

gate_proj up_proj

0.025 0.000

down_proj 5

modules remains stable across model sizes, at roughly 66– 68%, indicating that the approximation retains the dominant ranking structure needed for head screening. At the neuron level, where the table reports averages over all six tasks, the overlap is higher on the 3B model and decreases on larger models, but it still preserves a substantial portion of the exact ranking structure. Thus, while the approximation is not a substitute for exact WP, it retains sufficient ranking fidelity to support coarse-to-fine localization over both heads and neurons. Its practical value is evident at both levels, but it is especially important for neuron-scale analysis. At the head level, exact WP requires hundreds to thousands of seconds, whereas the approximation reduces screening to only a few seconds. At the neuron level, the gain is decisive: exact WP requires on the order of 105 to 106 seconds across the three model sizes, while the approximation remains in the range of only a few to a few tens of seconds. In this sense, the approximation makes head-level screening efficient and neuron-level screening feasible. Notably, this gain comes primarily from reduced wall-clock time rather than reduced memory footprint, since peak VRAM remains comparable and is sometimes slightly higher under the gradient-based implementation. Overall, the approximation should be viewed as a practical screening mechanism that

MLP Overall 15

Layer

Figure 15. Layer-wise parameter changes induced by instruction tuning. Top: relative changes in attention submodules (q, k, v, o, and overall). Bottom: relative changes in MLP submodules (gate, up, down, and overall).

Llama-3.2-3B

Neuron-level ✘ ✔ ✘ ✔ ✘ ✔

0.175 0.150 0.125 0.100 0.075 0.050 0.025 0.000 0.175

0 No Comma Title

Relative Change

Target Task (Restore)

Upstream AP for Task Vector Restoration (%) Upstream WP for Task Vector Restoration (%) No Comma

Llama-3.1-8B

Title

0.95

Title

No Comma

0.95

No Comma

0.91 0.83 0.76 0.73 0.66 0.57 0.57 Multiple Sections Multiple Sections 0.55 0.50 0.70 0.87 0.38 0.38 0.56 0.70 0.36 0.33 0.42 0.52 0.19 0.18 0.19 0.17 0.28 0.35 0.14 0.17 0.15 0.31 0.46 0.62 0.77AVG 0.15 0.30 0.45 0.60 0.75AVG 0.14 0.10 0.28 0.19 0.18 0.13 0.19 0.13 0.42 0.29 0.26 0.25 0.57 0.38 0.36 0.38 0.71 0.48 0.39 0.38 Quotation Quotation 0.54 0.57 0.52 0.50 0.71 0.75 0.65 0.63 0.76

English Capital 0.89 Number Highlighted Sections

Base Model (Vanilla) Base Model (AP Restore)

English Capital 0.94 Number Highlighted Sections

Base Model (WP Restore) Instruct Model (Vanilla)

Figure 16. Instruction-following recovery via injection of localized critical components. Injecting the identified head and neuron subsets into base models substantially restores performance across model sizes, supporting their reuse as transferable capability carriers.

makes fine-grained Weight Patching operational at model scale, while exact intervention remains the source of causal validation. 4.8 Application: Mechanism-Aware Expert Composition If the modules localized by Weight Patching indeed capture source-relevant parameter subsets, they should support not only post hoc explanation but also mechanism-aware reuse. As an initial reuse test, Fig. 16 restores only the top 5% AP- or WP-localized head and neuron parameters into the base model and compares the resulting performance with the vanilla base and instruct models. Even under this tight budget, the restored model recovers a substantial portion of instruction-following performance across the evaluated IFEval tasks, with WP-localized subsets consistently outperforming their AP-based counterparts. This suggests that the localized subsets capture compact and reusable capabilityrelevant structure, naturally motivating their reuse beyond post hoc analysis. We then ask whether this reuse potential transfers to multi-expert fusion. Table 3 shows that using localized

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Table 3 Performance comparison of the average baseline, prior merging methods, and precise fusion (ours) across four expert combinations. Model(s)

Method

Base Code Instruction Tuned Math

Code + Instruction

Code + Math

Instruction + Math

Code + Instruction + Math

Avg Baseline DARE Task Arithmetic DARE Ties Task Arithmetic Ties Merging WIDEN Ours Avg Baseline DARE Task Arithmetic DARE Ties Task Arithmetic Ties Merging WIDEN Ours Avg Baseline DARE Task Arithmetic DARE Ties Task Arithmetic Ties Merging WIDEN Ours Avg Baseline DARE Task Arithmetic DARE Ties Task Arithmetic Ties Merging WIDEN Ours

HumanEval 18.90 19.51 30.49 14.02 28.05 26.83 (−1.22) 30.49 (+2.44) 29.27 (+1.22) 16.46 (−11.59) 26.22 (−1.83) 30.49 (+2.44) 15.85 16.46 (+0.61) 17.07 (+1.22) 18.29 (+2.44) 15.85 (+0.0) 17.07 (+1.22) 17.68 (+1.83) 20.73 5.49 (−15.24) 8.54 (−12.19) 4.27 (−16.46) 28.05 (+7.32) 24.39 (+3.66) 22.56 (+1.83) 23.17 11.59 (−11.58) 13.41 (−9.76) 11.59 (−11.58) 21.34 (−1.83) 25.00 (+1.83) 22.56 (−0.61)

MBPP MMLU Base Models 28.20 52.34 29.20 52.96 32.20 52.23 23.40 50.00 Merged Models 31.60 55.04 34.40 (+2.80) 53.53 (−1.51) 35.20 (+3.60) 53.40 (−1.64) 33.80 (+2.20) 53.44 (−1.60) 23.60 (−8.00) 52.70 (−2.34) 35.60 (+4.00) 54.90 (−0.14) 35.20 (+3.60) 54.66 (−0.38) 30.20 52.22 28.60 (−1.60) 51.96 (−0.26) 27.40 (−2.80) 51.92 (−0.30) 28.60 (−1.60) 52.10 (−0.12) 26.80 (−3.40) 51.86 (−0.36) 29.40 (−0.80) 53.35 (+1.13) 30.60 (+0.40) 53.43 (+1.21) 29.00 54.33 19.00 (−10.00) 51.08 (−3.25) 23.80 (−5.20) 51.39 (−2.94) 20.20 (−8.80) 51.50 (−2.83) 34.60 (+5.60) 54.45 (+0.12) 30.40 (+1.40) 54.20 (−0.13) 32.40 (+3.40) 54.98 (+0.65) 30.20 54.19 19.60 (−10.60) 50.89 (−3.30) 21.20 (−9.00) 51.15 (−3.04) 19.60 (−10.60) 51.20 (−2.99) 29.20 (−1.00) 53.97 (−0.22) 33.20 (+3.00) 54.58 (+0.39) 31.00 (+0.80) 55.01 (+0.82)

source-level importance to allocate experts across heads and neurons improves expert composition over the average baseline and prior merging methods. Precise fusion achieves the best average result in three of the four expert combinations—Code + Instruction, Instruction + Math, and Code + Instruction + Math—while remaining competitive on Code + Math. The gains are largest when the fused experts are more behaviorally distinct, especially in mixtures involving the instruction expert and on IFEval, where preserving instruction-following-relevant components matters most. The smaller margin on Code + Math is also informative: code and math experts share stronger, structured reasoning updates, so generic merging is already effective, and the room for additional gains from source-aware allocation is correspondingly smaller. Rather than favoring one expert globally, Precise Fusion assigns experts at the component level according to localized source-level importance, better preserving sparse capability-relevant components while reducing the interference introduced by uniform averaging or globally shared merge coefficients. Taken together, these results show that Weight Patching is useful not only for localizing capability-relevant parameter subsets, but also for reusing the resulting localized structure in mechanismaware expert composition.

D ISCUSSION

Mechanistic Implication for Instruction Tuning. The recovered hierarchy in Fig. 2 suggests a staged division of labor,

MATH

GSM8K

IFEval

Avg.

0.80 5.90 7.30 12.90

3.00 24.60 42.10 57.80

23.59 23.55 35.51 21.80

21.14 25.95 33.31 29.99

7.20 8.40 (+1.20) 8.60 (+1.40) 8.60 (+1.40) 2.70 (−4.50) 8.30 (+1.10) 7.80 (+0.60) 10.40 15.10 (+4.70) 14.90 (+4.50) 15.00 (+4.60) 14.30 (+3.90) 14.20 (+3.80) 13.00 (+2.60) 12.70 9.80 (−2.90) 9.20 (−3.50) 10.00 (−2.70) 8.70 (−4.00) 14.60 (+1.90) 14.00 (+1.30) 11.30 9.10 (−2.20) 8.70 (−2.60) 9.00 (−2.30) 6.30 (−5.00) 13.50 (+2.20) 13.10 (+1.80)

42.10 45.80 (+3.70) 46.20 (+4.10) 47.10 (+5.00) 5.40 (−36.70) 45.00 (+2.90) 44.40 (+2.30) 49.00 64.70 (+15.70) 63.60 (+14.60) 64.70 (+15.70) 62.60 (+13.60) 64.40 (+15.40) 59.90 (+19.90) 61.80 54.30 (−7.50) 54.10 (−7.70) 54.20 (−7.60) 44.70 (−17.10) 66.00 (+4.20) 64.10 (+2.30) 54.20 49.70 (−4.50) 51.50 (−2.70) 52.70 (−1.50) 29.20 (−25.00) 64.20 (+10.00) 62.90 (+8.70)

40.30 33.42 (−6.88) 33.28 (−7.02) 31.60 (−8.70) 24.48 (−15.82) 30.42 (−9.88) 53.97 (+13.67) 21.05 22.02 (+0.97) 22.53 (+1.48) 21.92 (+0.87) 21.63 (+0.58) 24.02 (+2.97) 24.35 (+3.30) 29.11 32.35 (+3.24) 33.89 (+4.78) 31.31 (+2.20) 34.04 (+4.93) 30.82 (+1.71) 37.87 (+8.76) 27.37 33.20 (+5.83) 35.75 (+8.38) 32.87 (+5.50) 26.95 (−0.42) 31.44 (+4.07) 42.14 (+14.77)

34.05 33.73 (−0.32) 34.53 (+0.48) 33.97 (−0.08) 20.89 (−13.16) 33.41 (−0.64) 37.75 (+3.70) 29.79 33.14 (+3.35) 32.90 (+3.11) 33.43 (+3.64) 32.17 (+2.38) 33.74 (+3.95) 33.16 (+3.37) 34.61 28.67 (−5.94) 30.15 (−4.46) 28.58 (−6.03) 34.09 (−0.52) 36.74 (+2.13) 37.65 (+3.04) 33.41 29.01 (−4.40) 30.28 (−3.13) 29.49 (−3.92) 27.83 (−5.58) 36.99 (+3.58) 37.79 (+4.38)

in which shallow neurons emerge as candidate source-side carriers of instruction-conditioned control features, midlayer attention heads appear to aggregate and route these features into an instruction-following direction, and downstream instruction-specific modules help translate the recovered control state into response generation. Activation-space and parameter-space interventions provide complementary evidence for this picture: the former reveals where taskrelevant signals become effective during inference, whereas the latter identifies which post-training parameter changes contribute most strongly to reinstating the target control state. Scope and Assumptions. The interpretation developed here rests on three explicit assumptions. First, Weight Patching is defined in a paired-model setting: a base model and a behavior-specialized counterpart share the same architecture, so the causal object is the post-training parameter difference rather than an architecture-independent decomposition of a single checkpoint. Second, “source-level” is defined relative to the intervention granularity used here—namely, head- and neuron-level parameter slices—so the recovered carriers should be interpreted as granularitydependent rather than irreducible final origins. Third, in the generative-control setting studied here, Weight Patching is instantiated through a vector-anchor interface because textlevel counterfactuals are unstable and single-token readouts are incomplete. This interface is a practical criterion for the present analyses, not a requirement of Weight Patching in

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

general. Under the extraction procedure used here, anchorvector stability was not uniform across all IFEval tasks, so we focus on six representative tasks for which the analyses are reliable. Broader Potential. Although instruction following is our primary empirical setting, the framework may extend to other paired-model specializations for which a stable internal or behavioral interface can be defined; however, such extensions remain to be validated. In this paper, the resulting source-level component scores already prove useful beyond explanation, supporting mechanism-aware expert composition and achieving the best average performance among the compared methods in our evaluated fusion setting. More broadly, this source-level perspective may also aid training and post-training by identifying which parameterized components failed to acquire the intended specialization, introduced harmful interactions, or should be preferentially preserved, repaired, or reweighted. Developing such diagnostic and targeted repair mechanisms is a promising direction for future work.

R ELATED W ORK

6.1 Mechanistic Localization and Causal Intervention in Language Models Mechanistic interpretability aims to localize model behavior to the internal components that causally realize it, rather than merely describing correlations in activations or outputs [1], [2], [42]–[44]. In language models, this perspective has motivated a broad line of work that treats attention heads, MLPs, neurons, residual-stream updates, and information paths as meaningful computational units, supported by causal abstraction and causal scrubbing as mechanism-level evaluation frameworks [2], [45]–[52]. A central interventionist tool in this literature is activation patching [3], later extended to path patching, attribution patching, and circuit-discovery methods [4]–[6], [23]; related causal tracing work has further shown that internal interventions can identify behavior-relevant modules and even support subsequent parameter editing [10]. However, patching results are known to depend on corruption design, readout choice, and interpretation protocol [8], [9], and activation-space importance does not by itself establish that a capability is implemented in the patched module’s own parameters [2], [8], [10]. Our work addresses this gap by moving the intervention from activations to weights, with the goal of identifying source-level parameter carriers rather than downstream convergence states. 6.2 Interpreting Instruction Following and Behavioral Control Instruction following is one of the clearest capabilities induced by post-training and alignment [53]–[57], with improvements further supported by instruction collections, self-generated supervision, small high-quality alignment sets, and direct preference optimization [58]–[62]. Recent interpretability work has begun to characterize this specialization in terms of internal control representations: instruction-sensitive probes, task and function vectors, activation steering, and representation engineering all suggest

that instruction-conditioned behavior can often be monitored or manipulated through relatively structured directions in representation space [17]–[21]. Other studies further show that instructions and demonstrations need not rely on identical task representations, and that instruction tuning can systematically alter routing patterns related to goals, actions, and constraints [22], [63]. Yet this line of work remains largely focused on representations, activations, or inferencetime steering, rather than on which parameters act as source-level carriers of instruction-conditioned control; this limitation is especially important for generative behaviors, where text-level counterfactuals are hard to construct, and single-token metrics are incomplete [8], [9], [25]. Our work addresses this problem through a vector-anchor behavioral interface together with parameter-space intervention.

6.3 Parameter-Space Editing, Specialized Models, and Model Merging Related studies on direct parameter manipulation, particularly in knowledge editing, demonstrate that localized or structured weight updates can alter model behavior without full retraining [10], [64]–[66]. Work on task arithmetic and parameter-efficient finetuning further suggests that posttraining specialization is often concentrated in structured and transferable parameter subspaces [41], [67], [68]. Beyond editing, weight-space composition has become an increasingly important paradigm for combining specialized checkpoints, including Model Soups, Fisher Merging, TIESMerging, Model Ratatouille, Language Model Merging, and MergeKit [11]–[16]. These studies motivate our pairedmodel setting, where the difference between a base model and a specialized counterpart is treated as an analyzable object; unlike prior work on editing or merging, however, our primary goal is not to modify behavior as an end in itself, but to use parameter replacement as a causal probe for source-level mechanistic localization, while also showing that the recovered source signals can support mechanismaware model merging.

C ONCLUSION

This paper presents a framework for source-oriented mechanistic analysis in paired same-architecture large language models, centered on Weight Patching, a parameter-space intervention for identifying candidate source-side parameter carriers under a fixed input. For generative-control tasks, we instantiate this analysis through a vector-anchor behavioral interface and a scalable first-order screening strategy, making source-oriented analysis practical for open-ended generation. In the instruction following, the recovered evidence is consistent with a staged organization in which shallow components act as candidate source-side carriers, mid-layer modules serve as aggregation/routing interfaces, and downstream modules support execution. Beyond mechanistic explanation, the same recovered component scores also support practical reuse, including scalable fine-grained screening and WP-guided mechanism-aware expert merging.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

ACKNOWLEDGMENT This work was supported by the Natural Science Foundation of China under Grant 62571507.

R EFERENCES [1]

C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter, “Zoom in: An introduction to circuits,” Distill, vol. 5, no. 3, pp. e00 024–001, 2020. [2] A. Geiger, D. Ibeling, A. Zur, M. Chaudhary, S. Chauhan, J. Huang, A. Arora, Z. Wu, N. Goodman, C. Potts et al., “Causal abstraction: A theoretical foundation for mechanistic interpretability,” Journal of Machine Learning Research, vol. 26, no. 83, pp. 1–64, 2025. [3] K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt, “Interpretability in the wild: A circuit for indirect object identification in GPT-2 small,” in International Conference on Learning Representations (ICLR), 2023. [4] N. Goldowsky-Dill, C. MacLeod, L. Sato, and A. Arora, “Localizing model behavior with path patching,” arXiv preprint arXiv:2304.05969, 2023. [5] A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso, “Towards automated circuit discovery for mechanistic interpretability,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023, pp. 16 318–16 352. [6] A. Syed, C. Rager, and A. Conmy, “Attribution patching outperforms automated circuit discovery,” in Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2024, pp. 407–416. [7] M. Lan, P. Torr, and F. Barez, “Towards interpretable sequence continuation: Analyzing shared circuits in large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024, pp. 12 576–12 601. [8] S. Heimersheim and N. Nanda, “How to use and interpret activation patching,” arXiv preprint arXiv:2404.15255, 2024. [9] F. Zhang and N. Nanda, “Towards best practices of activation patching in language models: Metrics and methods,” in International Conference on Learning Representations (ICLR), 2024. [10] K. Meng, D. Bau, A. Andonian, and Y. Belinkov, “Locating and editing factual associations in GPT,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 17 359–17 372. [11] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt, “Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in Proceedings of the 39th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 23 965–23 998. [12] M. S. Matena and C. A. Raffel, “Merging models with fisherweighted averaging,” in Advances in Neural Information Processing Systems, 2022. [13] P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal, “Tiesmerging: Resolving interference when merging models,” in Advances in Neural Information Processing Systems, 2023. [14] A. Ramé, K. Ahuja, J. Zhang, M. Cord, L. Bottou, and D. LopezPaz, “Recycling diverse models for out-of-distribution generalization,” in Proceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 28 656–28 679. [15] L. Yu, Y. Bowen, H. Yu, F. Huang, and Y. Li, “Language models are super mario: Absorbing abilities from homologous models as a free lunch,” ArXiv, vol. abs/2311.03099, 2023. [16] C. Goddard, S. Siriwardhana, M. Ehghaghi, L. Meyers, V. Karpukhin, B. Benedict, M. McQuade, and J. Solawetz, “Arcee’s mergekit: A toolkit for merging large language models,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. Miami, Florida, US: Association for Computational Linguistics, 2024, pp. 477–485. [17] J. Heo, C. Heinze-Deml, O. Elachqar, K. H. R. Chan, S. Y. Ren, A. C. Miller, U. Nallasamy, and J. Narain, “Do LLMs "know" internally when they follow instructions?” in The Thirteenth International Conference on Learning Representations (ICLR), 2025. [18] E. Todd, M. L. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau, “Function vectors in Large Language Models,” in The Twelfth International Conference on Learning Representations (ICLR), 2024.

[19] A. Stolfo, V. Balachandran, S. Yousefi, E. Horvitz, and B. Nushi, “Improving instruction-following in language models through activation steering,” in The Thirteenth International Conference on Learning Representations (ICLR), 2025. [20] N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner, “Steering Llama 2 via contrastive activation addition,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024, pp. 15 504–15 522. [21] A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, S. Mallen, S. K. Basart, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks, “Representation engineering: A top-down approach to ai transparency,” arXiv preprint arXiv:2310.01405, 2023. [22] X. Wu, W. Yao, J. Chen, X. Pan, X. Wang, N. Liu, and D. Yu, “From language modeling to instruction following: Understanding the behavior shift in LLMs after instruction tuning,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024, pp. 2341–2369. [23] N. Nanda, “Attribution patching: Activation patching at industrial scale,” URL: https://www. neelnanda. io/mechanisticinterpretability/attribution-patching, vol. 15, p. 19, 2023. [24] J. Kramár, T. Lieberum, R. Shah, and N. Nanda, “AtP*: An efficient and scalable method for localizing llm behaviour to components,” arXiv preprint arXiv:2403.00745, 2024. [25] J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for Large Language Models,” arXiv preprint arXiv:2311.07911, 2023. [26] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. AlDahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [27] H. Touvron et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023. [28] Qwen Team, “Qwen2.5 technical report,” arXiv preprint arXiv:2412.15115, 2024. [29] Mistral AI Team, “Mistral-7B-v0.3 model card,” Hugging Face model card, 2024. [Online]. Available: https://huggingface.co/ mistralai/Mistral-7B-v0.3 [30] Gemma Team, “Gemma 2: Improving open language models at a practical size,” arXiv preprint arXiv:2408.00118, 2024. [31] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li, “Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement,” arXiv preprint arXiv:2408.03092, 2024. [32] A. H. Nobari, K. Alimohammadi, A. ArjomandBigdeli, A. Srivastava, F. Ahmed, and N. Azizan, “Activation-informed merging of large language models,” in Advances in Neural Information Processing Systems, 2025, neurIPS 2025. [Online]. Available: https://arxiv.org/abs/2502.02421 [33] C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang, “Wizardlm: Empowering large language models to follow complex instructions,” 2023. [34] H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang, “Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct,” 2023. [35] S. Chaudhary, “Code alpaca: An instruction-following llama model for code generation,” GitHub repository, 2023, accessed 2026-03-19. [36] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, Ł. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. Petroski Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. [37] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [38] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language under-

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

standing,” in International Conference on Learning Representations (ICLR), 2021. [39] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the MATH dataset,” in Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. [40] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, Ł. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168, 2021. [41] G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” arXiv preprint arXiv:2212.04089, 2022. [42] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly et al., “A mathematical framework for transformer circuits,” Transformer Circuits Thread, vol. 1, no. 1, p. 12, 2021. [43] L. Bereska and E. Gavves, “Mechanistic interpretability for AI safety: A review,” Transactions on Machine Learning Research (TMLR), 2024. [44] T. Räuker, A. Ho, S. Casper, and D. Hadfield-Menell, “Toward transparent ai: A survey on interpreting the inner structures of deep neural networks,” in 2023 ieee conference on secure and trustworthy machine learning (satml). IEEE, 2023, pp. 464–483. [45] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan et al., “In-context learning and induction heads,” arXiv preprint arXiv:2209.11895, 2022. [46] T. Lieberum, M. Rahtz, J. Kramár, G. Irving, R. Shah, and V. Mikulik, “Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla,” arXiv preprint arXiv:2307.09458, 2023. [47] M. Hanna, O. Liu, and A. Variengien, “How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model,” in Advances in Neural Information Processing Systems, vol. 36, 2023, pp. 76 033–76 060. [48] W. Zhang, C. Wan, Y. Zhang, Y.-m. Cheung, X. Tian, X. Shen, and J. Ye, “Interpreting and improving Large Language Models in arithmetic calculation,” in International Conference on Machine Learning (ICML), 2024. [49] D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei, “Knowledge neurons in pretrained Transformers,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022, pp. 8493–8502. [50] X. Wang, K. Wen, Z. Zhang, L. Hou, Z. Liu, and J. Li, “Finding skill neurons in pre-trained transformer-based language models,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022, pp. 11 132–11 152. [51] A. Geiger, H. Lu, T. Icard, and C. Potts, “Causal abstractions of neural networks,” Advances in Neural Information Processing Systems, vol. 34, pp. 9574–9586, 2021. [52] L. Chan, A. Garriga-Alonso, N. Goldowsky-Dill, R. Greenblatt, J. Nitishinskaya, A. Radhakrishnan, B. Shlegeris, and N. Thomas, “Causal scrubbing: A method for rigorously testing interpretability hypotheses,” vol. 2, p. 19, 2022. [53] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022. [54] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” arXiv preprint arXiv:2109.01652, 2021. [55] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei et al., “The flan collection: Designing data and methods for effective instruction tuning,” in International conference on machine learning. PMLR, 2023, pp. 22 631–22 648. [56] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862, 2022. [57] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon et al., “Constitutional ai: Harmlessness from ai feedback,” arXiv preprint arXiv:2212.08073, 2022.

[58] S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi, “Cross-task generalization via natural language crowdsourcing instructions,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 3470–3487. [59] Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, E. Pathak, G. Karamanolakis, H. Lai, I. Purohit, I. Mondal, J. Anderson, K. Kuznia, K. Doshi, K. K. Pal, M. Patel, M. Moradshahi, M. Parmar, M. Purohit, N. Varshney, P. R. Kaza, P. Verma, R. S. Puri, R. Karia, S. Doshi, S. K. Sampat, S. Mishra, S. Reddy A, S. Patro, T. Dixit, and X. Shen, “Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022, pp. 5085–5109. [60] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with selfgenerated instructions,” in Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), 2023, pp. 13 484–13 508. [61] C. Zhou et al., “Lima: Less is more for alignment,” arXiv preprint arXiv:2305.11206, 2023. [62] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” Advances in neural information processing systems, vol. 36, pp. 53 728–53 741, 2023. [63] G. Davidson, T. M. Gureckis, B. M. Lake, and A. Williams, “Do different prompting methods yield a common task representation in language models?” arXiv preprint arXiv:2505.12075, 2025. [64] K. Meng, A. S. Sharma, A. Andonian, Y. Belinkov, and D. Bau, “Mass-editing memory in a transformer,” in International Conference on Learning Representations (ICLR), 2023. [65] E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning, “Fast model editing at scale,” 2021. [66] E. Mitchell, C. Lin, A. Bosselut, C. D. Manning, and C. Finn, “Memory-based model editing at scale,” in Proceedings of the 39th International Conference on Machine Learning (ICML). PMLR, 2022, pp. 15 817–15 831. [67] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., “Lora: Low-rank adaptation of large language models.” Iclr, vol. 1, no. 2, p. 3, 2022. [68] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in neural information processing systems, vol. 36, pp. 10 088–10 115, 2023.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Supplementary Material Appendix Overview

A.1 Textual Sections and Visualizations The textual appendix contains three main domains, structured as follows:

localization (./code/src/main_wp.py). To enable efficient full-model screening without exhaustive enumeration, we provide its first-order gradient attribution variant, WP-Attr (./code/src/main_wp_attr.py). Both projection and KL-divergence metrics are supported. For comparative analysis, we also include a cross-model Activation Patching (AP) baseline. Notably, as formalized in App. A.3, unlike standard AP (which contrasts two inputs within a single model), our formulation performs a single-input, dual-model intervention by patching instruction-conditioned activations into the base model under a fixed prompt.

• Detailed Methodological Formulations (Appendix A):

• WP-Guided Model Merging: The fine-grained model fu-

This appendix provides comprehensive supplementary materials that expand on the methods, experiments, and findings presented in the main paper. The appendix is organized into detailed methodological formulations, rigorous experimental setups, and extended empirical evaluations. In addition, an accompanying codebase is provided in the supplementary materials.

Provides the complete mathematical and operational formulations underlying our framework. This includes architecture-specific parameterizations for attention and MLP components (App. A.1), the full construction of the vector-anchor interface (App. A.2), definitions for exact Activation Patching and Weight Patching operators (App. A.3, A.4), first-order approximation derivations for efficient full-model screening (App. A.5), the staged hierarchy recovery procedure (App. A.6), and the exact implementation rules for WP-guided model fusion (App. A.7). • Reproducibility

and Experimental Setup (Appendix B): Details the precise configurations required for reproducibility. This section covers the architectural statistics of the evaluated Llama models (App. B.1), task descriptions and evaluation protocols from IFEval (App. B.2), data splitting and vector-extraction strategies (App. B.3), granular hyperparameters for steering and patching interventions (App. B.4), component selection rules (App. B.5), and the explicit hardware/precision settings for model merging (App. B.6).

• Extended

Results and Visualizations (Appendix C): Introduces supplementary empirical evidence verifying the robustness of our core findings. This section provides rich visual and quantitative materials, including component localization heatmaps that demonstrate cross-scale source–aggregation separation (Llama-3.1-8B and 13B) and layer-wise parameter shifts (App. C.3, C.4, C.9). It also features causal validation radar charts detailing behavioral recovery trajectories across diverse LLM families (Gemma, Mistral, Qwen) (App. C.5), vocabulary-space projections (App. C.2, C.6), selective attention analysis (App. C.7), and qualitative text generation case studies showcasing explicit behavioral changes under targeted module ablation and restoration.

A.2 Supplementary Code Implementation As part of the supplementary materials, we provide the code implementation for our mechanistic probes and downstream applications. The repository highlights our core contributions in parameter-space localization: • Weight Patching (WP) and Gradient Approximations:

Features the core implementation of our proposed exact Weight Patching operator for source-level capability

sion (./code/src/core/model_merging.py), which leverages the derived head-level and neuron-level WP attribution scores to dynamically interpolate parameters across multiple SFT models. This implementation inherently supports Grouped-Query Attention (GQA) architectures. • Evaluation and Analysis Pipeline: Example shell scripts

orchestrating the workflow (e.g., ./code/sh/wp.sh and ./code/sh/wp_attr.sh)—from contrastive steering vector extraction and component-wise patching, to WP attribution scoring and multi-model merging.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

A PPENDIX A D ETAILED M ETHODOLOGICAL F ORMULATIONS A.1 Detailed Component Interface and ArchitectureSpecific Parameterization This subsection provides the full component-level interface used throughout the method. While the main text retains only the minimal residual-stream view needed to define intervention units, the detailed formulation is important for implementation and reproducibility. In particular, we specify how attention heads and SwiGLU neurons are represented as write-back units in the shared residual stream, and how architecture-specific details such as grouped-query attention affect the definition of replaceable parameter slices. These details justify the component granularity adopted by both Weight Patching and its first-order approximation. A.1.0.1 Residual-stream coordinate system.: Let the input token sequence be x = (x1 , . . . , xT ), and let

Z0 = Embed(x) ∈ RT ×dmodel

(19)

denote the input embedding matrix. For layer l ∈ {1, . . . , L}, let Zl ∈ RT ×dmodel denote the residual-state (t) matrix after layer l, and let zl ∈ Rdmodel denote its row at position t. Under a pre-norm decoder, the residual stream is updated as Z′l = Zl−1 + Attnl Norm(Zl−1 ) , (20) Zl = Z′l + MLPl Norm(Z′l ) . (21) This shared residual-stream space serves as the common coordinate system in which all later activation-space and parameter-space interventions are defined. A.1.0.2 Attention heads as write-back units.: Let X = Norm(Zl−1 ) denote the normalized input to layer l. For query head h, the head-level query slice is (l,h)

Q(l,h) = XWQ

∈ RT ×dk .

(22)

In standard multi-head attention, the key and value slices can be written as (l,h)

K(l,h) = XWK

(l,h)

V(l,h) = XWV

(23)

The resulting head output is

(l,h)

! Q(l,h) (K(l,h) )⊤ √ = Softmax + M V(l,h) , dk

(24)

where M is the causal mask. The head writes back to the residual stream through its output projection: (l,h)

(l,h)

∆Zattn = O(l,h) WO

∈ RT ×dmodel .

(25)

This write-back form makes the attention head a natural component-level intervention unit. A.1.0.3 Grouped-query attention and head granularity.: In architectures with grouped-query attention, multiple query heads may share the same key and value projections. In that setting, replacing WK or WV locally would affect an entire group of query heads simultaneously, rather than an individual head. This breaks the intended component granularity and introduces causal ambiguity in head-level parameter interventions. For this reason, the

parameter-space intervention in the main text and Appendix A.4 restricts the head-level replaceable slice to the head-specific query and output parameters, while treating shared key/value parameters as outside the scope of a clean single-head replacement. A.1.0.4 SwiGLU MLPs and neuron write-back.: For the SwiGLU MLP used in modern decoder-only LLMs, the feed-forward block at layer l is (l) (l) (l) MLPl (X) = SiLU(XWgate ) ⊙ (XWup ) Wdown . (26) Let N (l,j) denote the j -th hidden unit. Its contribution can be written as (l,j)

(l)

∆Zmlp = a(l,j) Wdown [j, :] ∈ RT ×dmodel ,

(27)

where a(l,j) denotes the corresponding hidden activation induced jointly by the gate and up-projection columns. Thus, an MLP neuron naturally forms a parameter-grounded feature-detection, gating, and write-back unit. A.1.0.5 Candidate component set.: Taken together, both attention heads and MLP neurons ultimately appear as additive component-level updates in the same residualstream space. We therefore define the candidate component set as n o n o C = H (l,h) ∪ N (l,j) , (28) where the first subset ranges over attention heads and the second over MLP neurons. All exact and first-order interventions in the paper are defined with respect to this common component interface. A.2 Anchor Construction, Layer Selection, and Steering Calibration This subsection gives the full construction of the vectoranchor behavioral interface used in the main text. The key idea is to replace unstable text-level counterfactuals with an internal control representation extracted from contrastive instruction-present and instruction-removed pairs. We detail how layer-wise task directions are computed, how the anchor layer and anchor position are selected, and how steering magnitude is calibrated in an input-adaptive way so that the resulting interface can be used consistently across steering, localization, restoration, and ablation. A.2.0.1 Paired data construction.: Let Dinst = (i) (i) (i) {(xr , xcf )}N i=1 denote a paired dataset, where xr is an (i)

instruction-present input and xcf is its instruction-removed counterpart with matched context. In all analyses, the extraction split used to construct task directions is kept disjoint from the evaluation split used for steering, localization, ablation, and restoration, so that the vector-anchor interface is not defined and tested on the same examples. A.2.0.2 Layer-wise task directions.: For each candidate layer l ∈ L, we extract an instruction direction by averaging residual-stream differences at a designated anchor position ta :

v(l) =

N i 1 X h (ta ) (i) (t ) (i) zl (xr | Msft ) − zl a (xcf | Msft ) . (29) N i=1

This construction suppresses shared semantic content while preserving the control-relevant variation induced by the instruction itself.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

A.2.0.3 Anchor position and anchor layer selection.: The anchor position ta is the designated readout position used to evaluate whether the control representation has been formed. Given the layer-wise directions in Eq. (29), the anchor layer is selected according to the behavioral recovery achieved by steering along each candidate direction: (30) la = arg max Recover v(l) ,

A.2.0.7 Summary.: Taken together, Eqs. (29)–(37) define the full vector-anchor construction used in the paper. The paired instruction/no-instruction contrast yields a task direction, anchor-layer selection turns this direction into a stable readout interface, and input-adaptive calibration makes the resulting intervention usable across examples without introducing an arbitrary global steering scale.

l∈L

where Recover(·) denotes the steering-based behavioral recovery score evaluated on a held-out split. The final task vector is then set to v = v(la ) . (31)

A.3 Exact Activation-Space Intervention and CrossModel Activation Patching

This subsection provides the exact activation-space intervention protocol used as auxiliary evidence in the main text. Activation patching is not the primary source-level method of this paper, but it remains useful for identifying In practice, the selected anchor layers tend to lie in middlewhere instruction-conditioned signals become causally efto-late layers, supporting the interpretation that the exfective during inference. We therefore distinguish the stantracted vector behaves as an internal control representation dard clean/corrupted activation-patching template from the rather than a shallow lexical residue. cross-model activation-patching setting used here, in which A.2.0.4 Anchor utility and anchor gap.: Based on activations from the specialized model are patched into the the selected anchor layer and task vector, the anchor utility base-model run under a fixed input. is defined as A.3.0.1 Standard activation patching.: In the con (ta ) ventional setting, activation patching is defined on a single Fa (M, x) = sim za (x | M ), v , za (x | M ) ≡ zla (x | M ), model with a clean input xclean and a corrupted input xcorr . (32) For a component c ∈ C , the patched run is and the specialized-to-base anchor gap is

G(x) = Fa (Msft , x) − Fa (Mbase , x).

(33)

This gap serves as the normalization term used throughout exact and first-order restoration analysis. A.2.0.5 Input-adaptive steering calibration.: To steer the base model using the extracted task vector, we first normalize the direction: v v̂ = . (34) ∥v∥2 Next, we estimate the average projection strength of instructed examples along this direction: N

µ̄ =

E 1 XD za (x(i) r | Msft ), v̂ . N i=1

(35)

(36)

The corresponding steering intervention at the anchor is

z̃a (x | Mbase ) ← za (x | Mbase ) + α(x)v̂.

(37)

This calibration ensures that the intervention magnitude adapts to the current base-model state rather than using a fixed global scale across all examples. A.2.0.6 Role of the interface.: If the steered base model reliably recovers the target behavior, the extracted direction is treated as a valid functional anchor for subsequent mechanistic localization. The vector-anchor interface therefore serves as a shared internal behavioral criterion under which steering, activation-space restoration, parameterspace restoration, and downstream hierarchy analysis can all be compared consistently.

(38)

Given a generic utility function F , the corresponding restoration effect is ∆Fpatch (c) = F M (xcorr | do(zc ← zc (xclean ))), xcorr − F M (xcorr ), xcorr . (39) This form measures whether a component’s clean-run activation is sufficient to restore behavior in the corrupted run. A.3.0.2 Cross-model activation patching under fixed input.: In the present paired-model setting, the goal is different. Rather than contrasting two inputs within one model, we compare two models under the same instructionpresent input x. For any component c ∈ C , let

zc (x | Mbase )

Given an input x, the input-adaptive steering coefficient is then defined as

α(x) = µ̄ − ⟨za (x | Mbase ), v̂⟩ .

M (xcorr | do(zc ← zc (xclean ))) .

and

zc (x | Msft )

(40)

denote the corresponding component activations in the base and specialized models. The cross-model activation-patched run is then defined as f(c) (x) := Mbase x | do zc ← zc (x | Msft ) . (41) M base This intervention asks whether the specialized activation of a local component is sufficient to restore the anchorside control representation when transplanted into the basemodel run under the same input. A.3.0.3 Anchor-based utility for activation-space restoration.: The paper evaluates activation-space restoration through the same vector-anchor interface used in Weight Patching. The anchor utility is (t ) Fa (M, x) = sim zlaa (x | M ), v , (42) and the corresponding specialized-to-base gap is

G(x) = Fa (Msft , x) − Fa (Mbase , x).

(43)

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Using this shared interface, the exact activation-side restoration effect of component c is defined as # " f(c) , x) − Fa (Mbase , x) Fa (M base . (44) Ea (c) = Ex∼Dinst G(x)

The corresponding set-wise replaced parameter tensor is defined elementwise as ( θsft [p], p ∈ Θ(S) , (S) θbase←sft [p] = (46) θbase [p], p ∈ / Θ(S) .

Large Ea (c) indicates that the component plays an important role in forming, routing, or restoring the task-relevant control signal during inference. A.3.0.4 Interpretation.: The role of cross-model activation patching differs from that of Weight Patching. Activation patching identifies where behavior-relevant information becomes causally effective during inference, such as restoration points, routing modules, or aggregation bottlenecks. It does not, by itself, establish that the corresponding capability is implemented in the patched component’s own parameters. This is precisely why activation-space evidence is used in the paper as a complementary tool for hierarchy recovery, whereas source-level parameter causality is assigned only through Weight Patching. A.3.0.5 Use in hierarchy analysis.: In the main text and Appendix A.6, exact activation-side restoration is used in two ways. First, components with large Ea (c) are treated as candidates for aggregation and routing modules. Second, the specialized activation difference induced by these components provides the supply-side signal used in the later supply–need matching procedure. In this sense, activationspace intervention is not a separate competing method, but an auxiliary mechanism for identifying where the control signal flows once it has been written. A.3.0.6 Summary.: Taken together, Eqs. (38)–(44) define the exact activation-space intervention protocol used in the paper. The standard clean/corrupted formulation provides the general causal template, and the cross-model fixed-input version adapts that template to the pairedmodel setting studied here. The resulting activation-side effect serves as evidence for information flow, restoration, and aggregation, complementing the source-level parameter evidence provided by Weight Patching.

This induces the set-wise parameter-patched model (S←sft) Mbase := Replace Mbase , Msft ; Θ(S) .

A.4 Detailed Weight Patching Operator and MultiComponent Restoration This subsection provides the full parameter-space intervention operator underlying Weight Patching. The main text focuses on the single-component case because source localization is defined at the component level, but the same replacement rule extends naturally to multi-component restoration and ablation. We therefore formalize both the single-component and set-wise parameter-patched models, specify the exact parameter slices used for attention heads and MLP neurons, and clarify why the chosen slices align with the functional granularity of the mechanistic units studied in this paper. A.4.0.1 Single-component and set-wise parameter replacement.: Let Mbase and Msft denote the base model and the specialized counterpart, with parameter sets θbase and θsft , respectively. For any component c ∈ C , let Θ(c) denote its replaceable parameter slice. More generally, for a component set S ⊆ C , define the union slice [ Θ(S) = Θ(c) . (45) c∈S

(47)

The single-component case is recovered by setting S = {c}: (c←sft) (48) Mbase := Replace Mbase , Msft ; Θ(c) . The distinction between Eqs. (47) and (48) is operationally important. Source localization is defined at the single-component level, because the goal is to test whether the specialized parameters of an individual head or neuron are sufficient to reintroduce the target control signal. By contrast, Eq. (47) is used only when evaluating multicomponent restoration or ablation, such as top-K recovery experiments, where the objective is to test whether a sparse set of already-ranked components can collectively close a substantial fraction of the anchor gap. A.4.0.2 Attention-head parameter slices.: For an attention head H (l,h) , the replaceable slice is defined as n o (l,h) (l,h) (l,h) ) Θ(H = WQ , W O . (49) This design deliberately excludes WK and WV . In standard multi-head attention, one could in principle define a larger head-level slice that also includes key and value projections. However, in architectures with grouped-query attention, key and value projections are shared across multiple query heads. A local replacement of WK or WV would therefore affect a group of query heads simultaneously, breaking the intended component granularity and introducing causal ambiguity. Restricting the head-level slice to the headspecific query and output projections preserves a clean intervention unit while remaining faithful to the head’s role in reading from and writing back to the residual stream. When needed for implementation, let Ih denote the index slice corresponding to query head h. Then Eq. (49) can (l) be understood as replacing the columns of WQ indexed by (l)

Ih together with the rows of WO indexed by Ih . This indexlevel view is also used in Appendix A.5 to derive head-level first-order aggregation formulas. A.4.0.3 MLP-neuron parameter slices.: For an MLP neuron N (l,j) in a SwiGLU block, the replaceable slice is defined as n o (l,j) (l) (l) (l) (50) Θ(N ) = Wgate [:, j], Wup [:, j], Wdown [j, :] . This slice preserves the full functional interface of the neuron. The gate and up-projection columns determine which input patterns activate the neuron and how strongly they are gated, while the down-projection row determines the direction in residual-stream space to which the activated feature is written back. Replacing only one of these three parts would no longer correspond to a coherent neuronlevel intervention. Equation (50) therefore treats the neuron as a complete feature-detection, gating, and write-back unit.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

A.4.0.4 Source-level exact restoration effect.: Under the vector-anchor behavioral interface defined in the main text, the source-level effect of component c is measured by how much of the anchor gap is restored when only that component’s specialized parameters are transplanted into the base model:  (c←sft)  Fa Mbase , x − Fa (Mbase , x) , Ew (c) = Ex∼Dinst  G(x) (51) where

G(x) = Fa (Msft , x) − Fa (Mbase , x)

(52)

is the specialized-to-base anchor gap. A large value of Ew (c) indicates that the specialized parameters of component c are sufficient to recover a substantial fraction of the control representation missing from the base model under the same input. This is the central source-level causal criterion used throughout the paper. For a component set S ⊆ C , the corresponding set-wise restoration effect is defined analogously:  (S←sft) Fa Mbase , x − Fa (Mbase , x) . Ew (S) = Ex∼Dinst  G(x) (53) Equation (53) is not used to define the basic localization unit, but it is useful in two situations. First, it supports multicomponent validation, where one tests whether a sparse set of top-ranked components can jointly recover the anchorside control signal. Second, it supports targeted ablations or restoration studies that probe whether the recovered source carriers behave collectively as a reusable mechanism rather than as isolated components. A.4.0.5 Relation to the sparse source-localization objective.: The single-component and set-wise formulations above also connect to the sparse restoration objective in a more explicit optimization form. At a high level, source localization seeks a sparse component set whose replacement substantially restores the anchor representation in the base model: 

min |S| S⊆C

s.t.

(t ) (S←sft) , v ≥ τ. sim zlaa x | Mbase

(54)

In practice, this objective is not solved directly. Instead, the paper proceeds by first ranking components using firstorder screening, then validating top candidates with the exact single-component effect in Eq. (51), and finally using Eq. (53) only for sparse set restoration or ablation. A.4.0.6 Summary.: Taken together, Eqs. (47)–(53) define Weight Patching as a parameter-space intervention family centered on component-aligned replacement. The single-component case provides the source-level causal primitive, the set-wise case supports sparse restoration and ablation, and the head/neuron slice definitions ensure that the intervention granularity remains consistent with the mechanistic units analyzed throughout the paper.

A.5 First-Order Approximation, Head/Neuron Aggregation, and Full-Model Screening This subsection develops the first-order approximation used to scale source localization to all candidate heads and neurons. The exact activation-space and parameter-space interventions defined in the main text provide the most direct causal evidence, but applying them exhaustively to every component in a large language model is computationally expensive, especially at neuron scale. To make full-model screening practical, we approximate each intervention by the local first-order response of the anchor utility. This yields efficient activation-side and weight-side attribution scores that preserve the distinction between dynamic signal flow and parameter-side responsibility, while allowing exact AP/WP validation to be restricted to a small top-ranked subset. A.5.0.1 A unified first-order view.: Both activation patching and Weight Patching can be interpreted as perturbations applied around the base model. Let Fa (Mbase , x) denote the anchor utility defined in the main text. For a small perturbation δu applied to an intermediate state or a parameter slice, the corresponding utility change admits the first-order approximation

∆Fa ≈ ⟨∇u Fa (Mbase , x), δu⟩ .

(55)

This form makes explicit that the intervention effect is approximated by the alignment between a local sensitivity direction and the perturbation induced by the specialized model. The approximation is used only for screening: exact source-level conclusions still rely on the exact Weight Patching effect in the main text. A.5.0.2 Activation-side first-order approximation.: For an input x and component c ∈ C , define the cross-model activation difference

∆zc (x) = zc (x | Msft ) − zc (x | Mbase ).

(56)

A first-order expansion of the anchor utility around the base-model activation gives

Fa (zc + ∆zc , x) − Fa (zc , x) ≈ ⟨∇zc Fa (Mbase , x), ∆zc (x)⟩ . (57) Accordingly, the activation-side attribution score is defined as ⟨∇zc Fa (Mbase , x), ∆zc (x)⟩ Attract (c) = Ex∼Dinst . (58) G(x) Large Attract (c) indicates that the specialized activation shift of component c is well aligned with the local direction that increases the anchor utility, making the component a strong candidate for activation-side restoration, aggregation, or routing analysis. A.5.0.3 Weight-side first-order approximation.: Let ∆θ = θsft −θbase denote the post-training parameter change. For a component c with parameter slice Θ(c) , the first-order approximation to the weight-side effect is P  ∂Fa (Mbase ,x) p∈Θ(c) ∆θp ∂θp  . (59) Attrwt (c) = Ex∼Dinst  G(x) Equation (59) measures whether the post-training parameter update on slice Θ(c) points in a direction that increases the

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

anchor utility at the base-model parameter point. Unlike Eq. (58), which approximates activation-side restoration, Eq. (59) is directly aligned with the source-level question of where the target control signal is written in parameters. A.5.0.4 Head-level aggregation.: For an attention head H (l,h) , the replaceable slice contains the head-specific query and output parameters only. Let Ih denote the index slice corresponding to query head h. The head-level firstorder score is aggregated from the query and output parts:

with

Attrgate (l, j) =  P  Ex∼Dinst 

 (l) ∆Wgate ⊙ ∇W (l) Fa (Mbase , x) [:, j]  gate , G(x) (64)

Attrup (l, j) =  P



(l) ∆Wup ⊙ ∇W (l) Fa (Mbase , x) [:, j] up

 Ex∼Dinst 

Attrwt (H (l,h) ) = AttrQ (l, h) + AttrO (l, h),

G(x)

  , (65)

Attrdown (l, j) =

(60)  P  Ex∼Dinst 

(l) ∆Wdown ⊙ ∇W (l) Fa (Mbase , x) [j, :] down

G(x)

  . (66)

where

AttrQ (l, h) =  P  Ex∼Dinst 

(l) ∆WQ ⊙ ∇W (l) Fa (Mbase , x) [:, Ih ] Q

G(x)

  , (61)

wt Ctop = TopK(C, Attrwt , Kwt )

AttrO (l, h) =  P  Ex∼Dinst 

This aggregation reflects the fact that a SwiGLU neuron is not a single scalar parameter but a structured functional unit whose feature detection, gating, and write-back direction are jointly responsible for its contribution to the control representation. A.5.0.6 Full-model screening and exact validation.: The first-order scores above are used to rank all candidate heads and neurons before exact intervention. Let

 (l) ∆WO ⊙ ∇W (l) Fa (Mbase , x) [Ih , :]  O . G(x) (62)

P Here, ⊙ denotes elementwise multiplication, and (·) sums all entries in the selected slice. This aggregation preserves the component granularity established in Appendix A.4: only the head-specific query and output interfaces contribute to the head-level WAP score. If activation-side head-level screening is also needed, an analogous aggregation can be defined by treating the head (l,h) write-back activation ∆Zattn as the intervention object and applying Eq. (58) at the head level. In practice, however, the main screening emphasis in this paper is on the weight-side score in Eq. (60). A.5.0.5 Neuron-level aggregation.: For an MLP neuron N (l,j) , the parameter slice contains the gate column, up-projection column, and down-projection row. The neuron-level first-order score is therefore aggregated over these three parts:

Attrwt (N (l,j) ) = Attrgate (l, j)+Attrup (l, j)+Attrdown (l, j), (63)

(67)

denote the top-ranked components under weight-side screening. If activation-side screening is also performed, define act Ctop = TopK(C, Attract , Kact )

(68)

Exact Weight Patching is then applied only to components wt , and exact activation-side intervention is applied in Ctop act when needed for hierarchy only to components in Ctop recovery. Thus, the overall pipeline separates screening from validation:

Attrwt (c) =⇒ ranking over all c ∈ C, Ew (c) =⇒ exact source-level validation on top-ranked candidates.

(69) (70)

This two-stage design is especially important for neuronscale localization, where exhaustive exact Weight Patching is prohibitively expensive. A.5.0.7 Role of the approximation.: The first-order approximation is not intended to replace exact intervention as the source of causal evidence. Its role is methodological: it preserves the distinction between activation-side and parameter-side scoring while making full-model screening computationally feasible. In the empirical pipeline, the approximation is therefore judged by two criteria only: whether it retains the components most likely to matter under exact validation, and whether it reduces runtime enough to make fine-grained head- and neuron-level localization practical.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

A.5.0.8 Summary.: Taken together, Eqs. (58)–(68) define a unified first-order screening layer for the proposed framework. Activation-side approximation highlights components whose cross-model state differences most strongly support anchor restoration, whereas weight-side approximation highlights components whose post-training parameter changes most strongly support source-level recovery. Head-level and neuron-level aggregation ensure that the approximation remains aligned with the same mechanistic units used by exact Weight Patching in the main text.

This readout is used in the main text because downstream execution modules are most directly characterized by how an intervention changes the distribution of the immediately next generated token. For completeness, the execution stage can also be evaluated with a projection-based readout that mirrors the upstream anchor utility, except that the readout is taken at the final layer rather than the anchor layer. Let

A.6 Hierarchy Recovery, Supplier–Target Tracing, and Source Validation

denote the final-layer residual state at the position used to predict the immediately next token, and let vout denote the task direction extracted by the same contrastive construction as in Eq. (11), but at layer L. The projection-based downstream utility is then

This subsection gives the full hierarchy-recovery procedure used to connect source carriers to aggregation, routing, and downstream execution modules. The main text states the high-level functional roles and introduces the tracing intuition; here we provide the corresponding staged utilities, exact activation-side recovery criterion, supplier–target tracing rule, and source-validation condition in a unified form. Importantly, this stage does not impose a fixed ActivationPatching-to-Weight-Patching pipeline. Rather, activationside and parameter-side analyses remain standalone views under the same anchor criterion, while their joint use provides a more structured mechanistic account. A.6.0.1 Why hierarchy recovery is needed.: Weight Patching identifies components whose own specialized parameters are sufficient to restore the target control representation, but this source-level evidence alone does not reconstruct the full mechanism by which instruction-conditioned control is formed, routed, and ultimately executed. In particular, a component can be highly important in activation space because it acts as a aggregation bottleneck or routing interface, even when the capability is not primarily written in that component’s own parameters. We therefore separate three functional roles: • source carriers, i.e., components whose specialized parameters causally reintroduce the target control signal; • aggregation or routing modules, i.e., components whose activations strongly restore or relay the control signal during inference; • downstream execution modules, i.e., components whose intervention most directly restores output behavior relative to the specialized model. This separation turns a flat importance ranking into a staged mechanistic account. A.6.0.2 Staged utilities for control formation and execution.: To distinguish control formation from output realization, two complementary utilities are used. For the control-formation stage, the utility is the anchor utility introduced in the main text:

Fup (M, x) = Fa (M, x).

(71)

This quantity measures whether the instruction-conditioned control representation has been formed at the anchor. For the execution stage, a behavioral utility is defined in output space by comparing the model under evaluation against the specialized model:

Fdown (M, x, y) = −DKL (psft (· | x, y<t ) ∥ pM (· | x, y<t )) . (72)

(t)

zout (M, x, y<t ) ≡ zL (x, y<t | M )

proj Fdown (M, x, y) = sim(zout (M, x, y<t ), vout ) .

(73)

(74)

This variant reuses the same vector-projection logic as the upstream metric, but moves the readout to the final layer. In practice, the projection-based and KL-based readouts yield qualitatively similar localization patterns. Equation (??) is used only for downstream execution analysis; it is not the source-level objective of the method. Whenever the projection-based variant is used, the downstream restoration effect in Eq. (64) is defined analogously by substituting proj Fdown for Fdown . A.6.0.3 Activation-side recovery of aggregation and routing modules.: Given the exact activation-side restoration effect " # f(c) , x) − Fa (Mbase , x) Fa (M base Ea (c) = Ex∼Dinst , (75) G(x) components with large Ea (c), or large activation-side firstorder scores Attract (c), are treated as candidates for aggregation and routing modules. These modules are not assumed to be source carriers. Rather, they mark locations where instruction-conditioned information becomes functionally important, is integrated from earlier suppliers, or is redirected toward later execution stages. Operationally, let Sconv = {c ∈ C : Ea (c) > τa } (76) denote the set of activation-side aggregation candidates, where τa is a threshold chosen from exact or screened activation-side validation. In practice, top-ranked components under Ea (c) or Attract (c) can also be used when a fixed-cardinality selection is preferred. A.6.0.4 Target-side need and supplier-side support.: To avoid ambiguity with the global anchor-relative upstream/downstream partition used in the main text, local component-to-component tracing is written here in supplier–target form. For a chosen target component ctar , define gneed (ctar ) = ∇Ψctar (Z) Fup (Mbase , x) (77) as the target-side need direction, where Ψctar (Z) extracts the residual-stream input to ctar . This direction describes which perturbation at the target input would most effectively increase the anchor-side control utility.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

For a candidate supplier component csup , define its weight-side support to ctar by (c

swt (ctar , csup ) =

X p∈Θ(csup )

∆θp

depending on the analysis. The corresponding downstream restoration effect is

)

tar ∂Fneed (Mbase , x) , ∂θp

(78) E(x,y)∼Deval

where (c

)

tar Fneed (M, x) = ⟨Ψctar (Z(M, x)), gneed (ctar )⟩ .

(79)

This quantity measures how strongly the post-training parameter change on csup supports the target-side need of ctar . A.6.0.5 Functional link strength.: The functional link between a target component and a candidate supplier is measured by swt (ctar , csup ) Elink (ctar , csup ) = Ex∼Dinst . (80) G(x) A large value of Elink (ctar , csup ) indicates that the posttraining parameter change on csup is well aligned with increasing the need utility of the target component ctar . This criterion is used to distinguish functionally relevant suppliers from unrelated components whose parameters changed during post-training but do not support the specific target under study. Given a target component ctar , one can therefore define a candidate supplier set

Ssup (ctar ) = {c ∈ C : Elink (ctar , c) > τlink } ,

Edown (c) = Fdown (Ic (Mbase ), x, y) − Fdown (Mbase , x, y) . Fdown (Msft , x, y) − Fdown (Mbase , x, y) (84)

(81)

where τlink is a link-strength threshold, or equivalently use the top-ranked suppliers under Elink . A.6.0.6 Source validation.: A strong supplier–target link does not by itself imply that the supplier component is a true source carrier. Some components may simply relay information that was formed even earlier, rather than storing the relevant post-training update in their own parameters. To distinguish genuine source carriers from relay nodes, candidate supplier components are further validated using exact Weight Patching. Specifically, a component is identified as a source carrier only if it satisfies both strong parameter-side restoration and strong functional linkage:

Downstream execution modules are then selected as components with strong output-space restoration, for example

Sexe = {c ∈ C : Edown (c) > τdown } .

(85)

This stage is used to characterize the final realization of behavior rather than to define the core source-localization objective. A.6.0.8 Recovered hierarchical circuit.: Combining the three stages above yields a directed hierarchical circuit representation rather than a flat component list. Let

Ssrc =

[

Ssrc (ctar )

(86)

ctar ∈Sconv

collect all validated source carriers linked to at least one traced target component. The recovered circuit set is then

Scircuit = Ssrc ∪ Sconv ∪ Sexe .

(87)

The resulting structure corresponds to a directional functional chain that begins at source-level carriers, passes through aggregation and routing interfaces, and culminates in downstream execution modules. A.6.0.9 Practical recovery procedure.: The full hierarchy-recovery pipeline can be summarized as follows.

1) Select target modules for tracing, either from activationside evidence, from Weight Patching evidence, or from prior mechanistic hypotheses. 2) For each target component ctar , compute the target-side need vector gneed (ctar ). 3) For each candidate supplier component csup , compute IsSource(csup ) ⇐⇒ Ew (csup ) > τw ∧ Elink (ctar , csup ) > τlink . its weight-side support swt (ctar , csup ) and the corre(82) sponding link strength Elink (ctar , csup ). The first condition asks whether the specialized parameters 4) Retain only those supplier candidates that also satisfy of csup are themselves sufficient to restore the missing the parameter-side source criterion in Eq. (82). control representation. The second condition asks whether 5) Use downstream behavioral recovery under Fdown to the component participates in the current supplier–target characterize the execution-stage modules Sexe when chain. Only when both conditions hold is the component needed. treated as a source-level carrier in the recovered hierarchy. This procedure yields a hierarchy rather than a single global Accordingly, for a given target component ctar , the validated ranking: source modules are supported by exact Weight source set is Patching, aggregation and routing modules are supported Ssrc (ctar ) = {c ∈ C : Ew (c) > τw ∧ Elink (ctar , c) > τlink } . by activation-side restoration, and downstream execution (83) modules are supported by output-level behavioral recovery. A.6.0.10 Summary.: Taken together, Eqs. (71)–(87) A.6.0.7 Downstream execution modules.: The final stage of the hierarchy concerns components that most di- define a staged hierarchy-recovery procedure that integrates rectly implement behavior at the output level. These are source-level parameter evidence, activation-side aggregaidentified by evaluating interventions under the down- tion analysis, supplier–target tracing, and output-level exstream utility in Eq. (72). For notational compactness, let Ic ecution analysis into a unified mechanistic account. The redenote the intervention applied to component c in the cho- sulting representation supports both standalone comparison sen evaluation protocol, which may be an exact activation- between activation-side and parameter-side localization and side intervention or an exact parameter-side intervention their joint use in a directional circuit hypothesis.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

A.7 Detailed WP-Guided Fusion Formulation and Implementation This subsection provides the full formulation of the exploratory WP-guided fusion extension. The main text gives the core component-wise weighting rule, but the implementation of the fused model is most transparent when written at the slice level and linked explicitly to the same head and neuron units used in Weight Patching. We therefore detail how exact or first-order component scores are converted into nonnegative expert weights, how those weights are normalized across experts at each component, and how the resulting coefficients are applied consistently to full head and neuron slices. A.7.0.1 Shared-base multi-expert setting.: Let Mbase denote a shared pretrained base model, and let {M (k) }K k=1 denote K expert models obtained by finetuning the same base toward different capabilities. Their parameters are denoted by {θ(k) }K k=1 , and the expert-specific parameter deltas relative to the common base are

∆θ

(k)

=θ

(k)

− θbase .

(88)

Because all experts are derived from the same base checkpoint, the differences among them can be expressed uniformly in terms of these component-wise parameter deltas. A.7.0.2 Component-level expert scoring.: For each expert model M (k) and each component c ∈ C , we compute a component score s(k) (c) that reflects how strongly expert k is supported on component c as a carrier of the target capability. In the exact setting, this score can be instantiated by the Weight Patching effect: (k) s(k) (c) = Ew (c),

(89)

(k) where Ew (c) denotes the exact source-level restoration ef-

fect obtained by patching the slice of expert k into the shared base model. In the efficient setting, the score can instead be approximated by the first-order weight attribution: (k)

s(k) (c) = Attrwt (c).

(90)

The formulation does not require the two choices to be used simultaneously; rather, either exact or first-order component scores can serve as the basis for the fusion rule depending on computational budget and granularity. A.7.0.3 Nonnegative truncation.: Since component scores can be negative, directly using them as fusion weights would allow suppressive or interfering updates to influence the merged model. To avoid this, only positive evidence is retained: s̄(k) (c) = max s(k) (c), 0 . (91) This step preserves components for which expert k receives positive support while assigning zero weight to experts whose updates on component c are estimated to be harmful or misaligned with the target behavior. A.7.0.4 Component-wise cross-expert normalization.: For each component c, the retained nonnegative scores are normalized across experts to obtain the fusion weight of expert k on that component:  PK s̄(k) (c)  (m)   PK , (c) > 0, m=1 s̄ (m) (k) s̄ (c) (92) α (c) = m=1  PK  (m) 1, s̄ (c) = 0. m=1 K

By construction, this yields K X

α(k) (c) = 1,

∀ c ∈ C.

(93)

k=1

Equation (92) has three useful limiting behaviors. When a component is strongly supported by a single expert, the weights approach a near one-hot allocation. When multiple experts have positive support on the same component, the rule reduces to local importance-weighted averaging. When no expert exhibits reliable positive evidence on that component, the rule falls back to uniform averaging. A.7.0.5 Component-wise fused model.: Let θfuse denote the parameters of the fused model Mfuse . For each component c, its fused parameter slice is defined by (c)

(c)

θfuse = θbase +

K X

α(k) (c) ∆θ(k,c) ,

(94)

(c)

(95)

k=1

where

∆θ(k,c) = θ(k,c) − θbase .

Using Eq. (93), Eq. (94) is equivalently written as (c)

θfuse =

K X

α(k) (c) θ(k,c) .

(96)

k=1

Equation (94) emphasizes the interpretation of fusion as a component-wise combination of expert deltas around a common base, whereas Eq. (96) emphasizes its interpretation as a convex combination of expert slices. A.7.0.6 Slice-level implementation.: The fusion granularity is kept strictly consistent with the component interfaces used throughout Weight Patching. For an attention head H (l,h) , the same coefficient α(k) (H (l,h) ) is applied to the full parameter slice defined in Eq. (49). For an MLP neuron N (l,j) , the same coefficient α(k) (N (l,j) ) is applied to the full slice defined in Eq. (50). Thus, the merged model is constructed at the same head-level and neuron-level structural granularity used by the mechanistic analysis, rather than by unconstrained scalar-wise mixing across the entire parameter space. A.7.0.7 Relation to uniform averaging.: A useful reference case is obtained by setting

1 , ∀ c ∈ C, k = 1, . . . , K. (97) K Under this choice, the method degenerates to componentindependent uniform averaging, which no longer distinguishes among experts according to their source-level support on different functional modules. The essential difference of the proposed fusion rule is therefore not merely that it averages parameters, but that it performs a componentaligned causal allocation of expert responsibility. A.7.0.8 Interpretation.: The component-wise coefficients in Eq. (92) make explicit the question of which expert should dominate which mechanistic unit. This differs from global averaging or globally shared task-vector combinations, where the same mixing rule is applied uniformly across the full parameter space. In the present formulation, a head or neuron is preserved from an expert model precisely when that expert receives stronger support on the corresponding localized capability carrier. As a result, the fused α(k) (c) =

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

model is designed to better preserve sparse source-level interfaces and to reduce destructive interference among experts at functionally distinct components. A.7.0.9 Practical construction.: The full fusion procedure can be summarized as follows. 1) For each expert model M (k) and each component c ∈ C , compute a component score s(k) (c) using either exact Weight Patching or first-order weight attribution. 2) Apply nonnegative truncation via Eq. (91). 3) Normalize the retained scores across experts for each component using Eq. (92). 4) Construct each fused component slice using Eq. (94) or, equivalently, Eq. (96). 5) Assemble all fused slices into the final fused model Mfuse . A.7.0.10 Summary.: Taken together, Eqs. (88)–(97) define a mechanism-aware multi-expert fusion rule aligned with the same interpretable units used throughout Weight Patching. The resulting fused model is not formed by a globally uniform parameter average, but by a structured component-wise allocation rule that reuses source-level importance signals as practical units for low-cost and interpretable multi-capability composition.

A PPENDIX B R EPRODUCIBILITY AND E XPERIMENTAL S ETUP B.1

Models, Checkpoints, and Architectures

Our mechanistic analysis is conducted on paired base/instruct models from the Llama family, including Llama-3.2-3B, Llama-3.1-8B, and Llama-2-13B. In each case, we analyze a base model together with its instruction-tuned counterpart under the same architecture, so that behavioral differences can be attributed to post-training parameter changes rather than architectural variation. We include 3B, 8B, and 13B scales to test whether the main observations of our framework remain consistent across model sizes and model generations. For the model-fusion experiments, we follow the common setup in recent merging work and use Llama-2-13B-based expert models as the shared backbone, which enables direct comparison to standard instruction, math, and code specialists [32]–[35]. These models also differ in attention parameterization, which affects the implementation of head-level parameter interventions. Llama-2-13B uses the same number of KV heads as attention heads, making head-level slicing more direct. By contrast, Llama-3.1-8B and Llama-3.2-3B adopt grouped-query attention, where KV heads are shared across multiple query heads. Accordingly, our head-level Weight Patching implementation follows the underlying architecture and uses architecture-consistent parameter slicing for each model family. Table 4 summarizes the key architectural statistics used throughout the experiments, including the number of layers, hidden size, attention heads, KV heads, MLP width, and vocabulary size. B.2

Tasks, Data Sources, and Evaluation Protocol

Our primary interpretability experiments are built on six instruction-following tasks drawn from IFEval [25]. Rather than aiming to exhaust the full benchmark, we focus on

Table 4 Architectural statistics of the model families used in our experiments. Model Llama-2-13B Llama-3.1-8B Llama-3.2-3B

Layers Hidden Dim Attention KV Heads MLP Width Vocab Size (L) (dmodel ) Heads 40 32 28

5120 4096 3072

40 32 24

40 8 8

13824 14336 8192

32000 128256 128256

a subset of tasks for which stable intermediate steering vectors can be reliably obtained under the current vectorextraction framework. In this sense, the selected tasks serve as an operational subset for mechanistic analysis: they provide sufficiently stable behavioral interfaces for localization, ablation, and restoration, while still covering diverse forms of procedural constraints, including punctuation control, title formatting, structural organization, quotation boundaries, highlighted numbered sections, and global capitalization. As discussed in the main text, our goal is not to design a stronger vector-extraction method, but to use the strongest available method as a stable interface for generative-behavior analysis [19]. For vector extraction, data usage, and preprocessing, we follow [19]. For final task evaluation on IFEval, we also follow the corresponding official evaluation protocol, and therefore do not repeat the low-level implementation details here. In addition, for the model-fusion experiments, we follow the task-specific evaluation protocols used in prior model-merging work, especially AIM and its referenced baselines [32]. Throughout the paper, this yields a consistent division between benchmark-defined end-task evaluation and our own mechanistic localization pipeline. B.3 Data Usage, Split Strategy, and Vector-Extraction Protocol For interpretability analysis, the data used to extract steering vectors and the data used for subsequent steering and localization are not shared. In the default setup, we split the available data into two halves: one half is used for steeringvector extraction, and the other half is used for downstream interpretability analysis. The forward-pass data used to produce the heatmaps in the main text follows this split strategy. We also ran an alternative setting in which vector extraction and subsequent interpretability analysis use the same data, and found that the resulting localization patterns are highly similar. This indicates that the main conclusions do not depend sensitively on the particular split choice. The task-wise sample counts used in vector extraction are 434 for english_capital, 479 for multiple_sections, 464 for number_highlighted_sections, 483 for title, 406 for no_comma, and 426 for quotation. In practice, we further observe that the localization heatmaps stabilize quickly: for the main interpretability analyses, using roughly 50 samples is already sufficient for the heatmap pattern to stop changing in any meaningful way. Increasing the sample size beyond this point has little effect on the qualitative conclusions. This empirical observation supports both the efficiency and the stability of the proposed localization framework. Since our focus is not on improving vector extraction itself, all vector-related procedures are kept fully aligned with [19],

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

No Comma Title Multiple Sections Quotation Number Highlighted Sections English Capital

250 200

No Comma Title Multiple Sections Quotation Number Highlighted Sections English Capital

160 140

Correction rate (%)

300

150 100 50

120 100 80 60 40 20

0 0

Layer

(a) Llama-3.1-8B Model

Layer

(b) Llama-2-13B Model

Figure 17. Layer-wise steering correction rates for Llama-3.1-8B and Llama-2-13B. This figure complements Fig. 5 in the main text by showing the steering effects of task vectors across all layers for the 8B and 13B models on six representative IFEval tasks. Similar to the 3B model, optimal instruction-following recovery consistently peaks in the mid-to-late layers across both larger models. This demonstrates the cross-model stability of the vector-anchor interface and confirms that instruction-conditioned control representations are reliably formed after early token-level processing.

0.4

Layer

0.0

0.2

25 30

0.4 0

Head

5 10

Layer

0.2

15 20 25 30

0.50.0 0.5

Neuron AVG

0.04

0.4

0.02

0.2

0.00

0.0

0.02

0.2

0.04 0

(a) Activation Patching

Head

0.50.0 0.5

0.4

Neuron Avg

Count

3 2 1 0 0

Layer

(a) Key Head Distribution

1 2 0.50.0 0.5

0 1

0.2 0

Neuron AVG

0.0

Head

0.4

2 0.50.0 0.5

Neuron Avg

(b) Weight Patching

Figure 20. Heatmaps of component importance under activation patching and Weight Patching on the English Capital task on Llama-2-13B.

Activation Patching Weight Patching Shared

Count

30 25 20 15 10 5 0 0

Count

0.2

(a) Activation Patching

Activation Patching Weight Patching Shared

Head

0.4

(b) Weight Patching

Figure 18. Heatmaps of component importance under activation patching and Weight Patching on the English Capital task on Llama-3.1-8B.

0 5 10 15 20 25 30 35

Layer

1.5 1.0 0.5 0.0 0.5 1.0 1.5

Activation Patching Weight Patching Shared

Layer

(b) Key Neuron Distribution

15 5

5 10 15 20 25 30 35

Layer

(a) Key Head Distribution 10

Activation Patching Weight Patching Shared

20 10

1 0 0

Count

0 5 10 15 20 25 30 35

Layer

and we leave the detailed discussion of vector-construction methodology to that work.

0 0

5 10 15 20 25 30 35

Layer

(b) Key Neuron Distribution

Figure 21. Layer distributions and overlaps of top-ranked components identified by activation patching and weight patching on Llama-2-13B.

Figure 19. Layer distributions and overlaps of top-ranked components identified by activation patching and weight patching on Llama-3.1-8B .

B.4

Hyperparameters and Implementation Details

These settings are used consistently across the main interpretability experiments unless otherwise noted. Unless otherwise specified, vector extraction follows [19] with a batch size of 10. For steering, we use projection-based intervention, with batch sizes of 50 for the 3B model, 20 for the 8B model, and 20 for the 13B model. The default analysis layer is selected according to steering performance; in our main setup, the default layer is 17 for 3B, 20 for 8B, and 27 for 13B. For patching, the batch size is 10 for 3B, 10 for 8B, and 1 for 13B. The implementation of parameter slicing is kept consistent with the underlying attention architecture of each model family. For knockout and restore experiments, we use a batch size of 10. In the restore radar plots, the restored fraction for

heads and neurons increases from 1% to 5% with a step size of 1%. In the joint AP+WP restoration setting, the restored fraction increases from 2% to 10% with a step size of 2%. In the attention-ratio analysis, we use the top-10 heads when comparing the average attention mass assigned to instruction and non-instruction regions. In the WP–WtP efficiency comparison, intersections are computed using Top-20 heads and Top-10 MLP layers. In the head-to-neuron analysis, we first select the Top-20 heads and then retrieve the Top-200 upstream neurons for each head, where neuron rankings are computed by absolute value. All generations used in the interpretability experiments are decoded with do_sample=False and max_new_tokens=512. For model-fusion evaluation, inference uses the default automatic batch scheduling provided by vLLM. All experiments are conducted on three NVIDIA GeForce RTX 4090 GPUs.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Table 5 Vocabulary projection results of task vectors from shallow to deep layers across six tasks in the Llama-3.2-3B model. Task No Comma

Layer 3 10 15 20 25

Projection (Top-10 vocabulary) <wear>, <adam>, <Dipl>, <cca>, <McCorm>, <<U+FFFD>>, <Synd>, <least>, <dex>, <ook> <-horizontal>, <ippi>, <Artem>, <<U+FFFD>>, <Cap>, <naked>, <urs>, <<U+FFFD>>, <imuth>, <eni> <gov>, <tv>, <repeat>, <nge>, <尖>, <alien>, <enez>, <U+0626>, <dont>, <око> <moreover>, <τί>, <(’.’)>, <hàng>, <doesnt>, <furthermore>, <dont>, <Dont>, <auen>, <fine> <youre>, <doesnt>, <its>, <theres>, <its>, <Dont>, <Lets>, <Its>, <dont>, <hey>

Title

3 10 15 20 25

Multiple Sections

3 10 15 20 25

Quotation

3 10 15 20 25

Number Highlighted Sections

3 10 15 20 25

<卒>, <adam>, <rch>, <doit>, <plet>, <Americ>, <wear>, <<U+FFFD>>, <holm>, <importantly> <seriousness>, <unter>, <Holy>, <aket>, <Matcher>, <Holy>, <ersh>, <orra>, <eller>, <unker> <Guides>, <,>, <Introduction>, <cord>, <section>, <Pil>, <chal>, <Ot>, <ulla>, <Technologies> <_>, <**>, <###>, <##>, <**>, <###>, <*_>, <Introduction>, <Welcome>, <*> <*>, <*>, <*_>, <*_>, <_*>, <,*>, <*,>, <.*>, <(*>, <-*>

English Capital

3 10 15 20 25

Figure 22. Causal validation of Llama-3.2-3B upstream and downstream modules on the Title task.

B.5 Ranking, Validation, and Component Selection Rules

the main reasons why attribution-based approximations are necessary for fine-grained large-scale analysis.

AAP and WAP are used directly for interpretability analysis and module localization, rather than as a preliminary screening step followed by manual top-k curation. In contrast, exact Weight Patching is substantially more expensive, and is therefore applied mainly at the attentionhead level. Exact neuron-level WP is only run on a limited set of cases for efficiency comparison, as reported in the main-text efficiency table. This computational gap is one of

Unless otherwise noted, the rankings used in heatmaps, layer distributions, and bar-chart summaries are all based on normalized scores. For restore and knockout experiments, the component sets are constructed by directly taking the top-ranked heads or neurons under the corresponding scoring rule, without additional filtering or reranking. In the joint AP+WP restoration experiments, the restored component set is defined as the union of the top-ranked modules

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Table 6 Vocabulary projection results of task vectors from shallow to deep layers across six tasks in the Llama-3.1-8B model. Task

No Comma

Layer 3 10 15 20 25 30

Projection (Top-10 vocabulary) <eca>, <ientes>, <holm>, <airport>, <ency>, <ᄃ ᆨ>, <elter>, <harma>, <hsi>, <Integral> ᅩ <enen>, <nge>, <bero>, <sic>, <458>, <.usermodel>, <vester>, <ome>, <alah>, <Wr> <;amp>, <129>, <<U+2510>>, <Hammer>, <ober>, <stra>, <geme>, <<U+0E46>>, <aram>, <ρίζ> <engu>, <some>, <alternate>, <first>, <rita>, <oko>, <Lomb>, <olut>, <lam>, <engo> <arto>, <folk>, <undi>, <oko>, <U+0627U+0644U+0646U+0627U+0633>, <stat>, <KEN>, <itemprop>, <ifications>, <νω> <we>, <oh>, <okay>, <one>, <Nicholson>, <first>, <народ>, <the>, <okay>, <an>

Title

3 10 15 20 25 30

<ープ>, <reater>, <foon>, <bane>, <levance>, <itsu>, <imest>, <fle>, <ingleton>, <rame> <сут>, <iffe>, <ancel>, <ỳ>, <andin>, <ieran>, <zd>, <adr>, <ubbo>, <ADR> <ěž>, <MAND>, <’gc>, <ANTA>, <Disposed>, </Branch>, <)prepare>, <EdgeInsets>, </crypto>, <anja> <луги>, <addCriterion>, <《>, <ait>, <ello>, <『>, <loser>, <olare>, <;>, <DCALL> <«>, <«>, <»>, <adel>, <надлеж>, <stick>, <sett>, <addCriterion>, <»>, <芝> <«>, <«>, <)«>, <]«>, <«>, <"«>, <()«>, <»,>, <"«>, <«>

Multiple Sections

3 10 15 20 25 30

Quotation

3 10 15 20 25 30

Number Highlighted Sections

3 10 15 20 25 30

English Capital

3 10 15 20 25 30

identified by the two methods. These rules ensure that all reported restoration and ablation results are derived from a fixed and reproducible protocol rather than post hoc module selection.

B.6 Implementation Settings for WP-Guided Model Merging In the model-fusion experiments, we apply the proposed WP-guided component-wise merging to a set of Llama-213B-based expert models (e.g., instruction, math, and code specialists) using the pretrained Llama-2-13B as the shared base model. The detailed implementation settings and hyperparameter choices for the merging pipeline are outlined below. B.6.0.1 Score Processing and Normalization.: To compute the fusion weights α(k) (c) for attention heads and MLP neurons, the raw attribution scores extracted from each expert are first passed through a ReLU activation (clamped at a minimum of 0.0) to filter out negative interference. The non-negative scores are then normalized across the K expert models. If the sum of scores for a specific component across all experts is strictly zero, the merging rule automatically falls back to a uniform distribution (i.e., assigning a weight 1 of K to each expert) to ensure parameter integrity. B.6.0.2 Component-Level Mapping Rules.: The normalized fusion weights are applied to the parameter ma-

trices of the Transformer layers based on the following mapping rules: sub-layers: Head-level fusion weights are explicitly broadcast to the query (WQ ) and output (WO ) projections. For architectures utilizing GroupedQuery Attention (GQA), the weights for the key (WK ) and value (WV ) projections are obtained by averaging the fusion weights of the corresponding query heads within each KV group. • MLP sub-layers: Neuron-level fusion weights are directly broadcast to the corresponding rows or columns of the gate, up, and down projections (Wgate , Wup , Wdown ). • Attention

B.6.0.3 Fallback Strategy for Structure-Agnostic Parameters.: Parameters that do not logically decompose into head or neuron components—specifically token embeddings, layer normalizations (e.g., RMSNorm), and the final language modeling head—are merged via uniform parameter averaging across all K experts. During this step, we account for potential vocabulary size mismatches caused by different tokenizer expansions during domain-specific supervised fine-tuning (SFT). If a shape mismatch is detected, we dynamically align the embedding matrices to the base model’s original shape by truncating the expanded dimensions prior to averaging. B.6.0.4 Hardware and Precision Settings.: To manage the high memory demands of merging multiple 13B-

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Figure 23. Causal validation of Llama-3.1-8B upstream and downstream modules on the Title task.

Figure 24. Causal validation of Llama-2-13B upstream and downstream modules on the Title task.

Head Knockout

Head Restore

0.4

Accuracy

Top-K

0.2

0.1

0.3

0.2

Neuron Restore

0.4

0.3

0.0

Neuron Knockout

0.1

0.1 0

Top-K

Activation Patching

0.0

200

400

Weight Patching

600

Top-K

800

1000

Random Select

200

400

600

Top-K

800

1000

Figure 25. Causal validation of Gemma2-2B global modules on the English Capital task.

scale models simultaneously, the state dictionaries of all expert models are loaded sequentially onto CPU memory, and the model objects are immediately garbage-collected to minimize peak RAM usage. The checkpoints are initially loaded in half-precision (FP16). During the layer-wise finegrained fusion, the weight slices are temporarily cast to FP32 for the weighted accumulation step to prevent numerical underflow and precision loss. Upon completion of a layer’s merge, the resulting parameters are cast back to FP16, ensuring that the final fused model remains memory-efficient

for downstream deployment.

A PPENDIX C E XTENDED E XPERIMENTAL R ESULTS AND A DDI TIONAL A BLATIONS C.1

Full Results for Vector-Anchor Validation

In Section 4.2 of the main text, we validated the vectoranchor interface using Llama-3.2-3B, demonstrating that injecting task vectors into mid-to-late layers yields the

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Table 7 Vocabulary projection results of task vectors from shallow to deep layers across six tasks in the Llama-2-13B model. Task

Layer 3 10 15 20 25 30 35

No Comma

Projection (Top-10 vocabulary) <prob>, <str>, <bon>, <mang>, <Bon>, <ling>, <aud>, <ieu>, <Kra>, <istan> <egy>, <Gram>, <firm>, <clam>, <haft>, <rails>, <Gal>, <WF>, <Wars>, <Sequ> <comb>, <zero>, <emb>, <connected>, <cord>, <succession>, <port>, <God>, <pair>, <esch> <zero>, <neg>, <without>, <Jordan>, <no>, <ash>, <sans>, <Without>, <single> <single>, <without>, <un>, <neg>, <:>, <,>, <negative>, <Single> <none>, <without>, <single>, <,>, <neg>, <None>, <acc>, <ats>, <tall>, <no> <single>, <none>, <None>, <Single>, <plain>, <none>, <FT>, <lack>, <absence>, <single>

Title

3 10 15 20 25 30 35

Multiple Sections

3 10 15 20 25 30 35

Quotation

3 10 15 20 25 30 35

Number Highlighted Sections

3 10 15 20 25 30 35

English Capital

3 10 15 20 25 30 35

Head Knockout

0.4

Head Restore

0.24

Neuron Knockout

0.4

0.22

0.3

0.22

0.2

0.20

0.2

0.20

Accuracy

0.3

0.1 0.0

0.1

0.18 0

Top-K

Activation Patching

0.0

Neuron Restore

0.24

0.18 0

200

Weight Patching

400

600

Top-K

800

1000

Random Select

200

400

600

Top-K

800

1000

Figure 26. Causal validation of Mistral-7B-v0.3 global modules on the English Capital task.

highest correction rates. To verify that this intermediate representation is a consistent mechanistic property rather than an artifact of a specific model scale, Fig. 17 presents the corresponding layer-wise steering correction rates for the larger Llama-3.1-8B and Llama-2-13B models across the same six IFEval tasks. Consistent with our observations on the 3B model, the

correction rate peaks consistently emerge in the mid-to-late layers across both the 8B and 13B architectures, while early lexical layers show minimal recovery. This cross-model stability further corroborates our conclusion in the main text: the extracted task vector is not merely a surface-level textual proxy, but a robust intermediate control representation. The consistency of these anchor locations justifies our use of a

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Head Knockout Accuracy

0.3 0.2 0.1 0.0

Top-K

Head Restore

0.475 0.450 0.425 0.400 0.375 0.350 0.325

0.4

Neuron Knockout 0.3 0.2 0.1

Top-K

Activation Patching

0.0

200

400

Weight Patching

600

Top-K

800

Neuron Restore

0.475 0.450 0.425 0.400 0.375 0.350 0.325

0.4

1000

Random Select

200

400

600

Top-K

800

1000

Figure 27. Causal validation of Qwen2.5-3B global modules on the English Capital task.

Figure 28. Vocabulary projection results of downstream neurons in the Llama-3.2-3B model, identified via weight patching across six tasks.

Attention Ratio of Instruction Prompt (%)

Llama-3.1-8B Model 4

Instruct (Top-K)

Base (Top-K)

Instruct (Random)

3 2 1 0

No Comma

Title

Multiple Sections

Quotation

Number Highlighted Sections English Capital

Attention Ratio of Instruction Prompt (%)

Llama-2-13B Model 4

Instruct (Top-K)

Base (Top-K)

Figure 30. Overlap of critical modules across tasks on Llama-3.1-8B.

Instruct (Random)

3 2 1 0

No Comma

Title

Multiple Sections

Quotation

Number Highlighted Sections English Capital

Figure 29. Selective attention of critical heads to instruction tokens.

shared internal criterion for subsequent Weight Patching (WP) and Activation Patching (AP) analyses across different models in the Llama family. C.2 Extended Results on Vocabulary Projection of Task Vectors In Section 4.2 of the main paper, we demonstrated that the extracted task vectors serve as a stable, intermediate behavioral interface for instruction following, rather than acting as a fragile, text-level proxy. To substantiate this claim, we discussed the vocabulary-space projection of these vectors. Due to space constraints in the main text, we provide the comprehensive, layer-wise projection results for all six representative IFEval tasks in this section. Tables 5, 6, and 7 display the top projected tokens from shallow to deep layers for Llama-3.2-3B, Llama-3.18B, and Llama-2-13B, respectively. As illustrated in these tables, projecting the intermediate steering vectors into the

Figure 31. Overlap of critical modules across tasks on Llama-2-13B.

vocabulary space offers only limited interpretability. While a few structurally explicit tasks occasionally exhibit partially recognizable, format-related cues (e.g., casing-related tokens in English Capital or punctuation-related tokens in No Comma), the projections for the majority of the tasks and layers remain diffuse, abstract, and largely disconnected from the explicit textual templates of the given prompts. Crucially, this phenomenon remains highly consistent across different model scales (3B, 8B, and 13B). This robust cross-scale evidence reinforces our conclusion in the main text: the extracted task vector is not merely a surface residue of the prompt memorized in the vocabulary space. Instead, it functions as a deeply embedded, abstract control representation formed after early token-level processing. This validation ensures that the vector anchor reliably reflects the presence of instruction-conditioned control, forming the foundation for our mechanistic comparisons between Activation Patching (AP) and Weight Patching (WP).

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

C.3 Extended Localization Heatmaps across Model Scales Attention Submodules Decomposition

shallow source-level parameter carriers, while AP highlights middle-layer routing and aggregation bottlenecks. To verify that this structural division of labor is a general mechanistic property rather than an artifact of a specific model scale, we extend this distributional analysis to larger models.

0.05 0.04 0.03

q_proj k_proj

0.01 0.00

v_proj o_proj

Attention Overall

MLP Submodules Decomposition

v_proj o_proj

Attention Overall

0.10 0.08 0.06 0.04 0.02

0.05

0.00

0.04 0.03 0.02

gate_proj up_proj

0.01 0

down_proj 10

MLP Overall 15

Layer

Relative Change

q_proj k_proj

0.12

0.02

0.00

Attention Submodules Decomposition

0.14

Relative Change

0.06

Figure 32. Layer-wise parameter changes induced by instruction tuning on Llama-3.1-8B.

To verify that the source–aggregation separation observed in Section 4.3 is not idiosyncratic to the Llama-3.23B architecture, we extend our component-level localization analysis to larger scales. Figures 18 and 20 present the component importance heatmaps under both Activation Patching (AP) and Weight Patching (WP) for Llama-3.18B and Llama-2-13B, respectively, evaluated on the English Capital task. Consistent with the findings in the main text, the mechanistic division of labor remains structurally stable across different model sizes. In both the 8B and 13B models, Activation Patching (AP) predominantly highlights attention heads and adjacent neurons in the middle-to-late layers. This corroborates the interpretation that these components act as aggregation bottlenecks or routing hubs, whose internal states become critical once the instruction-conditioned information is already propagating during inference. Conversely, Weight Patching (WP) strongly isolates parameter subsets within shallow-layer MLP neurons across both models. This distinct shift toward early layers reaffirms that the parameter weights most responsible for regenerating the control representation—the source-level capability carriers—are fundamentally distinct from the routing modules. Taken together, these supplementary heatmaps provide robust evidence that the hierarchical separation between parameter sources (revealed by WP) and activation routers (revealed by AP) is a general structural property of instruction-tuned LLMs, generalizing reliably from 3B up to 13B parameters. C.4 Source–aggregation Separation Across Model Scales In Sec. 4.3 of the main text, we demonstrated a systematic spatial misalignment between the components localized by Activation Patching (AP) and Weight Patching (WP) on the Llama-3.2-3B model. This misalignment supports a “source– aggregation separation” hypothesis, wherein WP identifies

0.07 0.06 0.05 0.04 0.03 0.02 0.01 0.00

MLP Submodules Decomposition

gate_proj up_proj 0

down_proj 10

MLP Overall 20

Layer

Figure 33. Layer-wise parameter changes induced by instruction tuning on Llama-2-13B.

Figures 19 and 21 illustrate the layer-wise distributions and intersection sets of the top-ranked neurons identified by AP and WP on the English Capital task for Llama-3.1-8B and Llama-2-13B, respectively. Consistent with the findings in the 3B model, the distributions remain highly disjoint across both larger scales. Specifically, WP consistently localizes the critical parametric changes to shallow layers, reflecting the early formulation of the control signal. In contrast, AP predominantly highlights neurons in the middle-to-late layers, where the instruction-conditioned information is already actively routed and integrated. Furthermore, the direct overlap between the top-k components identified by the two intervention methods remains remarkably limited in both the 8B and 13B models. These supplementary results confirm that the observed source– aggregation separation generalizes robustly across different model sizes within the Llama family, reflecting a fundamental architectural regularity in how instruction-tuned LLMs organize procedural control during generation. C.5

Additional Causal Validation Results

In Section 4.4 of the main text, we established a hierarchical division of labor—source-level carriers, aggregation/routing interfaces, and downstream execution units— by analyzing the causal roles of modules on the English Capital task using Llama-3.2-3B. To demonstrate that this source–aggregation–execution hierarchy is a robust mechanism rather than an artifact of a specific task or architecture, this section presents supplementary causal validation results along two distinct dimensions: cross-task generalization and cross-model generalization. Cross-Task and Cross-Scale Generalization. First, we verify whether the hierarchical organization generalizes to other instruction-following behaviors, and whether this

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Llama-3.2-3B

Llama-3.1-8B

Title

0.91

Llama-2-13B

Title

0.93

No Comma

Title

No Comma

0.85

No Comma

0.86 0.65 0.97 0.75 0.68 0.69 0.52 0.78 0.55 0.56 0.51 Multiple Sections Multiple Sections Multiple Sections 0.52 0.39 0.59 0.70 0.87 0.68 0.36 0.37 0.34 0.56 0.70 0.54 0.35 0.26 0.39 0.42 0.52 0.41 0.18 0.17 0.19 0.13 0.17 0.20 0.28 0.35 0.27 0.14 0.17 0.14 0.16 0.32 0.48 0.64 0.79AVG 0.15 0.30 0.45 0.60 0.75AVG 0.12 0.24 0.36 0.48 0.60AVG 0.20 0.19 0.13 0.39 0.37 0.26 0.18 0.17 0.19 0.14 0.12 0.08 0.59 0.56 0.40 0.34 0.29 0.16 0.78 0.75 0.53 0.36 0.38 0.24 0.98 0.93 0.66 0.51 0.43 0.23 Quotation Quotation Quotation 0.54 0.57 0.36 0.67 0.57 0.31 0.71 0.75 0.48 0.84 0.71 0.39 0.73

English Capital 0.89 Number Highlighted Sections

Base Model (Vanilla)

English Capital 0.94 Number Highlighted Sections

Base Model (Activation Steering)

English Capital 0.60 Number Highlighted Sections

Instruct Model (Vanilla)

Figure 34. Absolute performance comparison (IFEval strict accuracy) across three model scales: Llama-3.2-3B, Llama-3.1-8B, and Llama-2-13B. The radar charts illustrate the performance of the Vanilla Base Model (brown), the Vanilla Instruct Model (blue), and the Base Model with optimal layer-wise activation steering (green). By injecting the extracted task vector, the base model successfully recovers, and in some cases slightly exceeds, the instruction-following capabilities of the fully fine-tuned instruct model across all six representative tasks. Table 8 The intersection rate of neurons found by directly applying WP versus tracing WP from the top-20 heads identified by AP (Top-5000 Neurons). Metrics include the maximum overlap (Max), the average of the top-5 overlaps (Top-5 Avg), and the average overlap across all 20 heads (Top-20 Avg). Task

Max Top-5 Avg Llama-3.2-3B English Capital 0.761 0.703 Multiple Sections 0.403 0.380 Number Highlighted Sections 0.409 0.388 Title 0.495 0.446 No Comma 0.415 0.354 Quotation 0.696 0.620 Avg 0.530 0.482 Llama-3.1-8B English Capital 0.746 0.676 Multiple Sections 0.550 0.484 Number Highlighted Sections 0.392 0.322 Title 0.533 0.486 No Comma 0.613 0.551 Quotation 0.639 0.533 Avg 0.579 0.509 Llama-2-13B English Capital 0.635 0.595 Multiple Sections 0.446 0.431 Number Highlighted Sections 0.421 0.399 Title 0.494 0.463 No Comma 0.539 0.520 Quotation 0.595 0.573 Avg 0.522 0.497

Top-20 Avg 0.480 0.288 0.286 0.337 0.247 0.474 0.352 0.424 0.358 0.247 0.346 0.401 0.403 0.363 0.400 0.356 0.296 0.370 0.373 0.411 0.368

phenomenon remains consistent across different model capacities. To do so, we evaluate the Title task across three different scales of the Llama family: Llama-3.2-3B, Llama3.1-8B, and Llama-2-13B. As illustrated in Figures 22, 23, and 24, we perform knockout (ablation) and restoration interventions on the top upstream and downstream modules identified by Activation Patching (AP) and Weight Patching (WP), compared against a random baseline. The results strictly corroborate the patterns observed in the main text across all three model sizes: Upstream Modules (Source vs. aggregation): Across the

3B, 8B, and 13B models, the contrast between upstream heads and neurons remains distinct. Knocking out a small set of top AP-ranked upstream heads severely degrades task performance, yet restoring these same heads yields minimal recovery. This aligns with the conclusion that these middlelayer heads act as aggregation or routing bottlenecks—they are necessary for signal transmission but insufficient to generate the control representation from the base model. In contrast, WP-ranked upstream neurons show both pronounced degradation under knockout and significantly stronger recovery under restoration, reaffirming their role as genuine source-level parameter carriers of the instructionconditioned signal regardless of model scale. Downstream Modules (Execution stage): Consistent with the main findings and across varying parameter counts, downstream heads exhibit weak restorative capacity. Downstream neurons, however, demonstrate strong necessity and substantial sufficiency. Knocking out the topranked downstream neurons triggers an abrupt collapse in performance, while restoring them recovers a dominant fraction of the target behavior. This further substantiates the execution-stage hypothesis: once the abstract control representation is routed forward, downstream neurons serve as the primary units responsible for translating it into concrete textual outputs. Overall, the targeted ablation and restoration trajectories on the Title task mirror the exact functional signatures found in the English Capital task. The consistency of these results across 3B, 8B, and 13B parameter scales provides robust evidence for both task-level and scale-level generalization. Cross-Model Generalization. Beyond varying the task and scaling parameters within a single lineage, it is crucial to verify whether this mechanistic division of labor is a fundamental property of modern LLMs or merely an idiosyncrasy of the Llama architecture. To this end, we extend our causal validation to three other widely used open-weight model families: Gemma2-2B, Mistral-7B-v0.3, and Qwen2.5-3B. Figures 25, 26, and 27 present the global causal validation trajectories for these models on the English Capital task. For this macroscopic analysis, we aggregate modules globally

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

[User Instruction] "Write a rap about a new smartphone. Write it in all capital letters." Llama3.2-3B-Instruct LISTEN UP, Y'ALL, I GOT A NEW PHONE TO SHOW\nIT'S THE LATEST MODEL, THE BEST OF THE BEST, YOU KNOW\nIT'S GOT A 6.7 INCH SCREEN ...

+ Upstream Head Knockout by AP <Instruction Ignoring> Here's a rap about a new smartphone:\n\n[Intro music plays, followed by a smooth flow]\n\nYo, listen up, I got a new phone to show\nIt's sleek, it's sexy, and it's ready to go\nWith a 6.1-inch screen and a powerful processor\nThis phone is the latest and greatest, ...

+ Upstream Neuron Knockout by AP

+ Upstream Head Knockout by WP + Upstream Neuron Knockout by WP

ALRIGHT ALRTENS, LISTEN UP, I'VE GOT A RAP TO SPIT\nIT'S ABOUT THE NEW PHONE, IT'S THE ONE I'VE BEEN WAITING FOR\nIT'S GOT THE CAMERA, IT'S GOT THE SCREEN, IT'S GOT THE POWER\nIT'S THE ONE I'VE BEEN DROPPING, IT'S THE ONE I'VE BEEN CHOOSING\n\nIT'S THE NEW PHONE, IT'S THE ONE I'VE BEEN ...

OKAY, HERE IS A RAP ABOUT A NEW SMARTPHONE, WRITTEN IN ALL CAPITAL LETTERS:\n\nINTRO:\nI JUST GOT MY HANDS ON THE LATEST MODEL, IT'S A BEAUTY TO SEE\nA NEW SMARTPHONE THAT'S GOT ALL THE FEATURES I NEED\n\nVERSE 1:\nIT'S GOT A BIG SCREEN, IT'S GOT A FAST PROCESSOR TOO\nCAMERA IS TOP-NOTCH, ...

Sure, here's a fun and fast-paced rap about a new smartphone:\n\nNew Phone, New Life\nBatteries, cameras, and more\nSnap, share, and play\nAll in one device\n\nGigabytes, apps, and games\nIt's the ultimate gadget\nFrom calls to texts\nTo social media and more ...

Llama3.2-3B-Base I got a new smartphone, it’s the latest and greatest.\nIt’s got all the features that I need, and it’s got a great camera.\n ...

+ Upstream Head Restore by AP <Instruction Ignoring> I got a new smartphone, it’s the latest and greatest.\nIt’s got all the features that I need, and it’s got a great camera.\nIt’s got a big screen, and it’s got a fast processor.\nIt’s got a lot of storage, and it’s got a great battery life.\nIt’s got a lot of apps, and it’s got a lot ...

+ Upstream Neuron Restore by AP <Instruction Ignoring> AI got a new smartphone, it’s the latest and greatest.\nIt’s got all the features that I need, and it’s got a great camera.\nIt’s got a big screen, and it’s got a lot of storage.\nIt’s got a fast processor, and it’s got a great battery life.\nIt’s got a sleek design, ...

+ Upstream Head Restore by WP

+ Upstream Neuron Restore by WP

<Instruction Following> ALRIGHT ALRTENS, LISTEN UP, I'VE GOT A RAP TO SPIT\nIT'S ABOUT THE NEW PHONE, IT'S THE ONE I'VE BEEN WAITING FOR\nIT'S GOT THE CAMERA, IT'S GOT THE SCREEN, IT'S GOT THE POWER\nIT'S THE ONE I'VE BEEN DROPPING, IT'S THE ONE I'VE BEEN CHOOSING\n\nIT'S THE NEW PHONE, IT'S THE ONE I'VE BEEN ...

<Instruction Following> I’M A NEW SMARTPHONE AND I’M THE BEST PHONE YOU’VE EVER SEEN. I’M A NEW SMARTPHONE AND I’M THE BEST PHONE YOU’VE EVER SEEN. I’M A NEW SMARTPHONE AND I’M THE BEST PHONE YOU’VE EVER SEEN ...

Figure 35. Case study of Llama-3.2-3B on the Englist Capital task.

across all layers without the upstream/downstream split. The results demonstrate a remarkably consistent functional dichotomy across all three architectures: Taken together, these supplementary evaluations confirm that the fundamental mechanistic organization identified in the main text—where sparse MLP neurons act as the genuine parameter-level sources and executors of instruction-conditioned control, while attention heads function as routing bottlenecks—is a general architectural property shared by diverse tasks, scales, and modern instructiontuned LLMs.

the tokens activated by these deep neurons are directly associated with the final formatting, stylistic, or structural constraints of the respective prompts (e.g., capitalizationrelated tokens for the English Capital task). This strong alignment with output-space behavior provides direct visual corroboration for our causal and finegrained analyses. It confirms that the specific parameter subsets localized by WP in the later layers are indeed responsible for the final lexical instantiation of the instructionconditioned control, further validating the proposed source– aggregation–execution hierarchy.

C.6 Vocabulary-Space Projection of Downstream Neurons

C.7 Cross-Scale Validation of Selective Attention in Critical Heads

In Section 4.4 and Section 4.5 of the main text, we established a hierarchical division of labor for instruction-following mechanisms: while upstream neurons act as source-level carriers and middle-layer heads serve as routing bottlenecks, downstream neurons function primarily as execution units that translate abstract control representations into concrete output behavior. To provide qualitative evidence for this execution role, we further investigate the semantic properties of the late-layer neurons localized by Weight Patching (WP). Specifically, we project the weights of the top-ranked downstream neurons identified by WP on the Llama-3.2-3B model into the vocabulary space. By examining the tokens that maximally align with these neuron weights, we can visualize their functional semantics as word clouds. As shown in Fig. 28, the vocabulary-space projections of these downstream neurons exhibit highly recognizable, task-specific lexical patterns across different instructionfollowing tasks. Unlike the intermediate vector anchors (discussed in Sec. 4.2), which often remain diffuse and abstract,

In Section 4.5 of the main text, we demonstrated that the critical attention heads identified in the Llama-3.2-3B model exhibit a strong selective preference for instruction-bearing tokens, supporting their functional role as reading and routing interfaces. To verify that this fine-grained functional signature is not an artifact of a specific model size but a general mechanistic property, we extend this attention allocation analysis to larger model scales (Llama-3.1-8B and Llama-2-13B), as presented in Fig. 29. Following the same protocol used for the main results, we first localize the top-k critical attention heads for different instruction-following tasks using Weight Patching (WP). To quantify their reading behavior, we compute the ratio of the average attention weight per token allocated to the explicit instruction regions versus the non-instruction regions. We compare this selective attention ratio across three settings: (1) the critical heads in the instruction-tuned model, (2) the structurally matched heads in the corresponding base model, and (3) randomly sampled heads in the instructiontuned model.

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

[User Instruction] Could you give me a short summary of The Lord of the Rings that is child-friendly? The answer should have a title in double angular brackets, i.e. <<title>>." Llama3.2-3B-Instruct <<The Magical Adventure of Frodo and Friends>>\n\n\nA long time ago, in a magical land called Middle Earth, there was a hobbit ...

+ Upstream Head Knockout by AP <Instruction Ignoring> Here's a summary of The Lord of the Rings that's suitable for kids:\n\nThe story begins in the Shire, where hobbits live in peace and harmony. The main character, Frodo, inherits a magical ring from Bilbo Baggins that makes him the target of the dark lord Sauron. Sauron's power is so great ....

+ Upstream Neuron Knockout by AP <Instruction Following> Here is a summary of The Lord of the Rings that's suitable for kids:\n\n<<Summary of The Lord of the Rings for Kids>>\n\nThe Lord of the Rings is a story about a hobbit named Frodo who lives in a magical land called Middle-earth. Frodo's uncle, Bilbo, has a special ring that makes ...

+ Upstream Head Knockout by WP + Upstream Neuron Knockout by WP <Instruction Following> <<The Lord of the Rings Summary for Kids>> \n\nIn a magical land called Middle-earth, a hobbit named Frodo Baggins inherits a very special ring from his uncle. The ring is very powerful and belongs to a dark lord named Sauron. Sauron wants to get the ring back, and he's not ...

<Instruction Ignoring> Here's a summary of The Lord of the Rings that's suitable for kids:\n**A Kid-Friendly Summary of The Lord of the Rings**\n\nIn a magical land called Middle-earth, a hobbit named Frodo Baggie loves to garden and grow yummy food. But he also loves to go on adventures!\n\n ...

Llama3.2-3B-Base The Lord of the Rings is a trilogy of fantasy novels written by J.R.R. Tolkien. The story is set in a fictional world called ...

+ Upstream Head Restore by AP <Instruction Ignoring> The Lord of the Rings is a trilogy of fantasy novels written by J.R.R. Tolkien. The story is set in a fictional world called Middle-earth, which is inhabited by various races of elves, dwarves, men, and hobbits. The main character is a hobbit named Frodo Baggins, who is tasked with ...

+ Upstream Neuron Restore by AP <Instruction Ignoring> The Lord of the Rings is a trilogy of fantasy novels written by J.R.R. Tolkien. The story is set in a fictional world called Middleearth, which is inhabited by various races of beings, including Elves, Dwarves, Men, and Hobbits. The story begins with the Hobbit Bilbo Baggins, who is chosen ...

+ Upstream Head Restore by WP <Instruction Ignoring> The Lord of the Rings is a trilogy of fantasy novels written by J.R.R. Tolkien. The story is set in a fictional world called Middleearth, which is inhabited by various races of beings, including Elves, Dwarves, and Men. The main character is a Hobbit named Frodo Baggins, who is tasked with ...

+ Upstream Neuron Restore by WP <Instruction Following> <<summary>> The Lord of the Rings is a trilogy of fantasy novels written by J.R.R. Tolkien. The story is set in the fictional world of Middle-earth and follows the adventures of a group of characters as they struggle against the Dark Lord Sauron and his forces of evil ...

Figure 36. Case study of Llama-3.2-3B on the Title task.

As illustrated in Fig. 29, the strong selective preference consistently holds across both 8B and 13B models. The critical heads in the instruction-tuned models allocate substantially more attention to the instruction tokens compared to the baselines. In contrast, the exact same heads in the base models and the random heads exhibit a much lower, near-uniform ratio. This pronounced contrast at larger scales corroborates our conclusion in the main text: the attention heads localized by our method are specialized less for storing the downstream capability itself, and more for reading explicit rules from the input and routing this control signal into the internal pathway. This result confirms that the aggregation and routing roles of attention heads within the proposed source–aggregation–execution hierarchy generalize robustly across the Llama model family.

fraction of modules are shared across all or most tasks, whereas the majority of the critical components are highly task-specific. These supplementary results confirm that the limited cross-task component intersection is not an artifact of a specific model capacity, but rather a robust characteristic of how instruction-tuned models encode behaviors. It further reinforces our core mechanistic conclusion: the Llama family models execute instruction-following behaviors through a generalized, shared source–aggregation–execution mechanism, while instantiating this template via distinctly specialized local parameter carriers for different procedural constraints.

C.8 Cross-Task Overlap of Critical Modules on Larger Models

In Section 4.6 of the main text, we provided a static view of the parameter space for Llama-3.2-3B, demonstrating that post-training induces larger parameter shifts in MLP modules than in attention modules. To verify that this structural pattern is not an artifact of a specific model size but a consistent property of the instruction-tuning process, we extend this analysis to larger model scales: Llama-3.1-8B and Llama-2-13B. Fig. 32 and Fig. 33 illustrate the layer-wise and component-wise parameter differences between the base and instruction-tuned models for Llama-3.1-8B and Llama2-13B, respectively. Consistent with the observations on the 3B model, both larger models exhibit a pronounced disparity in parameter updates between different modules. Specifically, the MLP modules (representing neuron-level parameters) undergo substantially larger average parameter changes across layers compared to the attention modules (representing head-level parameters).

In Section 4.6 of the main text, we established that while different instruction-following tasks share a common hierarchical organization, they rely on partially distinct local components. This conclusion was drawn from the limited overlap of critical modules localized via Weight Patching (WP) on the Llama-3.2-3B model. To verify whether this shared-vs.-specific structural property generalizes across model scales, we extend the cross-task overlap analysis to the Llama-3.1-8B and Llama-2-13B models. Fig. 30 and Fig. 31 illustrate the intersection of the top-k attention heads and sparse neurons across the six representative IFEval tasks for the 8B and 13B models, respectively. Consistent with the observations on the 3B model (Fig. 13), the overlap among top-ranked components remains notably limited across both larger architectures. For both attention heads and MLP neurons, only a negligible

C.9 Layer-wise Parameter Changes across Different Model Scales

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Layer

0.0

15 20

0.1

0.2 0

Head

1.5 1.0

0.5

0.0

0.5 1.0 0.50.0 0.5

1.5

Neuron AVG

(a) Vec Method

Heads Distribution (Top-20)

0.10 0.05 0.00

15 20

0.05

0.10

Head

(b) KL Method

1.0

4.0

0.5

3.5

0.0

3.0

0.5 1.0 0.50.0 0.5

Neuron Avg

Figure 37. Heatmaps comparing component localization across distinct metrics (anchor similarity vs. KL divergence) on Llama-3.1-8B.

This persistent trend across different model capacities and Llama generations further corroborates our causal conclusion derived from Weight Patching (WP): the acquisition of instruction-following behavior during post-training relies more heavily on writing capabilities into sparse, neuroncentric parameter subsets rather than uniformly updating the attention routing mechanisms. The consistency of these static parameter drift patterns provides strong supplementary evidence for the source–aggregation hierarchy discussed in the main text. C.10 tion

Extended Analysis of Upstream Neuron Intersec-

In the main text (Sec. 4.4), we demonstrated a structural linkage between the middle-layer aggregation heads identified by Activation Patching (AP) and the shallow source neurons identified by Weight Patching (WP). To ensure that this systematic alignment is robust and not merely an artifact of a specific, stringent sparsity threshold, we extend our upstream tracing analysis here to a broader population of components. Specifically, Table 8 presents the intersection rates when expanding the retrieval scale to the top-5000 neurons. Following the exact protocol described in the main paper, we trace the upstream neuronal suppliers for the top-20 APranked heads using WP, and compute their overlap with the top-5000 neurons directly localized by WP. Consistent with our main findings, the results demonstrate that the substantial overlap is preserved even at this expanded scale. The intersection remains highly stable across multiple aggregation metrics, including the bestmatching head (Max), the top-5 heads average (Top-5 Avg), and the overall average across all 20 heads (Top-20 Avg). These extended results further corroborate the proposed hierarchical division of labor. They verify that the WPrecovered source neurons systematically act as upstream suppliers to the aggregation heads, and that this source-toaggregation routing pathway is a stable, large-scale structural property of the instruction-following mechanism. C.11 face

Absolute Performance of the Vector-Anchor Inter-

In Section 4.2 of the main text, we introduced the correction rate to quantify the fraction of the base-to-instruct performance gap recovered by activation steering. To provide a more comprehensive and absolute perspective on this validation step, Fig. 34 details the raw IFEval strict accuracy scores across all three evaluated model families

Neurons Distribution (Top-200)

Vec Method KL Method Shared

30 25

2.5 2.0 1.5

20 15 10

1.0

0.5 0.0

Vec Method KL Method Shared

Counts

0.1

Counts

0.2

Layer

Layers

(a) Key Head Distribution

Layers

(b) Key Neuron Distribution

Figure 38. Layer distributions and overlaps of top-ranked components identified across distinct metrics (anchor similarity vs. KL divergence) on Llama-3.1-8B.

(Llama-3.2-3B, Llama-3.1-8B, and Llama-2-13B) and all six representative instruction-following tasks. As demonstrated in the radar charts, the vanilla base models (brown lines) inherently lack the ability to consistently follow explicit formatting constraints, often yielding near-zero strict accuracy. The instruction-tuned models (blue lines) establish the empirical upper bound for these specific capabilities. When the extracted task vector is injected into the identified optimal anchor layer of the base model (green lines), we observe a dramatic and consistent behavioral shift. Across all scales and tasks, the simple activation steering intervention successfully bridges the massive performance gap, closely approximating the behavior of the instruction-tuned models. Notably, in several instances—such as the Title and No Comma tasks for Llama-3.2-3B, and the Title task for Llama2-13B—the steered base model even slightly outperforms the vanilla instruct model. This phenomenon further corroborates our central premise in Section 4.2: the extracted task vector effectively captures a compact and pure control representation for instruction following, largely devoid of conflicting signals. These absolute performance gains across multiple model scales confirm that the vector-anchor interface provides a reliable and stable criterion for the subsequent comparative analyses of Activation Patching and Weight Patching. C.12

Case Studies.

To intuitively ground the quantitative causal validation results, Figures 35 and 36 present text generation case studies using Llama-3.2-3B on the English Capital and Title tasks, respectively. The generated outputs visually corroborate the source–aggregation dichotomy. In the knockout scenarios (top panels), ablating either the AP-ranked upstream heads or the WP-ranked upstream neurons causes the Instruct model to ignore the formatting constraints, confirming the necessity of both modules. However, a stark contrast emerges in the restoration scenarios (bottom panels). Injecting the AP-identified modules back into the Base model completely fails to induce instruction-following behavior. Conversely, restoring the WP-identified modules (particularly the WP-ranked neurons) successfully enables the Base model to generate the correct format (e.g., producing text in all capital letters or enclosing the title in double angular brackets). These qualitative examples explicitly demonstrate that Weight Patching successfully locates the genuine

PREPRINT SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

parameter-level carriers of the capability, whereas Activation Patching highlights necessary but insufficient routing bottlenecks. C.13 Consistency of Localization Across Evaluation Metrics In our main methodology, we utilized the similarity to the intermediate vector anchor (the “Vec Method”) as the primary evaluation metric during our patching experiments to isolate critical parameters and activations. To verify that our component localization results are robust and not an artifact of this specific proxy metric, we compare it against the traditional Kullback-Leibler (KL) divergence metric (the “KL Method”), which measures the distributional shift over the entire output vocabulary space. Figure 37 presents a macroscopic comparison of the component importance heatmaps for both attention heads and MLP neurons on the Llama-3.1-8B model (evaluated on the English Capital task). As visually evident, the spatial distributions of critical components are nearly identical across the two metrics. The exact same specific heads and shallow/deep neurons are highlighted as highly influential, regardless of whether the intervention effect is measured via internal structural similarity (Vec Method) or final output probability shift (KL Method). To strictly quantify this alignment, Figure 38 illustrates the layer-wise distributions and the direct intersection of the top-ranked components (Top-20 for heads, Top-200 for neurons) identified by both methods. The bar charts demonstrate a remarkably high degree of overlap (represented by the dominant “Shared” teal bars). Both metrics precisely localize the same intermediate attention heads and the identical sparse subsets of source and execution MLP neurons. These supplementary results provide strong methodological validation. They confirm that the vector-anchor interface introduced in our framework is not only a valid proxy for traditional logit-based metrics like KL divergence, but it also yields highly stable and robust component localizations for tracing instruction-conditioned control mechanisms.

Record · ID 14083 · SHA-256 011d9cce73d46af9
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.