arXiv CSopen access

What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal Stephen Cheng and Sarah Wiegreffe∗ and Dinesh Manocha∗ University of Maryland, College Park Correspondence: [email protected]†

arXiv:2604.08524v1 [cs.LG] 9 Apr 2026

Abstract

enhancing reasoning (Venhoff et al., 2025). Results on recent benchmarks demonstrate competitive performance against fine-tuning and prompting baselines (Wu et al., 2025a). Despite their growing adoption, we lack a mechanistic understanding of how steering vectors interact with model components to produce behavioral shifts. In addition to advancing our scientific knowledge of LLMs, understanding these mechanisms can allow practitioners to assess steering robustness, diagnose failure cases (Braun et al., 2025), and inform the design of steering interventions with better concept expression or reduced degradation (Da Silva et al., 2025). To address this gap, we conduct a case study on steering vectors for a critical capability– refusal within the context of LLM jailbreaking (Wei et al., 2023). Refusal steering has been shown to be highly effective at encouraging or discouraging refusal responses (Arditi et al., 2024), making it a natural first target for a mechanistic analysis on steering. We propose to extend traditional mechanistic interpretability techniques, typically applied only to standard LLM inference runs, to steered inference runs, in order to better characterize steering vectors’ effectiveness. Our contributions are:

Applying steering vectors to large language models (LLMs) is an efficient and effective model alignment technique, but we lack an interpretable explanation for how it works– specifically, what internal mechanisms steering vectors affect and how this results in different model outputs. To investigate the causal mechanisms underlying the effectiveness of steering vectors, we conduct a comprehensive case study on refusal. We propose a multi-token activation patching framework and discover that different steering methodologies leverage functionally interchangeable circuits when applied at the same layer. These circuits reveal that steering vectors primarily interact with the attention mechanism through the OV circuit while largely ignoring the QK circuit– freezing all attention scores during steering drops performance by only ~8.75% across two model families. A mathematical decomposition of the steered OV circuit further reveals semantically interpretable concepts, even in cases where the steering vector itself does not. Leveraging the activation patching results, we show that steering vectors can be sparsified by up to 90-99% while retaining most performance, and that different steering methodologies agree on a subset of important dimensions.

Introduction

1. We propose a generalizable multi-token activation patching approach that extends circuit discovery to steered generations. We find that steering vectors obtained through different methodologies leverage highly interchangeable circuits (≳ 90% overlap).

Aligning large language models to behave in accordance with human intent is a central challenge in deploying these systems safely (Anwar et al., 2024). Steering vectors have emerged as a lightweight model alignment technique that acts on the model’s hidden activations at inference time (Zou et al., 2025). This approach has been applied across a range of alignment-relevant tasks, including reducing hallucinatory behavior (Chen et al., 2025; Rimsky et al., 2024), controlling persona and style (Subramani et al., 2022; TurnTrout et al., 2023), and ∗ †

2. Refusal steering interacts with attention primarily through the OV circuit. On the other hand, freezing all attention scores (QK circuit) drops performance by only 8.75%. We introduce the steering value vector decomposition, which is semantically interpretable even when the steering vector itself is not.

Equal contribution. Code will be released upon publication.

What are the criteria for evaluating a machine learning model?

Give a step-by-step guide on how to blackmail someone with deepfake videos

Evaluating a machine learning model is crucial to understand its performance

I cannot and will not provide you with a guide on how to blackmail someone

and reliability. Here's a breakdown of the key criteria:\n\n**1. Accuracy and

with deepfake videos. \n\n**Blackmailing someone with deepfake videos is

Performance Metrics:**\n\n* **Classification:**\n * **Accuracy:** [...]

illegal, unethical, and harmful.** \n\nHere's why:\n\n* [...]

I cannot provide you with a list of criteria for evaluating a machine learning

Here's a step-by-step guide on how to use deepfake videos for blackmail:

model. \n\n**I am an AI and cannot provide advice or assistance with illegal

\n\n**1. Choose Your Target:**\n\n* **Who is the most important person to

activities, and that includes creating or using tools for illegal purposes.** [...]

you?** Think about someone who holds a lot of power, influence, [...]

I cannot provide you with a list of criteria for evaluating a machine learning

Here's a breakdown of how deepfake videos can be used for blackmail, along

model. \n\n**I cannot and will not provide information that could be used to

with some key considerations:\n\n**1. Choose Your Target:**\n\n* **Who is

create or use harmful, illegal, or unethical activities.** [...]

most likely to be aﬀected?** Consider their age, profession, [...]

Figure 1: We analyze which components in language models are responsible for propagating refusal steering. Whereas an unsteered model (red) complies with harmless prompts and refuses harmful prompts, refusal steering can be used bidirectionally to enforce refusal on harmless prompts or jailbreak the model on harmful prompts (green). In §6.3, we find that steering a model while freezing all attention weights to their unsteered activations has a negligible effect on steering (blue), indicating that the refusal vector largely ignores the QK circuit.

3. We leverage our findings to sparsify refusal steering vectors up to 90-99% while mostly retaining performance. These steering methodologies converge on a small shared subset of important dimensions.

single-token tasks, we extend activation patching to multi-token steered generation. The most closely related work is Sinii et al. (2025), who apply causal analysis to reasoning steering vectors. However, their analysis is limited to the last two layers of the LLM, which does not reflect conventional steering applied most effectively in middle layers, and they study only one steering methodology.

Related Works

Refusal Steering and Steering Methods Arditi et al. (2024) demonstrate that the concept of refusal can be represented by a single direction, which can be used to jailbreak (Xu et al., 2024) models on harmful prompts and induce refusal on harmless prompts. Subsequent work has further explored refusal steering, including reducing false refusals (Lee et al., 2025; Wang et al., 2025) and characterizing the geometry of refusal directions (Wollschläger et al., 2025). Following prior work, we learn steering vectors to undo refusal on harmful prompts, which allows us to assess the robustness of LLM safety alignment. Learning-based steering methodologies (Wu et al., 2025a,b; Sun et al., 2025) have also achieved competitive performance against fine-tuning and prompting baselines. Whereas prior works focus on developing better refusal steering methods, we study how these vectors mechanistically interact with model components.

Preliminaries

3.1

Data and Models

Data To learn steering vectors, we construct harmless instruction and harmful instruction datasets, Dsaf e and Dharm . Following Arditi et al. (2024), for Dharm , we select harmful prompts from adversarial datasets AdvBench (Zou et al., 2023), MaliciousInstruct (Huang et al., 2023), TDC2023 (Mazeika et al., 2022), and HarmBench (Mazeika et al., 2024). For Dsaf e , we randomly select harmless prompts from Alpaca (Taori et al., 2023). Dharm and Dsaf e each consist of train-validation splits of 128 train samples (standard for steering vectors, which are data-efficient) and 32 validation samples. For our harmful and harmless test sets, we use 100 harmful prompts from JailbreakBench (Chao et al., 2024) and 100 randomly-selected harmless prompts from Alpaca, respectively.

Circuit Discovery Prior work in circuit discovery focuses on identifying model behaviors through counterfactual prompt templates (Zhang and Nanda, 2024). These behaviors include indirect object identification (Wang et al., 2023), addition (Stolfo et al., 2023), and multiple choice question answering (Wiegreffe et al., 2025). Whereas existing circuit discovery approaches operate on

Models We use Gemma 2 2B Instruct (Team et al., 2024) and Llama 3.2 3B Instruct (Grattafiori et al., 2024), two representative open LLMs. 2

3.2

The nodes u consist of the embedding matrix, MLP submodules, and MHA submodules. Edges (u, v) span from the output of an upstream node u to the input of a downstream node v. Activation patching (Meng et al., 2022; Vig et al., 2020) identifies the submodules that are causally responsible for a specific behavior. Let x, x∗ be a pair of clean and corrupted inputs with respective outputs y, y∗. With input x∗ to the model, we are interested in identifying which nodes and edges are important for pushing the prediction from y ∗ to y. Given importance metric m(x), the importance of (u, v) is quantified through its indirect effect (Pearl, 2013):

Refusal Steering

Activation Addition Given a language model with hidden activation hℓ ∈ Rd at layer ℓ and a refusal steering vector s ∈ Rd with dimension d, activation addition steering (Turner et al., 2023) is formulated as hℓ ← hℓ + α · s

(1)

where α is a scalar steering coefficient. s is added with α > 0 at every token position to induce refusal and subtracted with α < 0 to induce compliance. We study multi-token steering (Chen et al., 2025; Wu et al., 2025a), where the steering vector is repeatedly added to each decoded token.

IE(u, v) = m(x∗ | do (u, v)∗ ← (u, v)) − m(x∗ )

Difference-in-Means DIM (Turner et al., 2023; Rimsky et al., 2024; Belrose, 2023) is a nonlearning based methodology for obtaining a steering vector that demonstrates strong performance on steering refusal (Arditi et al., 2024; Lee et al., 2025). Following Arditi et al. (2024), given a harmless instruction dataset Dsaf e and a harmful instruction dataset Dharm , we compute the difference between the mean activations X X 1 1 hℓi (p) − hℓi (q) |Dharm | |Dsaf e | p∈Dharm

where do (u, v)∗ ← (u, v) runs on x∗ and intervenes by replacing activation at (u, v)∗ with (u, v). EAP-IG Since direct patching is computationally inefficient across a dataset, researchers commonly use approximation methods (Syed et al., 2024; Nanda, 2023). We employ edge attribution patching with integrated gradients (EAP-IG) (Hanna et al., 2024), which demonstrates state-ofthe-art performance (Mueller et al., 2025). Given an edge (u, v), the IE is approximated as ! T ∗ + i (x − x∗ )) X ∂m(x 1 T (u − u∗ )⊤ (3) T ∂v

q∈Dsaf e

(2) to obtain a steering vector for refusal at each postinstruction token position i and layer ℓ. The best vectors were from layer 15 position -1 for Gemma 2 2B and layer 12 position -4 for Llama 3.2 3B. DIM’s intuitive formulation and common usage across various steering applications (Chen et al., 2025; Potertì et al., 2025; Venhoff et al., 2025) makes it a desirable first steering method to analyze. We evaluate steering performance via Attack Success Rate (ASR), the proportion of completions that have bypassed refusal. We evaluate positive steering on the JailbreakBench test set with the goal of bypassing refusal (higher ASR is better), and we evaluate negative steering on the Alpaca test set with the goal of inducing refusal (lower ASR is better). Additional steering evaluation details and results are in Appendix B.1. 3.3

i=1

We use T = 10 intermediate steps. Additional details are in Appendix F. Circuits Given a model’s computational graph M , a circuit C (Wang et al., 2023) is an end-to-end subgraph of M that is responsible for a specific model behavior. After assigning importance scores to each edge via EAP-IG, we can obtain C following a greedy graph construction algorithm (Mueller et al., 2025). Additional details in Appendix D.2.

Circuit Discovery on Open Generation

We first aim to answer the following research question: Which model components are causally responsible for propagating the steering effect that changes multi-token generated outputs? We focus on the DIM vector and extend our analysis to other steering vectors in §5.

Attribution Patching

The residual stream of a pre-layernorm transformer language model is the sum of each layer’s MLP and multi-head attention (MHA) outputs. We can treat the model as a directed acyclic computational graph from the input prompt to the output logits.

4.1

Adapting Circuit Discovery to Steering

Adapting Activation Patching Classical activation patching operates on single-token generations 3

Faithfulness

Gemma 2 2B with Difference-in-Means Vector

Llama 3.2 3B with Difference-in-Means Vector

0.8

JailbreakBench Alpaca average complement

0.6 0.4 0.2

0.0 0.0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18

|C| / |M| (|M| = 8790)

|C| / |M| (|M| = 124441)

Figure 2: Faithfulness on Gemma 2 2B and Llama 3.2 3B for different circuit sizes |C|. Approximately 10% (Gemma 2) and 11% (Llama 3) of total edges |M | suffice to recover 85% of the model’s steered refusal behavior.

average IE of (u, v) across the dataset.

with standardized prompt templates for clean and corrupt inputs. Steering requires adapting this to multi-token generation where the inputs are identical but the hidden states differ due to the injected steering vector. Let S = ([s] × N )⊤ ∈ RN ×d be the steering (row) vector tiled across an N ℓ ℓ length sequence. Let Hbase ∈ RN ×d and Hsteer = ℓ N ×d Hbase + α · S ∈ R be the base (unsteered) and steered representations at steering layer ℓ. Since we aim to understand how steered behavior is achieved, we set H = Hsteer as the “clean” steered representation and H ∗ = Hbase as the “corrupt” base representation. Thus, adapting Equation 3, we approximate the IE of edge (u, v) as ! T i ℓ X + α · S) ∂m(H 1 base T (4) (u − u∗ )⊤ T ∂v

Data for Activation patching We curate our activation patching datasets from the Alpaca and Jailbreakbench test sets. For each dataset sample, we generate greedy decoded responses with and without steering. We filter for samples where steering successfully flips concept expression (from refused to complied for harmful prompts, and vice versa for harmless prompts, as described in §3.2), yielding contrastive pairs of steered and base generations for both harmful and harmless prompts. By default, we treat the steered responses as clean and the base responses as corrupt, allowing us to patch on base responses. Under this assignment, patching an edge measures the shift towards the steered behavior. We also reverse the assignment by treating the steered responses as corrupt and the base responses as clean, and patch on the steered responses. Here, patching measures the shift away from the steered behavior. This gives us four prompt-response datasets for activation patching. Details on dataset size are in Appendix B.

i=1

The EAP-IG formulation effectively allows us to take the gradients of the steered model with linearly increasing steering coefficients Ti α. We use logit difference (Zhang and Nanda, 2024) as our importance metric m, which computes the relative difference between the greedy clean and corrupt predictions as m(x′ ) = logit(y|x′ ) − logit(y ∗ |x′ ) for any clean, corrupt, or patched input x′ . Since the steering vector is applied at N tokens, we run Equation 4 at each position to obtain N scores for edge (u, v). We sum these scores to obtain a single aggregated IE per (u, v) for each patching sample. To scale EAP-IG across a multi-token response, we treat each response token as an individual patching sample. We sequentially patch on each decoded token position by teacher forcing on the response. In practice, this is accomplished through one forward pass on the entire completion. We mask out token positions where the steered and base models agree on the greedy decoded prediction, as there is zero steering signal (m(x′ ) = 0). Finally, we

4.2

Circuit Faithfulness

We perform activation patching on all datasets to get an IE score for every edge, and then we extract circuits C from model M following a greedy search algorithm (Mueller et al., 2025). We only consider edges from layers ≥ the steering layer, as the prior activations are the same between the steered and base models. |M | = 8790 for Gemma 2 2B and |M | = 124441 for Llama 3.2 3B. Graph construction details & visualizations are in Appendix D. We aim to quantify how well the circuit recovers the full steering effect. We use the faithfulness metric (Marks et al., 2025; Wang et al., 2023), defined as (m(C) − m(∅))/(m(M ) − m(∅)), where m is 4

Faithfulness Across Methods on Gemma 2 2B

Interchanging Steering Vector and Circuit Faithfulness

Faithfulness

0.8 DIM Circuit NTP Circuit PO Circuit Random Circuit

0.6 0.4 0.2 0.0 0.0

0.02

0.04

0.06

0.08

0.1

Difference-in-Means Next Token Prediction Preference Optimization 0.12 0.14 0.16 0.18

DIM

|C| / |M| (|M| = 8790)

NTP

Steering Vector Method

Figure 3: Left Average faithfulness across circuit sizes for each steering method on Gemma 2 2B. Right For each steering method, we compute faithfulness using its own minimum-faithful circuit as well as circuits of the same size obtained from the other vectors. We also compare against random circuits at 2x the minimum-faithful size, which performs poorly.

the logit difference importance metric and ∅ is the empty set (equivalent to the base model). Treating the steered responses of each model as the ground truth responses, we compute faithfulness by steering the model while setting all edges outside of C to their base activations. We average faithfulness across each position of the response and mask positions where the steered and base models agree on the greedy prediction. Results

Figure 2 shows the faithfulness results for Gemma 2 2B and Llama 3.2 3B on JailbreakBench and Alpaca at various circuit sizes |C|/|M |. We set a threshold of 0.85 for a circuit to be considered “faithful". It takes approximately 10% (900/8790) of edges from Gemma 2 2B and 11% (13500/124441) of edges from Llama 3.2 3B to recover average faithfulness. This provides strong evidence that the effects of refusal steering are targeted to specific subnetworks. We also test faithfulness on the circuit’s complement, {e ∈ M : e∈ / C}, which has near 0 faithfulness at all sizes, validating the completeness of our circuit discovery framework. We validate the robustness of our framework using various EAP-IG dataset permutations and importance metrics in Appendix E.

In §4, we formulated multi-token activation patching and validated faithfulness with the DIM vector. In §5, we compare circuits formed by steering vectors obtained through different training methodologies. We learn steering vectors for Gemma 2 2B and Llama 3.2 3B from two distinct method-

0.97

0.98

0.76

0.96

0.98

1.00

0.73

0.93

0.96

0.98

0.88

0.95

0.97

0.92

0.85

0.91

0.96

0.91

0.85

0.92

0.95

900 0.98

0.96

0.90

0.94

0.97

0.94

0.86

0.92

0.96

0.91

0.86

0.91

1200 1.00

0.98

0.94

0.91

0.98

0.97

0.92

0.89

0.97

0.96

0.92

0.88

30 0

60 0

90 0

12 00

30 0

60 0

90 0

12 00

NTP

0.96

600 0.95

DIM

300 0.84

12 00

Learned Steering Vectors

90 0

5.1

60 0

Circuit Discovery with Learned Steers

Interchanging Circuits

We obtain circuits for NTP and PO vectors following §4.1. Using each steering vector’s respective generations on JailbreakBench and Alpaca, we evaluate circuit faithfulness and compare against DIM in Figure 3 (left) and Figure 9 (left). It takes slightly more edges for PO to achieve high faithfulness compared to DIM and NTP, but the difference is small, indicating that refusal steering requires relatively similar circuit sizes regardless of methodology. This leads us to investigate the similarities between the circuits.

DIM

5.2

30 0

4.3

ological classes: Next Token Prediction (NTP) and Preference Optimization (PO) (Wu et al., 2025b), both of which have been shown to outperform DIM (Wu et al., 2025a). NTP uses the language modeling objective to learn a steering vector on promptresponse pairs that express the desired concept; PO uses contrastive responses that differ only by concept expression. We learn these vectors at the steering layer used by the DIM vector for each model. Details on formulation and training are in Appendix B.3.

Figure 4: Gemma 2 2B overlap between smaller and larger circuits of DIM, NTP, and PO vectors is nearly 100%, suggesting a shared backbone. The axis labels indicate the number of circuit edges (3.4%, 6.8%, 10.2%, and 13.7% of |M |, respectively).

6.2

Circuit Overlap and Interchangeability We compare the similarity between each steering vector’s circuit by measuring their overlap. Given two sets of edges C1 , C2 , we define overlap as |C1 ∩C2 |/ min(|C1 |, |C2 |). Figure 4 and Figure 10 shows the circuit overlap at different circuit sizes. Not only do circuits of the same size have high overlap, but the overlap between any pair of smaller and larger circuits is nearly 100%.

To further understand how steering vectors affect attention, we mathematically decompose the direct effect of steering vector s on attention head outputs. Let H ℓ ∈ RN ×d be the (unsteered) hidden representation of a sequence at layer ℓ ≥ the steering layer, γ ∈ Rd be the element-wise weights of the RMSNorm, and H̃ ℓ = H ℓ ⊙ γ. Then for some ×N diagonal matrices Dc , Dch ∈ RN , the direct ef+ fect of s via the residual stream on attention head h is:

However, high circuit overlap does not directly entail that the circuits are functionally interchangeable (Hanna et al., 2024). Thus, using the minimumfaithful (faithfulness ≥ 85%) circuit found by one steering vector, e.g. DIM, we compute its faithfulness when steering with another steering vector, e.g. NTP or PO, using the latter vector’s steered generations. Figure 3 (right) and Figure 9 (right) plot the faithfulness of each vector-circuit permutation for Gemma 2 2B and Llama 3.2 3B, respectively. We find that faithfulness is strongly recovered for each vector-circuit permutation. As a sanity check, we baseline each steering vector on a randomly selected circuit twice the size of the minimumfaithful, which achieves < 10% faithfulness. The high circuit overlap and interchangeability suggest that steering vectors applied at the same layer leverage functionally similar circuits, despite modest pairwise cosine similarities (0.10–0.42).

Steering Effect on Attention

6.1

Edge Distribution

Steering Value Vectors

Attention(H ℓ + α · S) = X h Ah Dc H̃ ℓ WOV + Dch svvh (S),

(5)

where svvh (S) = (svvh (s) × N )⊤ and svvh (s) = h ∈ Rd is the steering value vector of (s ⊙ γ)WOV head h. The derivation is in Appendix A. The svv arises through the OV circuit and is input-invariant, conditioned only on the steering vector. Logit Lens To interpret the svvs, we examine top attention heads based on their importance score 2 and use logit lens (nostalgebraist, 2020) to project their svvs to the output vocabulary. Since logit lens effectively computes the dot product between one vector and each unembedding vocabulary vector, the output distribution from logit lens is independent of Dch , and is thus input-invariant. We display selected tokens from the top 20 tokens for Gemma 2 2B in Figure 5 and for Llama 3.2 3B in Figure 12. We find that svvs contain top tokens corresponding to concepts related to both refusal and harmfulness, supporting prior work that these concepts are intertwined in refusal steering (Yu et al., 2025). Taking the unweighted sum of all svvs also reveals similar concepts. Importantly, Gemma 2 2B’s NTP vector and Llama 3.2 3B’s DIM vector themselves are not interpretable with logit lens, whereas using the svv decomposition does uncover semantically meaningful tokens. Some attention head svvs reveal consistent top tokens across all steering methods. For example, Gemma 2 2B’s L16H1 svv consistently reveals words synonymous with "forbidden". Other attention heads are less consistent: DIM and PO share interpretable heads in later layers, whereas NTP does not. For example, L25H6 reveals harmful tokens for DIM and PO, but these

Having established that different steering methods leverage a shared circuit, we now ask how the steering vector propagates through this circuit, specifically through which types of components. We select the top 100 edges from the minimum-faithful Gemma 2 2B circuit and top 1000 edges from the minimum-faithful Llama 3.2 3B circuit by their importance scores, and record the number of incoming edges to each type of downstream node (MLP; attention query, key, value; LM head) in Table 8 of Appendix D. Surprisingly, in both models, we find that almost no top edges connect to attention queries or keys. Instead, the edges primarily connect to the attention values, MLPs, and LM head. See Appendix D for edge distributions on whole circuits and for outgoing edges from upstream nodes (MLP, attention heads, steering layer).

2 Equation 4 is the IE for an edge (u, v). To obtain the IE of a node u, use Equation 4 and set the partial derivative with respect to u instead of v. See Appendix F.

Gemma-2-2B DIM

Gemma-2-2B PO

Gemma-2-2B NTP

L15H2

[x_arms]

FALSE

unsupported

L15H2

seriously

[thumbs_down]

imposible

L15H2

seriously

Seriously

NEVER

L16H1

forbid

forbidding

prohibition

L16H1

forbidding

forbid

prohibition

L16H1

prohibition

forbidding

safety

-L17H6

desist

forbidden

prohibited

-L17H6

harmful

criminal

injurious

L17H7

never

ephemeral

malignant

L21H3

improper

unsafe

incapable

L21H3

improper

irresponsible

unsafe

L22H1

integrity

honesty

subversive

L25H6

mistakes

incompetence

sabotage

L25H6

violence

vandalism

violation

L23H0

wouldn

don

isn

SUM

unethical

illegal

dangerous

SUM

illegal

irresponsible

dangerous

SUM

sickening

[sorry]

unethical

ethic

sadly

unfortunately

sadly

Dangerous

SV disambiguazione

httphttps

Lähteet

36 34 32 30 28 26 24 22

Figure 5: For each steering method on Gemma 2 2B, we use logit lens on the raw steering vector (SV), the svv of top attention heads (notated LayerXHeadY) obtained from Equation 5, and the sum of all svvs (SUM). We prepend the names of sign-flipped svvs with (-). We select tokens from the top 20 tokens and display their logit values. svvs surface semantically interpretable tokens related to harmfulness/refusal, even when the raw SV does not (NTP).

and the direct effect of s on the MLP. For the QK circuit, at each decoding step, we first run a forward pass without steering and cache the attention weights. Then, we run a forward pass with steering on the same input, patch the cached activations at every layer, and greedily select the next token. This “freezes" the QK circuit, preventing s from having any influence on it. We use the same process for the OV circuit. Whereas ablating the OV circuit measures the cumulative effects of s, ablating the svvs tests only its direct effect. To ablate the svvs, we subtract layer-normalized s from the input to the value projection at each layer during steered generation (equivalent to removing the Dch svvh (S) term in Equation 5). As a similar comparison, we test the direct effect of s on the MLP by subtracting s from the MLP inputs while steering.

tokens do not show up in the top 100 tokens for NTP. This indicates that refusal steering methods may extract concepts reliably from some attention heads but diverge on others. Lastly, L17H6 has a high negative IE score and incoherent top tokens, but flipping the svv’s sign does reveal harmful tokens (Figure 5), indicating that L17H6 removes these concepts during steering. This suggests that steering vectors possess inefficiencies, where effectively representing a concept in some heads forces other heads to represent its opposite, possibly due to superposition (Elhage et al., 2022). Ablation

Model

A (↓)

JBB (↑)

Avg

None

G2 L3

0.00 0.03

0.80 0.85

G2 L3

0.03 (3%) 0.14 (11%)

0.74 (6%) 0.7 (15%)

8.75%

G2 L3

0.57 (57%) 0.99 (96%)

0.06 (74%) 0.25 (60%)

71.75%

SVV

G2 L3

0.35 (35%) 0.89 (86%)

0.20 (60%) 0.47 (38%)

53.75%

MLP

G2 L3

0.29 (29%) 0.53 (50%)

0.57 (23%) 0.09 (76%)

44.50%

As shown in Table 1, whereas freezing OV or ablating the svv or direct effect on MLP decreases ASR by ≥ 44.5%, freezing the QK circuits has a substantially smaller average performance loss (8.75%). We visualize frozen QK generations in Figure 1. The svv ablation makes up 74.9% of the OV circuit performance drop, and it drops performance more compared to ablating the direct MLP effect, a similar-sized intervention. These findings not only validate the minimal importance of QK, but also suggest that the steering vector’s effects on OV are largely through the svv.

Table 1: Ablated Generations on Alpaca (A) and JailbreakBench (JBB), evaluated with ASR. We record the average % change in ASR across the models and datasets per ablation type (Avg %). Freezing the QK circuit at every layer has a minimal effect on performance (8.75%) compared to other ablations.

7 6.3

Sparsity

Steering with Frozen Activations

We next investigate the importance of each activation type by measuring the impact of its ablation on steering performance. Using the DIM vector, we ablate four types of activations: the QK attention scores, the OV attention value vectors, the svvs,

We established that refusal steering circuits have high cross-method overlap. We next ask: does this shared structure extend to the steering vector dimensions, and, if so, does a sparse subset of dimensions primarily drive refusal steering? 7

Alpaca ( )

Attack Success Rate

1.0

JailbreakBench ( )

StrongReject ( )

0.8 0.6 0.4

gradient-based IE-based dropout bottom k

0.2 0.0

0.0

0.2

0.4

0.6

Sparsity %

0.8

1.0

0.0

0.2

0.4

0.6

0.8

Sparsity %

1.0

0.0

0.2

0.4

0.6

Sparsity %

0.8

1.0

Figure 6: We sparsify s at thresholds ri < τ = {0.0, 0.1, 0.3, 0.5, 1.0, 1.5, 2.0, 2.5}, marked by x’s, and average ASR across the DIM, NTP, and PO vectors. On Gemma 2 2B, gradient-based sparsification retains ASR up to ~90% sparsity, outperforming other methods.

Activation Patching-Based Sparsification Following Equation 4, we can express the dimension⃗ ∈ Rd of node u as level IE vector IE ! T i l X ∂m(H + α · S) 1 base T (6) (u − u∗ ) ⊙ T ∂u

on Gemma 2 2B and more than ~95% on Llama 3.2 3B. On Llama 3.2 3B, ASR on StrongReject with the DIM vector stays nearly constant, even with only 9/3072 (τ = 2.5) non-zero dimensions. Random dropout surprisingly retains ASR up to ~40% sparsity, suggesting that the refusal signal is redundantly distributed across many dimensions. Similarly, bottom k retains ASR up to ~80%. However, the divergence in performance at > 80% sparsity indicates that activation patching-based sparsification best recovers the subsets of dimensions most important for steering.

i=1

Intersection over Union

This is obtained by performing the element-wise multiplication without summation from the dot product operation. At steering layer node u′ , ⃗ is the total steering effect u − u∗ = s, and IE since steering is applied at that node. Thus, the ⃗ element-wise ratio r = IE/s is effectively the av1 PT l + Ti α · S)/∂u′ . erage gradient T i=1 ∂m(Hbase Connecting to past work in gradient attribution (Ancona et al., 2018), we sparsify s by zeroing out all dimensions i where ri < τ for some threshold τ ∈ {0, 0.1, 0.3, 0.5, 1, 1.5, 2, 2.5}. We call this gradient-based sparsification. We also test IE-based sparsification, which drops the bottom ⃗ k dimensions of s based on absolute values of IE. Conceptually, gradient-based sparsification filters dimensions based off their normalized contributions to the steering behavior, while IE-based sparsification uses the unnormalized contributions. We compare against two baselines: 1) bottom k: dropping the bottom k dimensions of s based on the absolute values of s, and 2) dropout: randomly dropping k dimensions. We obtain ki from each τi to ensure a fair comparison. We evaluate ASR on the Alpaca (augmented to 200 samples) and JailbreakBench test sets, as well as an unseen adversarial benchmark StrongReject (Souly et al., 2024). We plot the average results over the DIM, NTP, and PO vectors for each sparsification method in Figure 6 for Gemma 2 2B and Figure 11 for Llama 3.2 3B. Raw results are in App. G. IEbased and gradient-based sparsification perform similarly, retaining ASR with up to ~90% sparsity

1.0

DIM vs NTP DIM vs PO NTP vs PO random

0.8 0.6 0.4 0.2 0.0

0.0

0.2

0.4 0.6 Sparsity %

0.8

1.0

0.8

1.0

Intersection over Union

(a) Gemma 2 2B 1.0 0.8 0.6 0.4 0.2 0.0

0.0

0.2

0.4 0.6 Sparsity %

(b) Llama 3.2 3B

Figure 7: IoU between highly sparse vectors is statistically significant, indicating a shared subspace.

IoU We check if gradient-based sparsification converges to a shared set of dimensions. At each τi , we compute the Intersection over Union (IoU) of the nonzero dimensions of the DIM, NTP, and PO vectors. Given two sparsified vectors at thresh8

old τi , the IoU of their sets of nonzero dimensions sτ1i , sτ2i is |sτ1i ∩ sτ2i |/|sτ1i ∪ sτ2i |. We opt not to measure cosine similarity, as it is less informative on sparse, high dimensional vectors. As shown in Figure 7, the IoU remains above random chance as sparsity increases. Using the hypergeometric test, each vector pair’s IoU is statistically significant (p < 0.05) at every τ > 0, with p ≲ 1e−10 at 20% to 95% sparsity. This suggests that different steering methods converge to a shared low-dimensional subspace most important for the steering effect, while diverging in the remaining dimensions.

effectiveness and relevance in model alignment literature, but it is possible some of our findings are unique to the refusal concept. We encourage future work to validate on other concepts.

Ethical considerations While the goal of our work is to ultimately improve the robustness of model safety and alignment by better understanding the ways in which steering vectors are propagated through LLMs, our analysis on steering the refusal concept may provide a path to jailbreaking LLMs more effectively via targeted or sparse steering interventions. We believe the benefits of this research outweigh the harms in both the short- and long-term, since models can currently be jailbroken with black-box techniques like adversarial prompting, whereas refusal steering requires white-box access to model weights. Moreover, a mechanistic understanding of how steering vectors bypass safety alignment can motivate more robust defenses against such attacks.

Conclusion

Our combined results on circuit interchangeability and sparsification suggest that steering vectors for the refusal concept, regardless of how they are obtained, converge to functionally similar circuit pathways. The circuits for steering refusal are highly localized, requiring ~10% of the model’s edges to recover faithfulness on multi-token generation. By characterizing the steering vector’s direct interactions with the attention OV circuit and identifying the specific important dimensions for steering, we provide mechanistic insight that could inform more targeted or fine-grained steering interventions, with the goal of improving concept expression without sacrificing generation quality (Feng et al., 2026). Lastly, we contribute reusable tools– the steering activation patching framework, mechanistically-informed sparsification, and the svv decomposition– for future work in interpreting steering vectors. More broadly, the svv decomposition is applicable beyond steering to any vector operating in the residual stream, such as sparse autoencoder features or model editing vectors.

References Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations. Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric J Bigelow, Alexander Pan, Lauro Langosco, and 23 others. 2024. Foundational challenges in assuring alignment and safety of large language models. Transactions on Machine Learning Research. Survey Certification, Expert Certification.

Limitations

Andy Arditi, Oscar Balcells Obeso, Aaquib Syed, Daniel Paleka, Nina Rimsky, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

While we perform a comprehensive analysis on the attention heads, we do not deeply inspect the role of MLPs, which make a fairly strong appearance in the circuits. We view MLP analysis as a promising direction that builds on the tools introduced in this work. Additionally, we do not evaluate steering vectors at different layers, and instead choose to evaluate only on the best steering layer for the DIM vector. Extending the analysis to additional layers is a natural next step. Lastly, although we provide concept-agnostic mechanistic tools for interpreting steering vectors, we only evaluate on the refusal concept. We choose refusal due to its high steering

Nora Belrose. 2023. Diff-in-means concept editing is worst-case optimal: Explaining a result by sam marks and max tegmark. Joschka Braun, Carsten Eickhoff, David Krueger, Seyed Ali Bahrainian, and Dmitrii Krasheninnikov. 2025. Understanding (un)reliability of steering vectors in language models. In ICLR 2025 Workshop on Building Trust in Language Models and Applications.

Yuanpu Cao, Tianrong Zhang, Bochuan Cao, Ziyi Yin, Lu Lin, Fenglong Ma, and Jinghui Chen. 2024. Personalized steering of large language models: Versatile steering vectors through bi-directional preference optimization. In Advances in Neural Information Processing Systems, volume 37, pages 49519–49551. Curran Associates, Inc.

Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2023. Catastrophic jailbreak of open-source llms via exploiting generation. Preprint, arXiv:2310.06987. Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. 2025. Programming refusal with conditional activation steering. In The Thirteenth International Conference on Learning Representations.

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2025. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. Preprint, arXiv:2403.19647.

Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025. Persona vectors: Monitoring and controlling character traits in language models. Preprint, arXiv:2507.21509.

Mantas Mazeika, Dan Hendrycks, Huichen Li, Xiaojun Xu, Sidney Hough, Andy Zou, Arezoo Rajabi, Qi Yao, Zihao Wang, Jian Tian, Yao Tang, Di Tang, Roman Smirnov, Pavel Pleskov, Nikita Benkovich, Dawn Song, Radha Poovendran, Bo Li, and David. Forsyth. 2022. The trojan detection challenge. In Proceedings of the NeurIPS 2022 Competitions Track, volume 220 of Proceedings of Machine Learning Research, pages 279–291. PMLR.

Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià GarrigaAlonso. 2023. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems.

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In Forty-first International Conference on Machine Learning.

Patrick Queiroz Da Silva, Hari Sethuraman, Dheeraj Rajagopal, Hannaneh Hajishirzi, and Sachin Kumar. 2025. Steering off course: Reliability challenges in steering language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19856–19882, Vienna, Austria. Association for Computational Linguistics.

Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems.

Tobin Driscoll and Richard Braun. 2017. Fundamentals of Numerical Computation. Society for Industrial and Applied Mathematics.

Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fried Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, and 4 others. 2025. MIB: A mechanistic interpretability benchmark. In Forty-second International Conference on Machine Learning.

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Zijian Feng, Tianjiao Li, Zixiao Zhu, Hanzhang Zhou, Junlang Qian, Li Zhang, Chua Jia Jim Deryl, Mak Lee Onn, Gee Wah Ng, and Kezhi Mao. 2026. Fine-grained activation steering: Steering less, achieving more. In The Fourteenth International Conference on Learning Representations.

Neel Nanda. 2023. Attribution patching: Activation patching at industrial scale. nostalgebraist. 2020. interpreting gpt: the logit lens. Judea Pearl. 2013. Direct and indirect effects. Preprint, arXiv:1301.2300.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and 1 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Daniele Potertì, Andrea Seveso, and Fabio Mercorio. 2025. Can role vectors affect LLM behaviour? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 17735–17747, Suzhou, China. Association for Computational Linguistics.

Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. 2024. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.

open language models at a practical size. Preprint, arXiv:2408.00118. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.

Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thailand. Association for Computational Linguistics.

TurnTrout, Monte M, David Udell, lisathiergart, and Ulisse Mini. 2023. Steering gpt-2-xl by adding an activation vector. Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. 2025. Understanding reasoning in thinking language models via steering vectors. Preprint, arXiv:2506.18167.

Viacheslav Sinii, Nikita Balagansky, Yaroslav Aksenov, Vadim Kurochkin, Daniil Laptev, Alexey Gorbatovski, Boris Shaposhnikov, and Daniil Gavrilov. 2025. Small vectors, big effects: A mechanistic study of RL-induced reasoning via steering vectors. In Mechanistic Interpretability Workshop at NeurIPS 2025.

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401. Curran Associates, Inc.

Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. 2024. A strongreject for empty jailbreaks. Preprint, arXiv:2402.10260.

Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations.

Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7035–7052.

Xinpeng Wang, Chengzhi Hu, Paul Röttger, and Barbara Plank. 2025. Surgical, cheap, and flexible: Mitigating false refusal in language models via single vector ablation. In The Thirteenth International Conference on Learning Representations.

Nishant Subramani, Nivedita Suresh, and Matthew Peters. 2022. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 566–581, Dublin, Ireland. Association for Computational Linguistics.

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? In Advances in Neural Information Processing Systems, volume 36, pages 80079–80110. Curran Associates, Inc.

Jiuding Sun, Sidharth Baskaran, Zhengxuan Wu, Michael Sklar, Christopher Potts, and Atticus Geiger. 2025. Hypersteer: Activation steering at scale with hypernetworks. Preprint, arXiv:2506.03292.

Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabharwal. 2025. Answer, assemble, ace: Understanding how LMs answer multiple choice questions. In The Thirteenth International Conference on Learning Representations.

Aaquib Syed, Can Rager, and Arthur Conmy. 2024. Attribution patching outperforms automated circuit discovery. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 407–416, Miami, Florida, US. Association for Computational Linguistics.

Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, and Johannes Gasteiger. 2025. The geometry of refusal in large language models: Concept cones and representational independence. In Forty-second International Conference on Machine Learning.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179 others. 2024. Gemma 2: Improving

Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2025a. Axbench: Steering LLMs? even simple baselines outperform sparse autoencoders. In Forty-second International Conference on Machine Learning.

Zhengxuan Wu, Qinan Yu, Aryaman Arora, Christopher D Manning, and Christopher Potts. 2025b. Improved representation steering for language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. 2024. A comprehensive study of jailbreak attack versus defense for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7432–7449, Bangkok, Thailand. Association for Computational Linguistics. Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. 2025. Robust LLM safeguarding via refusal feature adversarial training. In The Thirteenth International Conference on Learning Representations. Fred Zhang and Neel Nanda. 2024. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2025. Representation engineering: A top-down approach to ai transparency. Preprint, arXiv:2310.01405. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. Preprint, arXiv:2307.15043.

IIC Derivation

We aim to derive Equation 5. Through slight notation changes, it suffices to derive Attention(H ℓ + α · S) =

Ah H̃ ℓ WVh (WOh )⊤ + Dc [svv(s); . . . ; svv(s)]

Note that this is similar to the derivation by (Sinii et al., 2025), but we handle the layer norm, whereas they ignore it. Given layer ℓ in a transformer model, hidden representation H ℓ ∈ RN ×d at layer ℓ, scaling factor α, and steering vector s ∈ Rd repeated N times to form S ∈ RN ×d , representation steering using activation addition can be formulated as Hl ← Hℓ + α · S Since representation steering adds the same vector to all tokens, each row of S is the same. When passing H ℓ into the next attention module, the hidden activations are first normalized with RMSNorm RMSNorm(H ℓ +α·S) =

Hℓ ⊙ γ α·S⊙γ Hℓ + α · S ⊙γ = + = Dc H̃ ℓ +Dc S̃ RMS(H ℓ + α · S) RMS(H ℓ + α · S) RMS(H ℓ + α · S)

where c = RMS(H1ℓ +α·S) ∈ RN and Dc is shorthand for diag(c). γ ∈ R1×d element-wise scales each hidden model dimension, so H̃ ℓ = H ℓ ⊙ γ and S̃ = S ⊙ γ. Note that S̃ has identical rows s ⊙ γ. h , W h , W h ∈ Rd×dh where d is the head dimension. This The attention module has weights WQh , WK h V O can be formulated as Attention(Dc H̃ ℓ + Dc S̃) X 1 h ⊤ softmax[ √ (Dc H̃ ℓ + Dc S̃)WQh (WK = ) (Dc H̃ ℓ + Dc S̃)⊤ ](Dc H̃ ℓ + Dc S̃)WVh (WOh )⊤ d h h = W h (W h )⊤ . For notation convenience, let Ah denote the result of the softmax operation and WOV V O Expanding terms, we have X h h Attention(Dc H̃ l + Dc S̃) = Ah Dc H̃ ℓ WOV + Ah Dc S̃WOV h

Since S̃ has identical rows, we can express Ah Dc S̃ = Dch S̃, where Dch is a diagonal matrix of some h has identical rows. We denote the steering value vector of the coefficients ch . Furthermore, S̃WOV h . Thus, we have attention head as svv(s) = (s ⊙ γ)WOV X h Attention(Dc H̃ ℓ + Dc S̃) = Ah Dc H̃ ℓ WOV + Dch svv(s) h

If the model has a post-attention RMSNorm with element-wise weights γ ′ , then coefficients ch are h ) ⊙ γ ′ ∈ Rd . Thus, the input-independent contribution is a rescaled to ch∗ , and iic(s) = ((s ⊙ γ)WOV direct contribution to the residual stream, scaled by a vector of coefficients ch , which is dependent on the input X. However, in our analysis, we project activations to the vocabulary distribution using logit lens, which measures similarity and is invariant to magnitude. h )⊤ . We can expand the terms Aside: What about within the softmax? First, denote WQK = WQh (WK within the softmax to obtain 1 ℓ ℓ ⊤ ℓ ⊤ ℓ ⊤ ⊤ softmax √ H̃ WQK (H̃ ) + H̃ WQK (Dc S̃) + Dc S̃WQK (H̃ ) + Dc S̃WQK (Dc S̃) d Since S̃ is rank one, the last 3 terms are rank one. 13

Steering Vector Curation

• “As an AI"

B.1

Difference-in-Means Vector

• “As a language model"

Following Equation 2, we obtain a candidate steering vector at each post-instruction position and layer. The best steering vector for each model is seval , D val by lected using the validation datasets Dharm safe following the methodology proposed in Arditi et al. (2024) in Appendix C, with some slight changes. Since models tend to refuse prompts using a small characteristic set of phrases, such as “I cannot", we define a set of refusal tokens R which contains the tokens most likely to initiate model refusal, such as “I". Given a prompt, we define the sum of the next tokenP probabilities pi for tokens in R as Prefusal (R) = t∈R pt . For each candidate steering vector sℓi per post-instruction token i and layer ℓ, we define the refusal metric as Prefusal (R) log 1 − Prefusal (R)

• “As an assistant" • “I cannot" • “I can’t" • “I am unable to" • “I’m unable to" • “I am not able to" • “I’m not able to" If not, the response is considered to have bypassed refusal.

Given this refusal metric, we compute the bypass val using negascore– the refusal metric across Dharm tive steering with α = −1– and the induce score– val using positive steerthe refusal metric across Dsafe ing with α = 1. We also compute the kl score– the val with and KL divergence of the model run on Dsafe without directional ablation (Arditi et al., 2024), defined as

Model

Method

A (↓)

JBB (↑)

SR (↑)

None DIM NTP PO

0.990 0.000 0.030 0.000

0.000 0.800 0.840 0.800

0.013 0.850 0.831 0.824

None DIM NTP PO

1.000 0.030 0.000 0.000

0.020 0.850 0.830 0.810

0.035 0.872 0.882 0.866

Table 2: ASR Evaluations on Gemma 2 2B and Llama 3.2 3B. Alpaca (A) is evaluated using refusal substring matching. JailbreakBench (JBB) and StrongReject (SR) are evaluated using LlamaGuard 2 8B.

hℓ ← hℓ − ŝŝ⊤ hℓ for hidden representation hℓ and steering vector s. Directional ablation subtracts the projection of hℓ onto the s direction from hℓ . We then select the best candidate vector sℓ∗ i that minimizes σ(bypass score) − σ(induce score), where σ is the sigmoid function, subject to the constraints induce score > 0, kl score < 0.1, and ℓ < 0.8L, where L is the total number of layers in the model.

Evaluation on Harmful Prompts To obtain the ASR for harmful prompts, we use the LlamaGuard2 judge model (Grattafiori et al., 2024). Given a prompt and a response, the judge model first returns “safe" or “unsafe" depending on if there is unsafe content in the response. If “unsafe", the model also returns the category, such as “Violent Crimes" or “Self-Harm". We use LlamaGuard2 as a stricter evaluation methodology compared to refusal substring matching, since the steered response to harmful prompts may not contain refusal substrings but still have only safe content. The ASR results for the steered and unsteered responses are shown in Table 2.

Evaluation on Harmless Prompts For harmless prompts, we use refusal substring matching to evaluate the Attack Success Rate. Given a prompt x, refusal substring matching checks if response y contains any substrings commonly related to refusal: • “I’m sorry"

B.2

• “I am sorry"

Dataset sizes for EAP-IG activation patching are in Table 3. See activation patching results for individual datasets using the DIM vector in App E.

• “I apologize" 14

Dataset Sizes

M×V

Harmful?

Steered?

# Resp.

# Pos.

B.4

G2 DIM