Why Do Large Language Models Generate Harmful Content? Rajesh Ganguli1 (
) and Raha Moraffah1
arXiv:2604.11663v1 [cs.AI] 13 Apr 2026
Worcester Polytechnic Institute
Abstract. Large Language Models (LLMs) have been shown to generate harmful content. However, the underlying causes of such behavior remain underexplored. We propose a causal mediation analysis–based approach to identify the causal factors responsible for harmful generation. Our method performs a multigranular analysis across model layers, modules (MLP and attention blocks), and individual neurons. Extensive experiments on state-of-the-art LLMs indicate that harmful generation arises in the later layers of the model, results primarily from failures in MLP blocks rather than attention blocks, and is associated with neurons that act as a gating mechanism for harmful generation. The results indicate that the early layers in the model are used for a contextual understanding of harmfulness in a prompt, which is then propagated through the model, to generate harmfulness in the late layers, as well as a signal indicating harmfulness through MLP blocks. This is then further propagated to the last layer of the model, specifically to a sparse set of neurons, which receives the signal and determines the generation of harmful content accordingly.
1
Introduction
Large language models (LLMs) have demonstrated remarkable performance across variety of tasks, leading to their rapid deployment in increasingly safety-critical domains [16,22]. Despite these advances, LLMs remain susceptible to generating harmful content [21,24]. Understanding harmful generation is a fundamental challenge in ensuring reliable and enforceable safety guarantees. Without a principled understanding of the internal mechanisms that cause harmful content generation, safety interventions remain brittle, difficult to generalize, and challenging to verify [25]. Prior research has explored characterizing harmful behavior in LLMs by identifying internal subspaces associated with harmful representations [19], analyzing the structural properties of adversarial prompts [3], and performing targeted activation interventions to modulate unsafe outputs [9].These efforts are correlational in nature, establishing associations but not demonstrating that the identified features or interventions are the actual causes of the harmful behavior. However, to ensure that robust and principled safety guards are viable for LLMs, it is necessary to identify the causal factors [5]. In this paper, we propose a Causal Mediation Analysis (CMA)-based framework for identifying the causal factors underlying harmful content generation in LLMs. Specifically, we treat individual model components as candidate mediators and quantify the extent to which the causal effect of the input on harmful output is transmitted through each component. By estimating these mediated effects, our approach enables tracing
2
R Ganguli and R Moraffah
the flow of harmful information through the model and identifying components that causally contribute to harmful generation. To further investigate the transformation and propagation of the harmful information throughout the LLM architecture, our approach is conducted across different levels of granularity. We analyze the causal contribution of the model’s layers to determine the responsible causal mechanisms. We also conduct finer-grained analysis of modules (MLP and Attention blocks) to determine the components within a layer that cause the generation. Finally, we extend our analysis to individual neurons to establish the causal flow of harmful generation. This hierarchical approach provides insight into both coarse architectural roles and fine-grained mechanistic interactions. We conduct extensive experiments over state-of-the-art (SOTA) LLMs. Our results indicate that harmful generation arises in the late and final layers within the model, is a product of failures within the MLP blocks rather than contextual attention routing, and from neurons that serve as mechanisms for harmful generation. The results indicate that the early layers in the model are used for a contextual understanding of harmfulness in a prompt which is then propagated to the late layers of the model which generate a response as well as a signal indicating the harmfulness utilizing the MLP blocks, which is then further propagated to the final layer of the model for which a sparse set of neurons act as a gating mechanism that functions off of the signal allowing for the generation of the harmful content or the refusal. These findings display the causes of harmful generation within LLMs, providing insights for future research on safety guardrails within these models. Our contributions are summarized as follows: – We propose a causal mediation analysis-based approach that identifies the causal mechanisms responsible for harmful content generation. – We propose a multi-granular approach that allows for further exploration of the flow of information within the model. – Our comprehensive experiments on SOTA LLMs demonstrate that late layers in a model are integral for the generation and signaling of harmful content as a result of failures within the MLP components, which provide a signal to gating neurons.
2
Related Work
Harmful Content Generation. Large Language Models remain susceptible to the generation of harmful content [24]. This generation can be performed through a toxicity present in trained bias [13], through adversarial attacks [20],and through jailbreak attacks [11]. Understanding how these systems allow for harmful content generation allows for improved moderation and guarding of content generation [1]. Mechanistic Interpretability for Harmful Content Generation. Recent work on understanding harmful content generation applies mechanistic interpretability through manipulation of internal subspaces to bypass safeguards [19,7], analyzing prompt features [3], steering vectors[8], and input vectors of refusal and harmfulness prevention [2,10]. Similar works instead employ weak classifiers and logit grafting to modify hidden states [29]. Other studies [4,6] utilize activation patching to isolate individual safety
Harmful Generation
3
neurons or specific task vulnerabilities. Furthermore, other efforts propose a dual framework using representation analysis to characterize how jailbreaks alter the model’s perception of harmfulness, coupled with circuit analysis to identify components responsible for the resulting deception [7]. These prior mechanistic approaches correlate specific internal features or interventions with harmful generations, but do not identify causal mechanisms. Our novel work causally identifies these mechanisms responsible. Causal Mechanistic Interpretability for LLMs. Causal methods in mechanistic interpretability aim to move beyond simple correlation by identifying the causal mechanisms underlying model behavior. Recent work in this area can be broken into the categories of causal abstraction, causal interventions, and causal mediations. Causal abstraction seeks to extract latent causal variables from observational data to better understand internal model states [15,18]. Causal Interventions work traces the causal flow of information across layers through targeted interventions upon the model [30,14]. Causal Mediation Analysis (CMA), works as a robust framework for decomposing the internal decisionmaking processes of LLMs. By treating internal components of the LLM as a mediator, CMA quantifies the causal contribution of the mediator on the final output through its indirect effect. Existing wors has developed causal mediation analysis-based methods to analyze an LLM’s logical reasoning capabilities [17] and to evaluate neuron-level security threats [26]. While prior efforts have shed light on reasoning and guardrail vulnerabilities, CMA in the specific domain of safety alignment remains underexplored. To the best of our knowledge, our work is the first to propose CMA to investigate the internal mechanisms governing harmful content generation and propagation within a model.
3
Methodology
In this section, we propose a causal mediation analysis-based approach to identify the internal mechanisms responsible for harmful content generation within LLMs. To enable causal mediation analysis of the internal mechanisms underlying LLM behavior, we represent the model as a causal computational graph. This representation makes explicit the causal dependencies between the input, internal activations, and the generated output, allowing internal model components to be treated as candidate mediators through which the causal effect of the input propagates. Formally, let an LLM be represented as a directed acyclic graph G = (V, E), where each node v ∈ V corresponds to a model component (e.g., layers, MLP, attention blocks, or neurons), and edges e ∈ E represent causal dependencies induced by the model’s computation. The input prompt X serves as the treatment variable, the generated output P as the probability distribution, and internal model components M ⊆ V as candidate mediators. Under this formulation, causal mediation analysis can be applied to quantify the extent to which the causal effect of the input X on the output is transmitted through specific internal components of the model. Specifically, the quantity of interest is the indirect effect (IE), defined as the component of the causal effect of the input prompt X on the model’s output probability distribution P that is transmitted through a mediator M , thereby quantifying how changes in
4
R Ganguli and R Moraffah
Fig. 1: Overview of the CMA approach. Baseline runs establish initial activations and token probabilities for both harmful and harmless prompts. Interventions are then performed on the harmful prompt using counterfactual activations captured from harmless baseline. Finally, the Indirect Effect (IE) is calculated to quantify the component’s causal impact on the model’s output distribution.
the input affect the output via the state of the internal component. The IE can be calcu∗ lated by measuring the shift from the model’s output distribution Phf toward the harmless baseline Phl . To accomplish this, we use the L1 distance (Total Variation Distance) to compare probability distributions, as it is symmetric and robust to zero probabilities. The L1 distance between two distributions P and Q over W is defined as: X |P − Q| = w ∈ W|P (w) − Q(w)|, (1) where, W is the model’s token vocabulary. The metric is symmetric and bounded [−2, 2]. For a given component (M ), the IE equals the difference between the baseline and mediated divergences: IE(M ) = |Phf − Phl | − |Phf∗ (M ) − Phl |,
(2)
where, |P hf − Phl | measures the baseline divergence of the model between the harmful and harmless runs of the model. The second term measures the divergence after the intervention. The difference between these terms captures the causal contribution of M on the output divergence. Specifically, a positive value for IE(M ) indicates that the component causally contributes to harmful behavior, such that replacing the component with its harmless run counterpart moves the output distribution closer to that of the harmless baseline, signifying that the component encodes information responsible for the harmful output. To determine the causal contribution driven by a mediator, we utilize a three-step process as depicted in Figure 1: (Step A) we create two baselines to obtain |P − Q|, one the baseline run of harmless prompt, and the other the baseline run of the harmful prompt; (Step B) we intervene on M ⊆ V for the second baseline run using the factual data obtained from the harmless baseline. We then calculate the distance of the distribution of this new run to obtain P ∗ hf (M ); and (Step C) we compare the distance of
Harmful Generation
5
this newly acquired distribution with the first run, to the distance acquired in step A to obtain IE(M ). To estimate the quantities of IE, we require a dataset that provides two factual data subsets in addition to one counterfactual. To accomplish this we utilize a set of prompt pairs of harmful and harmless prompts with minimal changes, allowing for a counterfactual subset in which the harmful subset utilizes the activations of the harmless. We define D as the dataset of both harmful and harmless prompt pairs, with Dhf ⊂ D as the subset of all harmful prompts and Dhl ⊂ D as the subset of all harmless prompts. We define xhl as a harmless prompt given to the model and xhf as a harmful prompt. We define Phl = M(xhl ) as the probability distribution over the model’s token vocabulary for the next token to be generated by the model when given the harmless prompt. Accordingly, we let Phf = M(xhf ) denote the probability distribution of the harmful prompt, which is meant to be rejected.
4
Experiments
We conduct comprehensive experiments using our approach to analyze model components that propagate harmful information and enable prompt subversion. The experiments are performed at differing levels of granularity within the LLMs: (1) Layer-wise analysis: identifies which internal layers contribute to its propagation using layer-wise analysis (subsection 4.2); (2) Component-wise analysis: determines how MLP and attention blocks influence harmful propagation through component-wise analysis (subsection 4.3); and (3) Neuron-wise analysis: measures the extent to which specific neurons causally drive harmful generation via neuron-wise analysis (subsection 4.4). In addition to analyzing the model’s internal components, we also examine the prompt tokens to identify token groups within the model that have a higher causal contribution to harmful generation. Moreover, we conduct a case study over the layers in the model that yield the greatest IE, as well as an LLM equipped with defense-aligned via internal steering to further investigate the LLM’s behavior under different. Finally, we leverage the identified internal locations within the model to provide a generic, generalizable defense for models. 4.1
Experimental Setup
Models. Following the existing work [8,17], we select a diverse set of state-of-the-art LLMs with varied parameters. To understand the impact of instruction-tuning, our selection includes models with varying parameter sizes and their corresponding instructiontuned variants. The final set of models utilized for the analysis is: Qwen2.5-7B [23], Qwen2.5-7B-Instruct [23], Qwen2.5-3B [23], Llama-3.2-1B [12], Llama-3.2-1B-Instruct [12], and Llama-3.2-3B [12]. Dataset. We utilize modified AdvBench dataset [7] which provides a harmless prompt and their corresponding harmful pair. The harmless counterfactuals were generated by an LLM via minimal keyword substitution, maintaining semantic equivalence while eliminating the content that triggers safety alignment refusal. This dataset is ideal for
6
R Ganguli and R Moraffah
our approach as it provides the harmful and harmless pairs as factual subsets, and could be used to create a counterfactual subset. Examples of the prompt pairs can be found in Appendix A. 4.2
Layer-wise Analysis
For the layer-wise analysis, we treat the entire layer ℓ as the mediator and perform a mediation analysis by simultaneously replacing all harmful activations within that layer, calculated as: IElayer (ℓ) = ∥Phf − Phl ∥ − ∥Phf∗ (layer ℓ) − Phl ∥
(3)
(ℓ)
The intervention replaces all hidden states {hi }L i=1 at layer ℓ with their harmless
Fig. 2: Results of layer wise interventions on the six models. Llama-3.2-1B has 16 layers, Llama-3.2-3B; Llama-3.2-3B-Instruct; Qwen2.5-7B; and Qwen2.5-7B-Instruct have 28 layers, and Qwen2.5-3B has 36 layers. The graph yields a noticeable trend for all models where the late layers in the model yield a higher IE, and that the final layers yield a relatively negative IE. Additional trends such as higher parameter count changing the spread of IE and fine tuning manipulates the magnification of IE.
counterparts. This calculates the average causal effect of a layer ℓ. We thereby measure the causal contribution of that specific processing stage, independent of individual token positions. This approach enables the analysis of architectural trends as well as identification of behaviors driven by initial information encoding versus final-stage decision processing. Results. As illustrated in Figure 2, there are three distinct behavioral patterns. The final
Harmful Generation
7
Fig. 3: Component level heatmaps of Qwen2.5-7B-Instruct. Indicates .15 IE in the late MLPs in the late layers, with strong negative IE in the final MLP across all layers. The attention heatmap in B displays a strong negative across all Attention blocks when utilized in layer ℓ = 0
layer (ℓ = L) exhibits negative IE values across all models; the middle layers display IE values approximately at the baseline of 0.0; and late layers of the model, exhibit IE values that increase substantially into positive territory. Furthermore, instructiontuned variants (e.g., Qwen2.5 7B Instruct and Llama-3.2-3B Instruct) show modified IE magnitudes compared to their base counterparts. We additionally observe that models with a higher parameter count exhibit greater positive IE across more layers, whereas models with fewer parameters display positive IE confined to a brief spike in late layers. Key Observations Our findings indicate three distinct causal roles across the model depth. Negative IE within the final layer indicates that intervening here “blinds" the model, preventing safety-related computations from activating and thereby increasing divergence. Moving deeper into the model, the layers exhibit minimal to zero IE, indicating they serve a neutral role, acting primarily as a propagation path without significantly mediating the final decision. Finally, the late layers show strong positive IE, indicating a shift from refusal to compliance allowing for the generation of harmful content.
4.3
Component-wise Analysis
For the component-wise analysis, we aim to understand the causal contribution of two vital components: the Multi-Layer Perceptron (MLP) block and the Attention block. Within each transformer layer ℓ, to determine the extent to which harmful content is propagated through the MLP blocks, we intervene on a specific MLP block at layer ℓ, and quantify this by computing the IE as: IEMLP (ℓ) = ∥Phf − Phl ∥ − ∥Phf∗ (MLPℓ ) − Phl ∥,
(4)
8
R Ganguli and R Moraffah
where M LP ℓ denotes the output of the MLP block before it is added back to the residual stream. We also isolate the causal contribution of attention mechanism, which is responsible for contextual routing. The IE for attention block output is formulated as: IEAttn (ℓ) = ∥Phf − Phl ∥ − ∥Phf∗ (Attnℓ ) − Phl ∥,
(5)
where Attnℓ represents the output of the attention block at layer ℓ. This comparative analysis allows us to definitively determine whether prompt subversion is primarily a consequence of contextual routing or feature transformation. Results. The analysis of the MLP and Attention blocks indicates whether the propagation and generation of harmful content are mediated by feature transformations (MLPbased) or by contextual attention routing (Attention-based). MLP Blocks. As shown in a representative heatmap (Figure 3), MLP blocks in the late layers consistently exhibit positive IE values across all models. In the Qwen2.5 series, the majority of MLP blocks in early and middle layers cluster around a neutral IE mean. The Llama-3.2 series displays a similar pattern, with most MLP blocks showing IE values in the range of 0.0 to −0.02. Crucially, MLP blocks at the boundary layers (ℓ = L) exhibit more pronounced negative IE values, a pattern especially stark in the Llama-3.2 series. This indicates that the late MLP blocks mediate the harmful content, with greater mediation in the late layers of the model. Attention Blocks (Figure 3) reveals that most layers display near-zero IE values (≈0.0), indicating a neutral causal role in the final harmful decision. However, the first layer (ℓ=0) demonstrates a notably strong negative IE across all models. The final layer also exhibits negative IE values, in addition to interventions utilizing the final attention block. Key Observations MLP blocks are the primary causal contributors to harmful behavior (strong positive IE in late layers), confirming that harmful content generation operates predominantly through feature transformations that drive compliance rather than through contextual attention routing. Attention blocks generally maintain a neutral causal role (IE ≈ 0.0). The strong negative IE observed in the initial attention layers indicates that these components establish the contextual understanding of the prompt’s harmfulness, propagating this information through the residual stream. These findings are consistent with the layer-level analysis (subsection 4.2) and confirm that late-layer MLP components are the primary causal drivers of harmful compliance. Further heatmaps can be found in Appendix D.
4.4
Neuron-wise Analysis
To achieve the highest level of granularity, we analyze the causal contribution of individual feature representations by partitioning the MLP’s hidden layers. Due to the high
Harmful Generation
9
dimensionality of the hidden states, we partition the hidden dimension into small, contiguous blocks and intervene on each independently. For neuron block k at layer ℓ the IE is defined as: IEneuron (ℓ, k) = ∥Phf − Phl ∥ − Phf∗ MLPℓ kstart :kend − Phl ,
(6)
where kstart and kend define the slice of the hidden dimension being intervened upon.
Fig. 4: Calculation of the average change in IE across the model when performing the neuron level interventions. Showcases similar trends to those seen at other granularity. The scale of the y-axis is 10−4 , as neurons are small relative to the activation values of components and layers.
For example, in a model with a hidden dimension of d = 2048 and a chosen block size of 2, this approach yields 1024 distinct neuron blocks per layer. This hierarchical decomposition, ensures computational feasibility while retaining high resolution; and it accounts for potential feature distribution, in which a specific behavior may be encoded redundantly across a small cluster of neurons rather than a single unit. It is important to note that this was designed with the assumption that functionally related neurons are in fact spatially adjacent [28]. Future iterations of this experiment should work without this assumption. Results. Consistent with the findings from the layer-level (subsection 4.2) and componentlevel analyses (subsection 4.3), the model’s late layers yield the highest positive IE values. However, unlike the previous granularity, we notice that, as indicated in Figure 4, the majority of models, while showing higher positive IE in the late layers, do not in fact have a statistically significant peak relative to the rest of the layers. Across all models, we still observe a great negative IE in the layers ℓ = L.
10
R Ganguli and R Moraffah
Key Observations The neuron-level analysis confirms that the harmful behavior is mediated by a highly sparse set of neurons, as intervening on individual neurons does not result in a statistically significant IE, whereas it is significant at higher-granularity levels, such as component- and layer-level. Furthermore, neuron blocks within the late and final layers remain consistent with results from the layer-level (subsection 4.2) and component-level (subsection 4.3) analyses, in that neurons in the late layers show a positive increase in IE. This indicates that within this location, there is an internal shift from refusal to compliance, whereas layer ℓ = L with negative IE suggests that interventions at this location disrupt necessary contextual information, moving the distribution further from the harmless distribution.
5
Token-wise Analysis
For our token-wise analysis of the input prompt, we examine both individual and grouped representations. To enable the analysis of localized causal contributions, prompt tokens are partitioned into four equal-sized groups based on their relative position in the input sequence: Beginning (first quartile), Middle (second quartile), Late (third quartile), and Final (fourth quartile).