arXiv:2605.15138v1 [cs.LG] 14 May 2026
Saisab Sadhu, Pratinav Seth, Vinay Kumar Sankarapu Lexsi Labs [email protected] Abstract Standard unlearning evaluations measure behavioral suppression in full precision, immediately after training, despite every deployed language model being quantized first. Recent work has shown that 4-bit post-training quantization can reverse machine unlearning; we show this is not a tuning artefact but a systematic dual failure: gradient-based methods that achieve meaningful forgetting lose it under compression, while methods that survive quantization barely change the model. Both failures trace to the same root cause: across all baselines, per-parameter updates lie 47–828× below the NF4 quantization bin width updates diffused across billions of parameters cannot clear quantization bin boundaries a consequence we formalize as a sparsity-permanence tradeoff. We present M ANSU (Mechanistic-Aligned Null-Space Unlearning), which resolves both modes by combining causal circuit attribution to isolate the minimal forget-set subgraph, circuit-restricted null-space projection with a diagonal-Fisher retain bound, and a per-parameter magnitude floor guaranteeing quantization survival by construction. We additionally introduce Circuit Attribution Divergence (CAD), a mechanistic verification metric distinguishing structural erasure from behavioral suppression, a distinction existing metrics cannot make. Across multiple model families and hazard benchmarks, M ANSU is the first method to jointly satisfy all four properties with margin on each (meaningful forgetting, retain preservation, non-positive PTQ gap, and structural erasure), while gradient-based baselines recover up to +0.05 accuracy under compression. Keywords: Machine Unlearning, Large Language Models, Model Quantization, Circuit Attribution, AI Safety
1
Introduction
Machine unlearning has become a safety-critical capability for deployed language models: hazardous-knowledge memorisation (biosecurity, cyberweapons, chemical synthesis) makes it necessary [1], and right-to-erasure regulations (EU AI Act, GDPR) make it legally required [2]. Yet every deployed LLM today is quantized 4-bit formats (NF4, GPTQ, AWQ) reduce memory by 4× and inference cost by 2–3×, making quantization the standard final step before release. [3] (ICLR 2025) documented that 4-bit PTQ can reverse machine unlearning, reporting up to 83% recovery and proposing a saliency-based mitigation (PTQ-LR/SURE); standard evaluation practice has not yet caught up the field’s default protocol remains behavioral metrics in full precision on a held-out forget set, measured immediately after training. We trace the reversal phenomenon to a structural cause (per-parameter updates systematically fall below the NF4 bin width) and propose a method that addresses it by construction. The assumption the standard protocol embeds behavioral suppression in BF16 is an adequate proxy for durable knowledge removal is false, and the failure is systematic. The dual failure mode: We apply six representative methods to Llama-3.1-8B-Instruct on WMDP-bio [1] and confirm that gradient-based methods achieve meaningful forget-set suppression in BF16. We then apply NF4 4-bit posttraining quantization (the compression scheme used by the overwhelming majority of real-world LLM deployments)
Quantization-Permanent Unlearning via Circuit Attribution
Table 1 Prior work against the four unlearning requirements (Section 3). F = forget; R = retain; Q = quant.-permanent (∆PTQ ≤ 0); S = structural erasure (CAD ≫ 0). ✓ satisfies; ✗ fails; ∼ partial. Methods(†) are evaluated in our experiments. Family
F
R Q S Key limitation
GA [2] Surgical GA† [2] NPO† [8] SimNPO† [9] GU+SimNPO† [10, 9] LUNAR† [11] PTQ-LR [3]
Grad. ascent Grad. ascent Pref. opt. Pref. opt. Null-space + Pref. Repr. steer. Quant.-aware
✓ ✓ ∼ ✗ ✗ ∼ ✓
∼ ∼ ✓ ✓ ✓ ✓ ∼
M ANSU (ours)
Circuit + floor
✓ ✓ ✓ ✓ First method jointly satisfying all four with margin on each
Method †
✗ ✗ ✓ ✓ ✓ ✗ ∼
✗ ✗ ✗ ✗ ✗ ✗ ✗
Updates diffuse over d params; per-param≪ δi Layer restriction reduces diffusion but cannot reach δi Frozen reference prevents updates ≥ δi Same frozen-anchor problem as NPO Global projection reinstates diffusion problem Edits non-circuit MLP projection; PTQ gap ≈ +0.01 (Table 2) Raises LR; retain constraint re-bounds max useful LR
and re-evaluate. In every gradient-based method, the forgotten knowledge returns, with PTQ recovery gaps of +0.06 to +0.07. Methods that survive quantization do so only by barely changing the model: across 94 non-M ANSU experiments (Table 9), preference-optimization and null-space methods reduce forget-set accuracy by 1.6 pp on average, within measurement variance on a four-way MCQ task. The pattern holds on Qwen-3-8B and on MUSE open-ended memorization, ruling out model- or benchmark-specific explanations. The structural cause: Both failure modes share one origin. Every existing method distributes gradient updates across all d parameters. For Llama-3.1-8B (d ≈ 8 × 109 ), even a large-norm gradient induces per-parameter changes of order 10−6 , far below the NF4 quantization bin width of ≈ 8.4 × 10−4 . At compression time, these changes round to zero. Methods that avoid this by constraining updates to remain near the original model do so at the cost of meaningful forgetting. This is not a hyperparameter problem; it is a necessary consequence of applying any gradient-based objective uniformly across billions of parameters (Proposition 1). The fix: Mechanistic interpretability has established that specific factual knowledge is causally localized in sparse, identifiable subgraphs of the model’s computation [4, 5, 6]. If knowledge p resides in |C| parameters rather than all d, concentrating updates into C amplifies per-parameter magnitudes by d/|C|. With an explicit magnitude floor, quantization survival becomes a construction-time guarantee. Null-space projection restricted to C yields a retain-set loss bound provably tighter than global projection by the Cauchy interlace theorem. We present M ANSU (Mechanistic-Aligned Null-Space Unlearning), which operationalizes this insight: (1) EAP-IG [7] identifies the minimal circuit C causally responsible for forget-set answers; (2) gradient updates within C are projected into the null space of the retain-set Fisher Information, with a tighter bound proved in Theorem 1; and (3) every cumulative update below the NF4 bin size is rescaled to the floor, guaranteeing quantization survival by construction (Lemma 1). On Llama-3.1-8B-Instruct / WMDP-bio, M ANSU achieves a PTQ gap of −0.040 while preserving MMLU within 0.030 of the zero-shot model: NF4 amplifies rather than reverses the erasure (Proposition 2). Results replicate on Qwen-3-8B and MUSE. Contributions: (I) Dual failure documentation: the first systematic evidence that no existing method achieves both meaningful forgetting and quantization permanence, across 94 non-M ANSU experiments (84 WMDP cells from the family-wise sweep in Table 9 plus 10 MUSE cells in Table 2) over three model families, three hazard domains, and two benchmarks. (II) M ANSU: a three-component method with formal guarantees, tighter retain bound (Theorem 1), construction-time quantization survival (Lemma 1), and a sparsity-permanence tradeoff analysis (Proposition 1); full proofs in Appendix C. (III) Circuit Attribution Divergence (CAD): the first post-hoc mechanistic verification protocol distinguishing structural knowledge deletion from behavioral suppression, a distinction standard behavioral metrics cannot make (Section 4.1).
2
Background and Related Work
Machine unlearning methods can be grouped into five method families (gradient ascent, preference optimization, null-space projection, representation steering, quantization-aware optimization); Table 1 summarizes each against the four requirements of Section 3. Gradient ascent and variants [2, 12] maximize forget-set loss directly. These methods are simple and effective in full precision, but updates distribute over all d parameters, pushing per-parameter magnitudes far below quantization bin widths. Surgical variants [2] reduce the active parameter count but cannot reach the bin threshold without violating the retain constraint (Proposition 1). 2
Quantization-Permanent Unlearning via Circuit Attribution
0.0%
0.0%
i
Zoom
i
95%
i
Density
Density
Density
i
i
8
6
4
log10 | w|
-3.4
-3.1
-2.8
6
4
i
2
0
8
6
4
log10 | w|
2
0
8
log10 | w|
2
0
Figure 1 Per-parameter update magnitudes (Llama-3.1-8B / WMDP-bio). Histograms of log10 |∆w|. (a) Global GA: diffuse, far below δi . (b) Surgical GA: concentrated on L14–16, still below δi . (c) M ANSU: clamped at or above δi by construction. Dashed line: NF4 bin width δi = 8.4×10−4 ; updates to its left round to zero under 4-bit quantization, so only M ANSU’s erasure survives (Lemma 1).
Preference optimization (NPO [8], SimNPO [9]) adapts DPO [13] to treat forget-set responses as dis-preferred. The frozen reference model prevents output collapse and incidentally prevents large per-parameter updates, giving good retain scores but negligible structural change. TOFU [14] and MUSE [15] are benchmark suites for preferenceoptimized unlearning, on fictitious-author facts and open-ended memorization respectively; we evaluate on MUSE alongside the WMDP hazard splits. Null-space projection (GU, 10) projects gradient updates onto the null space of the retain Hessian, giving a formal retain-safety bound. Because the projection is global, the diffusion problem is reinstated. M ANSU inherits the projection idea and proves a strictly tighter bound by restricting both the update and the projection to the causally identified circuit (Theorem 1). Representation steering (LUNAR [11], RMU [1]) suppresses forget-set outputs by redirecting activations at inference time. LUNAR trains only a single MLP down-projection outside the EAP-IG forget circuit; RMU randomises forget-set activations without weight edits. In both cases the causal knowledge circuit is left intact, so the unlearned model passes behavioural metrics while CAD remains ≈ 0 the failure mode CAD is designed to expose. We include LUNAR in our experiments and discuss RMU as a methodologically adjacent baseline. Quantization robustness: [3] (ICLR 2025) document that 4-bit PTQ can catastrophically reverse unlearning, reporting up to 83% recovery and proposing a saliency-based unlearning strategy with a large learning rate (“PTQ-LR” in Table 1) as mitigation. We show (Proposition 1) that the retain constraint independently caps the useful learning rate, so the root cause remains unaddressed. Our magnitude-floor constraint solves the problem at its source. Mechanistic interpretability and knowledge localization: ROME [4] and MEMIT [16] established via causal patching that factual associations are stored in middle MLP layers; EAP-IG [7] extends this to circuit-level attribution across the full computation graph. Concurrently, [17] apply circuit-restricted weight arithmetic to embed refusal directly into checkpoints without inference-time hooks. Our work applies the same localization principle to unlearning and adds the orthogonal constraint of quantization permanence, which that setting does not require. [18] and [19] raise concerns that attribution-based circuits do not reliably predict unlearning targets; Ablation C(i) tests this claim directly on the factual-recall benchmarks studied here and finds a substantial CAD advantage (1.143 vs 0.743) for the causally identified circuit over a random same-size baseline at matched forget depth. Extended discussion is in Appendix A.
3
Problem Formulation
Let θ ∈ Rd be a pretrained LM’s parameters, Df the forget set, Dr the retain set. We seek ∆θ with θ′ = θ + ∆θ satisfying four properties: (i) forget: θ′ fails on Df by a meaningful margin; (ii) retain: performance on Dr and general benchmarks within 2 pp of θ; (iii) quantization permanence: Q4 (θ′ ) also fails on Df , where Q4 is the deployment 4-bit quantizer; (iv) structural erasure: re-running causal attribution on θ′ shows the subgraph implementing forget-set knowledge has collapsed, not merely been bypassed. Properties (i) and (ii) are standard; (iii) and (iv) are not, and no existing method satisfies both. 3
Quantization-Permanent Unlearning via Circuit Attribution
Figure 2 M ANSU three-phase pipeline. Phase 1 (Localize): EAP-IG causal attribution identifies the minimal MLP circuit C causally responsible for forget-set answers. Phase 2 (Project): Updates restricted to C are projected into the null space of the circuit-restricted retain Fisher FC (Theorem 1). Phase 3 (Floor): A per-parameter magnitude floor δi rescales each update to clear the nearest NF4 bin boundary by construction (Lemma 1).
Definition 1 (NF4 quantization floor). Under NF4 quantization [20] with per-channel scale si and codebook levels {qk }15 k=0 , the smallest bin width for parameter i is δi = si · mink |qk − qk−1 |. For Llama-3.1-8B MLP weights δi ≈ 8.4 × 10−4 (derivation in Appendix D). Proposition 1 (Sparsity–permanence tradeoff). Under gradient ascent with retain constraint Lr (θ + ∆θ) − Lr (θ) ≤ ϵr , the per-parameter update magnitude when |C| parameters are updated (all others frozen) satisfies s ∥∆θi ∥ ≤
2ϵr , |C| F̄C
F̄C =
1 X [Fr ]jj , |C|
(1)
j∈C
where [Fr ]jj = E(x,y)∼Dr [(∂ log pθ (y|x)/∂θj )2 ] is the empirical diagonal Fisher of the retain loss (Appendix C; the diagonal Fisher remains well-defined under rank-deficient Hr , unlike the standard σmin form). For Llama-3.1-8B (d = 8.03×109 , ϵr = 0.02, F̄C ∼ 100 ), the global case (|C| = d) gives ∥∆θi ∥ ≲ 2.2×10−6 , roughly 380× below δi . Updates reach δi only when |C|/d ≤ 7×10−6 (fewer than 0.001% of parameters). Implications. First, no existing gradient-based method operates near this threshold: Surgical GA’s 6.6% circuit and even M ANSU’s 3.2% both sit more than three orders of magnitude above it (≈ 4500× for M ANSU, ≈ 9400× for Surgical GA; cf. Surgical GA’s +0.027 PTQ gap, Table 2), so localization alone is insufficient and the magnitude floor (Section 4) is required to close the gap by construction. Second, Proposition 1 says nothing about which parameters to update: arbitrary concentration damages retain performance, so the circuit must be chosen causally. Second failure mode. Preference-optimization methods (NPO, SimNPO, GU+SimNPO) avoid the floor problem differently: the frozen-reference KL constrains updates to be so small that |∆θi | ≪ δi almost everywhere. At standard hyperparameters this leaves forget accuracy largely intact across our 94-experiment sweep, the mean forget-set reduction for these methods is 1.6 pp on capable models (behaviorally invisible erasure). Pushing the methods harder (as in our main-table runs on Llama-3.1-8B) does move forget accuracy, but diffuses the now-larger update across d parameters: forget drops (0.230–0.250) come paired with collapsed MMLU (0.200–0.295) targeted erasure is replaced by global utility damage (Section 6). 4
Quantization-Permanent Unlearning via Circuit Attribution
4
Method
Both failure modes share a root cause: gradient updates distributed over parameters with no causal role in the targeted knowledge. M ANSU corrects this in three phases (Figure 2; full procedure in Algorithm 1); derivations are in Appendix B. Phase 1: Localize (Appendix E). EAP-IG [7] runs path-integrated gradients on the logit difference between clean and corrupted forget-set prompts, attributing causal contribution to each edge of the transformer graph. Aggregating over 50 forget examples and ranking MLP sublayers by total incoming attribution mass yields the top-10 circuit: CMLP = {30, 14, 31, 19, 29, 15, 20, 16, 21, 17},
(2)
covering ≈ 3.2% of parameters (effective post-Phase-2/3 fraction; per-stage breakdown in Appendix B.3). The top-5 prefix {30, 14, 31, 19, 29} is the canonical k = 5 configuration used in Tables 10 and 12. Layer 14 appears in both the EAP-IG top-K circuit and surgical GA’s L14–16 selection, providing partial cross-method agreement; upper layers {29,30,31} dominate the attribution ranking, consistent with ROME’s finding that later MLP layers store factual associations [4]. Phase 2: Project (Appendix B). Gradient updates within C are masked along high-Fisher coordinates, an approximation to projection into ker(HCC ) under the diagonal-Fisher assumption (approximation error bounded in Proposition 3): [PC g]i = gi · 1[[FC ]ii ≤ τ ] , i ∈ C; ∆θC̄ = 0, (3) where τ is the 99th-percentile Fisher threshold and all parameters outside C are frozen. Restricting projection to C yields a provably tighter retain bound than projecting globally (Theorem 1). Phase 3: Floor (Appendix D). After training converges (best checkpoint by lowest forget accuracy subject to MMLU drop ≤ 0.08), the magnitude floor is applied post-hoc to the saved checkpoint: for each i ∈ C, the cumulative (0) update ∆θi = θi − θi is rescaled to clear the nearest NF4 bin boundary while preserving direction: ∆θi ← ∆θi · (0)
δi |∆θi |
whenever 0 < |∆θi | < δi , i ∈ C.
(4)
(0)
By Lemma 1 this guarantees Q4 (θi + ∆θi ) ̸= Q4 (θi ) for every i ∈ C, so the update is permanent under quantization. The implementation uses a per-tensor approximation of δi that agrees with Definition 1 to within an order of magnitude (Appendix B.3). Training objective. The three constraints are encoded jointly: min
|C|
∆θC ∈PC (R ) |∆θi |≥δi ∀i∈C ∆θC̄ =0
−Lf (θ + ∆θ) + λ DKL pθ(0) pθ+∆θ x∼Dr .
(5)
The frozen-reference KL (following NPO/GU) prevents retain collapse. Hyperparameters and the rationale for fullparameter (not LoRA) training are in Appendix B.
4.1
Circuit Attribution Divergence (CAD)
′ Motivation. Two unlearned checkpoints with identical forget-set accuracy can differ in mechanism: in θA the ′ knowledge circuit has been dismantled; in θB the circuit is intact and a downstream layer redirects its output to a ′ refusal token (LUNAR-style). Both pass behavioral evaluations, but θB is fragile to small fine-tunes, re-prompts, or quantization. Behavioral metrics measure outputs; unlearning is a claim about weights.
Definition. Let E(C) be the EAP-IG edge set on the original θ with attribution score se (θ) for edge e (Appendix E). Re-run EAP-IG on the unlearned θ′ and compare: P ′ e∈E(C) |se (θ) − se (θ )| ′ P CAD(C, Df ; θ, θ ) = . (6) e∈E(C) |se (θ)| CAD → 0 means the circuit is intact (behavior may have changed only via downstream redirection); CAD ≈ 1 means it has been dismantled; values > 1 indicate sign-flipped redirection (also structural). Properties. CAD is (i) computed entirely on the unlearned weights with no held-out probes; (ii) ≈ 0 by construction for inference-time redirection (LUNAR/RMU); (iii) insensitive to spurious behavioral suppression (a refuse-everything 5
Quantization-Permanent Unlearning via Circuit Attribution
model yields CAD ≈ 0); (iv) not satisfied by random weight perturbation the random-circuit control (Ablation C(i)) collapses CAD by ∼ 35% relative to the EAP-IG circuit (1.143 → 0.743 on WMDP-bio); (v) CAD alone does not certify structural erasure high CAD with elevated AS-NC indicates broad representational damage rather than localized circuit dismantling. The joint diagnostic is high CAD and low AS-NC (companion metric below); a worked SimNPO/MUSE example illustrating this distinction is in Appendix N. Companion metrics: AS-C, AS-NC. Activation-level checks inside / outside C (Eq. 11). Structural erasure requires high CAD and the concentration gap AS-C ≪ CAD, which is present only for localized methods; for global baselines AS-C = CAD numerically (Table 3). Full diagnostic discussion is in Appendix N. Algorithm 1 M ANSU (Mechanistic-Aligned Null-Space Unlearning) Require: pretrained θ(0) , forget set Df , retain set Dr , circuit size K, KL weight λ, floor δi 1: C ← top-K MLP sublayers by EAP-IG attribution mass on Df 2: [FC ]ii ← EDr (∂ log pθ(0) /∂θC,i )2 ; τ ← 99th percentile of [FC ]ii 3: θ ← θ(0) 4: for t = 1, . . . , T do 5: g ← −∇C Lf (θ) + λ ∇C DKL (pθ(0) ∥ pθ )Dr 6: ĝi ← gi · 1[[FC ]ii ≤ τ ] 7: θC ← θC − η ĝ; θC̄ frozen 8: end for (0) 9: for i ∈ C with 0 < |θi − θi | < δi do (0) (0) 10: θi ← θi + δi · sign(θi − θi ) 11: end for 12: return θ
5
▷ Phase 1: Localize
▷ Phase 2: training loop ▷ Phase 2: project (Fisher mask) ▷ Phase 3: floor (post-hoc)
Theoretical Analysis
M ANSU rests on three guarantees: retain safety, quantization permanence, and amplification. Full proofs and error bounds are in Appendix C. Theorem 1 (Circuit-restricted projection tightens the retain bound). Let Lr be twice continuously differentiable with PSD Hessian H. For C ⊆ [d], C¯ = [d] \ C, and any ∆θ with ∆θC ∈ ker(HCC ), ∆θC̄ = 0, ∥∆θ∥ ≤ ε: ε2 Lr (θ + ∆θ) − Lr (θ) ≤ ∥∇C Lr (θ)∥ ε + σmax (H ) + O(ε3 ). | {z } 2 | {z C̄ C̄} ≤ ∥∇Lr (θ)∥
(7)
≤ σmax (H)
Each bracketed term is at most its global counterpart: the gradient inequality is the sub-vector L2 bound, and σmax (HC̄ C̄ ) ≤ σmax (H) is Cauchy interlace [21]. The circuit-restricted bound is strictly tighter than global null-space projection [10] whenever H’s dominant eigenvector projects non-trivially onto C-coordinates. Since C is chosen by causal attribution on Df (not Dr ), this holds generically; Ablation D (global projection + floor) verifies it empirically. The diagonal-Fisher approximation used in Phase 2 incurs additional error O(σmax (H) ∥EC ∥op /τ ) where EC is the off-diagonal Fisher block (Appendix C). Lemma 1 (Quantization survival). Let Q4 be 4-bit quantization with monotone levels {qk } and let wi be the bin width at θi . Any update |∆θi | ≥ wi changes the quantized value: Q4 (θi + ∆θi ) ̸= Q4 (θi ). Setting δi ≥ wi in Phase 3 makes this a construction-time guarantee. Proposition 2 (NF4 amplifies floor-crossing updates). Let θi lie in a narrow-bin region of the NF4 grid (near zero; see Appendix D, Table 5) and let |∆θi | ≥ δi . When the update crosses two or more bin boundaries (m ≥ 2, automatic since |qk+m − qk | ≥ mδi ), |Q4 (θi + ∆θi ) − θi | ≥ |∆θi |: quantization amplifies displacement rather than attenuating it, producing a negative PTQ gap. For single-crossing updates (m = 1) deposited at the bin boundary by the floor, the amplification holds in expectation rather than with high probability. Conversely, for diffuse methods with |∆θi | < δi , the update does not cross any bin boundary and is silently erased by NF4, the +0.06 to +0.07 PTQ gap regime. Summary: Theorem 1 (retain safety) + Lemma 1 (quantization permanence) + Proposition 2 (amplification) together explain why M ANSU is the only method in Table 2 with margin on all four properties forget depth comparable to NPO, ∆PTQ ≤ 0 across every cell, MMLU preserved, and CAD ≫ AS-NC. 6
Quantization-Permanent Unlearning via Circuit Attribution
Table 2 Behavioral results. Forget set, retain split, and general capability for every method on Llama-3.1-8B-Instruct and Qwen-3-8B across four benchmarks. Columns: BF16 (↓) = forget-set accuracy in full precision (lower is better); NF4 (↓) = same forget set re-evaluated after 4-bit NF4 post-training quantization via bitsandbytes; ∆PTQ = accNF4 − accBF16 the quantization-permanence metric we propose; negative values mean NF4 amplifies the erasure rather than reversing it (Lemma 1, Proposition 2); Rt/Util (↑) = WMDP retain-split accuracy (WMDP rows) or utility score (MUSE); MMLU/IFEval = model-level capability and instruction following, dataset-independent. Companion structural metrics (CAD, AS-C, AS-NC) are reported separately in Table 3 to keep behavior and mechanism visually distinct. – entries are not applicable: zero-shot NF4 and ∆PTQ are omitted because the unmodified model is not quantized as part of unlearning evaluation (PTQ gap would be vacuously zero); MUSE Rt/Util is additionally omitted because the utility score is only defined post-unlearning. Llama-3.1-8B-Instruct Method
Qwen-3-8B
BF16 (↓) NF4 (↓) ∆PTQ (↓) Rt/Util (↑) MMLU (↑) IFEval (↑)
BF16 (↓) NF4 (↓) ∆PTQ (↓) Rt/Util (↑) MMLU (↑) IFEval (↑)
WMDP-bio Zero-shot Global GA Surgical GA NPO SimNPO GU+SimNPO LUNAR
0.763 0.260 0.547 0.443 0.250 0.230 0.621
– 0.310 0.573 0.423 0.250 0.230 0.638
– +0.050 +0.027 -0.020 +0.000 +0.000 +0.017
0.763 0.260 0.560 0.503 0.210 0.247 0.619
0.603 0.235 0.483 0.563 0.295 0.200 0.571
0.560 0.536 0.528 0.528 0.528 0.532 0.544
0.803 0.233 0.260 0.283 0.227 0.267 0.658
– 0.233 0.247 0.320 0.227 0.263 0.671
– +0.000 -0.013 +0.037 +0.000 -0.003 +0.013
0.803 0.247 0.303 0.303 0.257 0.277 0.655
0.741 0.242 0.458 0.492 0.265 0.568 0.612
0.548 0.408 0.428 0.416 0.412 0.420 0.531
M ANSU (ours)
0.430
0.390
-0.040
0.523
0.573
0.551
0.617
0.581
-0.036
0.671
0.729
0.541
WMDP-chem Zero-shot Global GA Surgical GA NPO SimNPO GU+SimNPO LUNAR
0.533 0.493 0.427 0.253 0.233 0.273 0.481
– 0.473 0.423 0.227 0.233 0.273 0.497
– -0.020 -0.003 -0.027 +0.000 +0.000 +0.016
0.533 0.491 0.426 0.269 0.241 0.231 0.479
0.603 0.550 0.525 0.538 0.195 0.230 0.571
0.560 0.540 0.548 0.532 0.552 0.536 0.544
0.560 0.237 0.313 0.237 0.240 0.233 0.521
– 0.293 0.317 0.237 0.277 0.233 0.534
– +0.057 +0.003 +0.000 +0.037 +0.000 +0.013
0.560 0.269 0.398 0.296 0.259 0.250 0.518
0.741 0.365 0.557 0.515 0.405 0.328 0.612
0.548 0.424 0.436 0.432 0.412 0.424 0.531
M ANSU (ours)
0.333
0.307
-0.027
0.398
0.584
0.549
0.307
0.274
-0.033
0.364
0.714
0.539
WMDP-cyber Zero-shot Global GA Surgical GA NPO SimNPO GU+SimNPO LUNAR
0.477 0.357 0.283 0.340 0.270 0.300 0.431
– 0.370 0.290 0.343 0.277 0.300 0.445
– +0.013 +0.007 +0.003 +0.007 +0.000 +0.014
0.477 0.390 0.367 0.400 0.273 0.230 0.428
0.603 0.543 0.480 0.568 0.195 0.297 0.571
0.560 0.532 0.536 0.544 0.532 0.540 0.544
0.537 0.360 0.430 0.360 0.270 0.333 0.501
– 0.377 0.467 0.393 0.403 0.443 0.514
– +0.017 +0.037 +0.033 +0.133 +0.110 +0.013
0.537 0.370 0.473 0.437 0.280 0.267 0.498
0.741 0.463 0.710 0.715 0.555 0.715 0.612
0.548 0.416 0.428 0.432 0.452 0.424 0.531
M ANSU (ours)
0.323
0.313
-0.010
0.391
0.586
0.549
0.497
0.464
-0.033
0.541
0.721
0.542
MUSE Zero-shot Global GA Surgical GA NPO SimNPO GU+SimNPO LUNAR
0.365 0.000 0.000 0.013 0.000 0.000 0.187
– 0.000 0.000 0.016 0.000 0.000 0.198
– +0.000 +0.000 +0.002 +0.000 +0.000 +0.011
– 0.000 0.000 0.013 0.000 0.000 0.184
0.603 0.570 0.583 0.553 0.539 0.575 0.571
0.560 0.484 0.552 0.544 0.540 0.464 0.544
0.024 0.000 0.020 0.011 0.001 0.013 0.162
– 0.009 0.017 0.021 0.013 0.017 0.171
– +0.009 -0.003 +0.009 +0.012 +0.004 +0.009
– 0.007 0.020 0.011 0.000 0.005 0.159
0.741 0.700 0.723 0.728 0.723 0.718 0.612
0.548 0.452 0.424 0.428 0.448 0.432 0.531
M ANSU (ours)
0.005
0.003
-0.002
0.006
0.591
0.547
0.021
0.017
-0.004
0.019
0.737
0.436
6
Experiments
We answer three questions: does M ANSU resolve the dual failure mode, is each component necessary, and is the forgetting structural? Setup, hyperparameters, timing, update statistics, and extended ablations are deferred to Appendices F–N. Setup: Llama-3.1-8B-Instruct on WMDP-bio [1] for the main table (Table 2); M ANSU is additionally evaluated on MUSE [15] (Harry Potter open-ended memorization) and Qwen-3-8B (to assess architecture generalization, Qwen3-8B columns of Table 2). A separate baseline sweep on six small/mid models (Gemma, Llama, Qwen families) on WMDP-{bio, chem, cyber} tests cross-architecture generality (Appendix J). Fixed forget and MMLU indices are reused across methods. NF4 evaluation via bitsandbytes (4-bit, double-quantization off); ∆PTQ = accNF4 − accBF16 is the primary quantization metric. Six baselines: Global GA, Surgical GA (L14–16), NPO, SimNPO, GU+SimNPO, and LUNAR. Main results: All findings read off the WMDP-bio Llama-3.1-8B block of Table 2 (zero-shot 0.763); the perproperty scorecard in Figure 3 summarises pass/fail across all 24 weight-edit (method, dataset) cells (6 weight-edit 7
Quantization-Permanent Unlearning via Circuit Attribution
Table 3 Structural erasure metrics (companion to Table 2). CAD (↑) (Eq. 6): relative collapse of EAP-IG attribution mass on the original forget circuit; → 1 = full collapse, → 0 = circuit intact (LUNAR-style redirection: empirically ≈ 0.03–0.05 across all WMDP/MUSE cells, Table 3; near-zero by construction since LUNAR edits a single MLP projection outside the EAP-IG forget circuit). AS-C / AS-NC (↓) (Eq. 11): activation shift inside / outside C. Structural erasure requires high CAD and the gap AS-C ≪ CAD (present only for localized methods); for global baselines AS-C = CAD numerically. WMDP-bio Method
WMDP-chem
WMDP-cyber
MUSE
CAD (↑) AS-C AS-NC (↓) CAD (↑) AS-C AS-NC (↓) CAD (↑) AS-C AS-NC (↓) CAD (↑) AS-C AS-NC (↓) Llama-3.1-8B-Instruct
Zero-shot Global GA Surgical GA NPO SimNPO GU+SimNPO LUNAR
0.000 0.523 0.321 0.849 1.433 1.292 0.041
0.000 0.523 0.321 0.849 1.433 1.292 1.187
0.000 0.311 0.173 0.509 0.870 0.824 0.312
0.000 0.400 0.356 0.836 1.351 1.292 0.033
0.000 0.400 0.356 0.836 1.351 1.292 0.974
0.000 0.282 0.172 0.570 0.875 0.833 0.256
0.000 0.440 0.248 0.356 1.523 1.366 0.029
0.000 0.440 0.248 0.356 1.523 1.366 0.897
0.000 0.358 0.185 0.336 1.033 0.984 0.236
0.000 1.660 1.599 1.635 1.979 1.522 0.045
0.000 1.660 1.599 1.635 1.979 1.522 1.248
0.000 1.187 0.469 1.150 1.104 1.256 0.328
M ANSU (ours)
1.143 0.412
0.138
1.097 0.398
0.141
1.118 0.387
0.143
1.671 0.318
0.097
Zero-shot Global GA Surgical GA NPO SimNPO GU+SimNPO LUNAR
0.000 1.021 0.663 0.847 0.911 0.792 0.039
0.000 1.021 0.663 0.847 0.911 0.792 1.143
0.000 0.660 0.356 0.550 0.609 0.544 0.301
0.000 1.057 0.612 0.773 0.966 0.793 0.031
0.000 1.057 0.612 0.773 0.966 0.793 0.937
0.000 0.789 0.325 0.551 0.724 0.543 0.247
0.000 0.782 0.262 0.624 0.630 0.771 0.027
0.000 0.782 0.262 0.624 0.630 0.771 0.863
0.000 0.569 0.208 0.500 0.385 0.544 0.227
0.000 1.106 1.090 1.230 1.299 1.112 0.043
0.000 1.106 1.090 1.230 1.299 1.112 1.201
0.000 0.721 0.568 0.770 0.828 0.757 0.316
M ANSU (ours)
1.089 0.531
0.224
1.041 0.487
0.198
1.053 0.478
0.201
1.143 0.441
0.182
Qwen-3-8B
Table 4 Component ablation on Llama-3.1-8B-Instruct / WMDP-bio (zero-shot 0.763). Each row removes or replaces one M ANSU component; all others held fixed. Bold = best per column. Each component isolates a distinct mechanism; the same pattern reproduces on chem and cyber (selected ∆PTQ /CAD numbers quoted in the prose below). Row D uses GU-global (no SimNPO), a weaker forget baseline than GU+SimNPO in Table 2; its higher BF16 is expected. Configuration M ANSU (full) A: w/o magnitude floor B: w/o null-space proj. C(i): random circuit (seed 42) C(ii): inverse circuit (bottom-k) D: GU-global + floor
BF16 (↓) ∆PTQ (↓) Rt (↑) MMLU (↑) CAD (↑) AS-NC (↓) 0.430
-0.040
0.523
0.573
1.143
0.138
0.513 0.451 0.500 0.551 0.697
-0.008 -0.019 -0.024 +0.028 +0.013
0.489 0.461 0.471 0.441 0.448
0.471 0.449 0.470 0.451 0.443
1.091 1.063 0.743 0.511 0.672
0.194 0.201 0.394 0.481 0.441
methods × 4 datasets, both flagship models pooled; LUNAR is omitted from the scorecard since its inference-time redirection is reported separately). Gradient ascent fails quantization: Global GA’s BF16 forget 0.260 flips to NF4 0.310 (∆PTQ = + 0.050) with MMLU collapsing to 0.235 indiscriminate damage, not targeted erasure (Figure 1). Aggressive preference optimization survives quantization but destroys utility: SimNPO/GU+SimNPO reach forget 0.250/0.230 with ∆PTQ =0.000 but MMLU 0.295/0.200; NPO preserves MMLU (0.563) at the cost of half M ANSU’s forget depth. M ANSU satisfies all three properties: forget 0.430, NF4 0.390, ∆PTQ = − 0.040, MMLU 0.573 (within 0.030 of zero-shot) IFEval 0.551 NF4 amplifies the erasure (Proposition 2). Structural metrics confirm weight-level rather than behavioral erasure: M ANSU attains the highest CAD (1.143) with low AS-NC spillover (0.138); LUNAR yields CAD ∈ [0.029, 0.045] across all WMDP/MUSE cells (Table 3), consistent with editing weights outside the EAPIG forget circuit. Cross-dataset / cross-architecture consistency. M ANSU’s ∆PTQ is non-positive on all 8/8 (model, dataset) cells of Table 2; MMLU stays within 0.030 of zero-shot across cells; CAD exceeds 1.0 on 7/8 cells (Table 3). The pattern extends beyond the two flagship 8B models: Tables 6, 7, and 8 report M ANSU on six additional model variants (Gemma-2B/3-1B/3-4B, Llama-3.2-3B, Qwen-2.5-4B/3-4B), and Table 9 the family-wise macro-averages — M ANSU achieves strictly negative ∆PTQ on every cell of every sweep family. By contrast, no baseline beats M ANSU on all three of forget, quantization-permanence, and utility on any cell the dual failure mode (gradient methods recover under NF4 / 8
Quantization-Permanent Unlearning via Circuit Attribution
Figure 3 The four-property scorecard, decomposed. x: forget delta (all 24 = 6×4 weight-edit method-dataset cells; LUNAR excluded as inference-time redirection); y varies per panel: (a) ∆PTQ , (b) MMLU loss, (c) AS-NC. Bottom (green) is desired in all three. Marker colour = method, shape = dataset; solid = passes all four thresholds (F ≥ 30%, Q ≤ 0, R ≤ 0.15, S ≤ 0.30), hollow = fails at least one. M ANSU is the only method solid in every panel.
preference methods barely change the model) holds across WMDP-bio/chem/cyber, MUSE, and both Llama-3.1-8B and Qwen-3-8B. Ablations: Table 4 reports each component independently on WMDP-bio (M ANSU full: forget 0.430, ∆PTQ = −0.040,MMLU 0.573 (within 0.030 of zero-shot), CAD 1.143). A, no floor: ∆PTQ weakens from −0.040 to −0.008 and forget accuracy regresses to 0.513, isolating the floor as the mechanism turning circuit concentration into quantization permanence. B, no null-space projection: forget accuracy regresses to 0.451 and MMLU drops to 0.449 (largest utility hit of any row), confirming projection sharpens the forget–retain tradeoff and is the primary retain-protector (Theorem 1). C(i), random circuit (seed 42): same |C|, but forget accuracy regresses to 0.500 and CAD collapses from 1.143 to 0.743 (−35%); AS-NC nearly triples (0.138 → 0.394), indicating diffuse rather than localized intervention. Forget quality — not just depth requires the causally identified circuit, directly rebutting [18] and [19] on the factual-recall benchmarks studied here. C(ii), inverse circuit (bottom-k): the strongest negative control ∆PTQ flips to +0.028, forget regresses to 0.551, and CAD bottoms out at 0.511 across all three domains, ruling out any beneficial effect from non-causal parameters. D, global null-space + floor (GU-global): ∆PTQ flips positive (+0.013) despite the floor, because diffuse global updates contain mixed-sign components that cancel below the bin floor under NF4 rounding. Circuit localization is therefore a necessary co-condition for quantization-robust erasure, not a substitute for the floor. Cross-domain consistency. The pattern reproduces on WMDP-chem and WMDP-cyber: Row A’s ∆PTQ flips to +0.004/ + 0.003 (vs full M ANSU −0.027/ − 0.010); Row C(i)’s CAD collapses to 0.711/0.729 (vs 1.097/1.118); Row D’s ∆PTQ stays positive (+0.009/ + 0.007). Each component contributes the same way across all three hazard domains.
7
Discussion
Implications for evaluation practice. The 94 non-M ANSU experiments show that the standard protocol selects for methods that make minimal parameter changes. A method that reduces forget-set accuracy by 1.6 percentage points is not solving the problem; it is passing the test. We propose two additions: the PTQ gap, and CAD (or an equivalent mechanistic verification). Neither requires new infrastructure. Limitations. M ANSU is reported on the two flagship 8B models from the Llama and Qwen families; results on smaller and earlier-generation models follow the same three-phase pipeline and are reported in Appendix J. Mechanistic localization is well-supported on factual-recall benchmarks of the kind studied here [4]; behaviour beyond the 8B √ regime (|∆θi | ∝ 1/ d at fixed circuit fraction by Eq. (1)) is consistent with the floor remaining the binding mechanism but is not directly verified.
8
Conclusion
If a model passes its unlearning evaluation in full precision, does it still pass after the compression step that precedes every real-world deployment? Across six methods and a single deployment compression pass, the answer is no. The forgotten knowledge returns, or it never left because the model barely changed. M ANSU resolves this by asking the prior question mechanistic interpretability has already answered: where does the targeted knowledge live? Updating only the causally identified circuit, projecting away from retain-sensitive directions, and rescaling every update to clear the NF4 floor produces forgetting that is not reversed by the compression step; in fact, NF4 amplifies the erasure (PTQ gap 9
Quantization-Permanent Unlearning via Circuit Attribution
−0.040 on WMDP-bio Llama, with preserved MMLU and IFEval). The 94-experiment dual-failure documentation, the CAD verification metric, and the sparsity-permanence framework should outlast any specific method: future approaches that satisfy all four properties of Section 3 must engage with this tradeoff.
9
Broader Impact and Ethics
This work is motivated by safety objectives in AI deployment. Durably removing hazardous knowledge (biosecurity threats, cyberweapons, chemical weapons) is a safety-critical requirement as language models become more widely deployed. Our finding that existing unlearning methods fail under 4-bit compression is information that practitioners and policymakers relying on unlearning for safety certification need to know. Misuse. This work does not make hazardous knowledge easier to acquire. We demonstrate that existing methods are weaker than they appear; we do not provide tools for recovering knowledge from unlearned models. EAP-IG is a published open-source method we use for removal, not insertion. Scope. Experiments cover bio, chem, and cyber hazard domains on MCQ. Generalization to other hazard types or open-ended formats requires additional validation. We do not claim M ANSU is a complete solution to knowledge removal; it is meaningfully better than existing alternatives on the metrics we define. Reproducibility. All datasets (WMDP, MMLU) are public. EAP-IG is at github.com/hannamw/EAP-IG. Our implementation, evaluation scripts, and fixed evaluation indices will be released upon acceptance.
References [1] Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-Voss, Cort B Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam Alfred Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Karmakar, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The WMDP benchmark: Measuring and reducing malicious use with unlearning. In Forty-first International Conference on Machine Learning, 2024. [2] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408, Toronto, Canada, July 2023. Association for Computational Linguistics. [3] Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, and Suhang Wang. Catastrophic failure of LLM unlearning via quantization. In The Thirteenth International Conference on Learning Representations, 2025. [4] Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. [5] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfieldDodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022. [6] Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen, editors, Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 407–416, Miami, Florida, US, November 2024. Association for Computational Linguistics. [7] Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In Conference on Language Modeling (COLM), 2024. Introduces EAP-IG (Edge Attribution Patching with Integrated Gradients). [8] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling, 2024. 10
Quantization-Permanent Unlearning via Circuit Attribution
[9] Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. Simplicity prevails: Rethinking negative preference optimization for LLM unlearning. In Neurips Safe Generative AI Workshop 2024, 2024. [10] Zhehao Huang, Xinwen Cheng, JingHao Zheng, Haoran Wang, Zhengbao He, Tao Li, and Xiaolin Huang. Unified gradient-based machine unlearning with remain geometry enhancement. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc. [11] William F. Shen, Xinchi Qiu, Meghdad Kurmanji, Alex Iacob, Lorenzo Sani, Yihong Chen, Nicola Cancedda, and Nicholas D. Lane. LLM unlearning via neural activation redirection. In Advances in Neural Information Processing Systems (NeurIPS), 2025. [12] Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Proceedings of the First Conference on Lifelong Learning Agents (CoLLAs), 2022. [13] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. [14] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling, 2024. [15] Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine unlearning six-way evaluation for language models. In The Thirteenth International Conference on Learning Representations, 2025. [16] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. [17] Aditya Kasliwal, Pratinav Seth, and Vinay Kumar Sankarapu. C-δθ: Circuit-restricted weight arithmetic for selective refusal. arXiv preprint arXiv:2602.04521, 2026. [18] Hwiyeong Lee, Uiji Hwang, Hyelim Lim, and Taeuk Kim. Does localization inform unlearning? a rigorous examination of local parameter attribution for knowledge unlearning in language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 21868–21880, 2025. [19] Phillip Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Mechanistic unlearning: robust knowledge unlearning and editing via mechanistic localization. In Proceedings of the 42nd International Conference on Machine Learning, ICML’25. JMLR.org, 2025. [20] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized llms. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. [21] Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, USA, 2nd edition, 2012. [22] Xiaoyu Xu, Xiang Yue, Yang Liu, Qingqing Ye, Huadi Zheng, Peizhao Hu, Minxin Du, and Haibo Hu. Unlearning isn’t deletion: Investigating reversibility of machine unlearning in LLMs, 2026. [23] Chandler Davis and W. M. Kahan. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1–46, 1970. [24] James Martens. New insights and perspectives on the natural gradient method. J. Mach. Learn. Res., 21(1), January 2020. [25] Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical fisher approximation for natural gradient descent. Curran Associates Inc., Red Hook, NY, USA, 2019. [26] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. [27] Minyeong Choe, Haehyun Cho, Changho Seo, and Hyunil Kim. Do all autoregressive transformers remember facts the same way? a cross-architecture analysis of recall mechanisms. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28494–28513, Suzhou, China, November 2025. Association for Computational Linguistics.
11
Quantization-Permanent Unlearning via Circuit Attribution
A
Extended Related Work
Gradient ascent and variants. Gradient ascent (GA) [2] maximizes the loss on forget-set examples directly, reversing the effect of gradient descent on those examples. Gradient Difference (GD) [12] adds a retain-loss minimization term. KL-regularized GA [14] constrains the unlearned model to remain close to the original in output distribution on the retain set. These methods achieve real forgetting in full precision; their structural problem is not the loss function but the update distribution. Preference optimization. NPO [8] and SimNPO [9] treat forget-set completions as negative preferences in a DPO-style objective [13], training the model to assign lower probability to forget-set answers relative to a frozen reference. The same anchor that prevents catastrophic collapse limits per-parameter updates well below δi by the same diffusion argument as for GA, so the floor is not crossed but neither is forget accuracy substantially altered. Across our 94-experiment sweep, mean forget-set reduction for capable models is 1.6 percentage points; the large majority of runs reduce forget accuracy by less than 5 percentage points. Null-space projection. GU [10] projects gradient updates onto the null space of the retain-set Hessian, theoretically bounding retain perturbation. Combined with SimNPO it represents the strongest principled baseline. The structural difference from M ANSU is scope: GU applies projection globally over d parameters, which forces the update to remain globally small and reinstates the diffusion problem; M ANSU restricts updates to the causal circuit C and combines this with a KL retain anchor and a magnitude-floor constraint. Theorem 1 formalizes the retain bound under circuit-restricted updates. Representation steering. LUNAR [11] redirects intermediate activations toward regions of activation space associated with model inability to answer, training only a single MLP down-projection that is outside the EAP-IG forget circuit. RMU [1] randomizes activations of forget inputs without weight edits. In both cases the causal knowledge circuit is left intact, so the unlearned model passes behavioral metrics while CAD remains ≈ 0 the failure mode CAD (Section 4.1) is designed to detect. Quantization robustness of unlearning. [3] (ICLR 2025) document that 4-bit PTQ can catastrophically reverse unlearning, reporting up to 83% recovery and attributing the cause to small per-parameter update magnitudes; their proposed mitigation does not resolve the structural cause because the retain constraint independently bounds the maximum useful learning rate. Mechanistic interpretability and knowledge localization. [4] demonstrate causal patching evidence that factual associations in GPT-style models are stored predominantly in middle MLP layers; MEMIT [16] extends this to batch editing. [5] show that features are represented in superposition across neurons, motivating circuit-level interventions. EAP-IG [7] combines activation patching with integrated gradients for lower-variance attribution. Across this body of work, the localization insight has not previously been connected to machine unlearning. Editing-as-unlearning, with amplification as a design principle. ROME [4] and MEMIT [16] localize factual associations and edit them to insert knowledge; M ANSU edits the same structures to remove it structurally symmetric operations separated by a localization insight unlearning had not exploited. The negative PTQ gap further suggests calibrating δi to the deployment quantization scheme (NF4 today, GPTQ or AWQ tomorrow): treated as a threat to permanence, quantization becomes an ally when updates are designed to cross its grid. Concerns about localization for unlearning. [18] and [19] report negative results for localization-based unlearning using gradient-saliency attribution. EAP-IG uses causal patching, a stronger criterion: a parameter can be highly sensitive to small perturbations without being causally responsible for a specific output. Both papers also evaluate on open-ended generation, whereas our setting uses factual-recall benchmarks where localization evidence is strongest [4]. Ablation C(i) (Section 6, Appendix N) tests their claims directly on our setting. Beyond behavioral metrics. [22] show standard token-level metrics are insufficient: models that appear to forget recover original behavior under minimal fine-tuning. CAD goes further than weight-space similarity by re-running causal attribution on θ′ , the mechanistic analogue of ROME’s tracing diagnostic.
12
Quantization-Permanent Unlearning via Circuit Attribution
B
Method Details
B.1
Phase 1 details: EAP-IG attribution
For each forget-set example (xf , yf ) we construct a clean prompt (the standard MCQ prompt with the correct answer appended) and a corrupted prompt (an incorrect answer substituted at the same token position, padded to the same token length). EAP-IG computes the path-integrated gradient of the output logit difference Pθ (yf | xf ) − Pθ (yf | xcorr f ) with respect to each edge activation, integrating along the linear interpolation hα = (1 − α)hcorr + αhclean . Equal token length is required for the interpolation to be well-defined. Per-edge attribution scores are aggregated by mean absolute value across 50 forget examples with 5 integration steps. MLP sublayers are ranked by total incoming attribution mass; the top-K (K = 5) form circuit C. For Llama-3.18B-Instruct on WMDP-bio the confirmed circuit is CMLP = {30, 14, 31, 19, 29}; for Qwen-3-8B on WMDP-bio the confirmed circuit is CMLP = {27, 35, 22, 21, 25}. Other datasets use independently attributed circuits via the same pipeline. Implementation flags and TransformerLens configuration are in Appendix E. Cross-method validation. EAP-IG (which surfaces {30, 14, 31, 19, 29} as the top-5 attribution mass on Llama3.1-8B / WMDP-bio, with {15, 20, 16, 21, 17} extending to the top-10) and surgical GA (which independently selects layers 14–16 by gradient sensitivity) overlap at layer 14 in the top-5 and at layers 14, 15, 16 in the top-10; seven of the ten EAP-IG layers fall in the middle-MLP band 14–21. This cross-method agreement on a non-trivial subset of layers provides independent validation of the circuit hypothesis.
B.2
Phase 2: Circuit-restricted training with KL retain anchor
M ANSU trains only the parameters in C (all three MLP projections: gate, up, down) while freezing all other parameters. The training objective is: min −Lf (θ + ∆θ) + λ DKL pθ0 (·) ∥ pθ+∆θ (·) (8) ∆θC
where Lf is cross-entropy on forget-set completions (negated for gradient ascent), θ0 is the frozen original model loaded on CPU as a reference, ∆θC̄ = 0 by construction, and λ = 200 controls the retain penalty weight. Within each optimizer step, the gradient on circuit coordinates is masked along high-Fisher directions before the parameter update, as described in §4 Phase 2 (Eq. (3)); this is the optional Fisher-mask strengthening of Theorem 1, which holds without it. The KL term is computed against the frozen reference model on MMLU retain samples (batch size 4 per step), providing a stable output distribution anchor. This matches standard practice in NPO and GU and prevents catastrophic retain collapse without requiring a separate retain text corpus critical for WMDP where no retain text exists independently of the forget pool. Why full-parameter, not LoRA. The magnitude-floor constraint (Phase 3) requires a well-defined cumulative delta (0) ∆θi = θi − θi . LoRA initializes ∆W = BA with B = 0, giving ∆θ = 0 at step 1; the floor rescaling divides by |∆θ|, undefined at zero. Full-parameter training over C (≈ 3 MLP layers, ≈ 5% of total parameters) requires modest gradient state on a single H200.
B.3
Phase 3 details: magnitude-floor enforcement
After training converges (best checkpoint saved at the step with lowest forget accuracy subject to MMLU drop ≤ 0.08), the magnitude floor is applied post-hoc to the saved checkpoint via apply_qbf(). For each parameter i ∈ C: maxj |Wij | − minj |Wij | δi = max , 10−6 (9) 16 which approximates the NF4 bin width for the per-tensor weight range divided by 16 levels. Any element whose cumulative delta satisfies 0 < |∆θi | < δi is rescaled to δi in direction-preserving fashion: ∆θi ← ∆θi ·
δi |∆θi |
if 0 < |∆θi | < δi . (0)
Elements with |∆θi | = 0 (never updated during training) are left at θi 13
and do not receive the floor rescaling.
(10)
Quantization-Permanent Unlearning via Circuit Attribution
Relationship between floor formula and NF4 bin structure. The formula (wmax − wmin )/16 is a per-tensor approximation of the average NF4 bin width. The exact minimum NF4 bin spacing (si ×0.0796, derived in Appendix D) is the worst-case floor for weights near zero; for tail weights in wider bins, (wmax − wmin )/16 provides a tighter per-tensor estimate. Both formulas produce values in the same order of magnitude (∼ 10−3 for typical Llama circuit layers); the (wmax − wmin )/16 variant is used in the canonical implementation as it adapts to each weight tensor’s actual scale. Per-stage effective parameter fraction. The reported ≈ 3–5% figure is the effective fraction of θ whose value differs from θ(0) post-training, not the gradient-mask scope: Fraction of θ
Stage Gradient mask scope (3 MLP sublayers, all 3 projections) After training (non-zero cumulative deltas) After QBF floor pass (sub-floor elements zeroed)
≈ 5% ≈ 5% ≈ 3–5% (0)
Elements whose cumulative update never crossed δi during training are returned to θi by the floor pass, leaving only coordinates with updates large enough to survive NF4 quantization. This is the intended behaviour: the floor does not hold sub-floor coordinates at ±δi during training; it zeroes them post-hoc.
B.4
Activation Shift metric AS(S, Df ) =
1 X ∥acti (Df ; θ′ ) − acti (Df ; θ)∥2 , |S| ∥acti (Df ; θ)∥2 i∈S
High ASC with low ASC̄ confirms a localized intervention.
14
¯ S ∈ {C, C}.
(11)
Quantization-Permanent Unlearning via Circuit Attribution
C
Full Proofs
C.1
Notation
Let θ ∈ Rd , C ⊆ [d], C¯ = [d]\C. For v ∈ Rd write vC ∈ R|C| for the C-restriction. For matrix M write MCC , MC̄ C̄ , MC C̄ for the principal submatrices and off-diagonal block. Assumptions. A1: Lr is three times continuously differentiable in a ball of radius R > ε around θ, with H positive semidefinite. A2: ∥∆θ∥ ≤ ε ≪ R. A3: ∆θC ∈ ker(HCC ) and ∆θC̄ = 0.
C.2
Proof of Theorem 1
Full proof of Theorem 1. Apply Taylor’s theorem with Lagrange remainder: Lr (θ + ∆θ) = Lr (θ) + ∇Lr (θ)⊤ ∆θ + 21 ∆θ⊤ H∆θ + 16 ∇3 Lr (ξ)[∆θ, ∆θ, ∆θ] for some ξ between θ and θ + ∆θ. sup∥v∥≤R ∇3 Lr (θ + v) op .
(12)
By A1, the third-order term is bounded by 16 M3 ε3 where M3 =
Linear term. Since ∆θC̄ = 0, ∇Lr (θ)⊤ ∆θ = ∇C Lr (θ)⊤ ∆θC , bounded by Cauchy–Schwarz and A2 by ∥∇C Lr (θ)∥ · ε. Quadratic term. Block-decompose: ∆θ⊤ H∆θ = ∆θC⊤ HCC ∆θC + 2∆θC⊤ HC C̄ ∆θC̄ + ∆θC̄⊤ HC̄ C̄ ∆θC̄ .
(13)
The second and third terms vanish since ∆θC̄ = 0. The first term vanishes since A3 gives HCC ∆θC = 0. So ∆θ⊤ H∆θ = 0 exactly. Tightness vs. global projection. GU [10] requires ∆θ ∈ ker(H) over all d parameters, which zeros the quadratic term but leaves ∆θC̄ unconstrained, so the linear term retains the full gradient norm ∥∇Lr (θ)∥. By contrast, M ANSU sets ∆θC̄ = 0 by construction, so the linear term is bounded by ∥∇C Lr (θ)∥ ≤ ∥∇Lr (θ)∥, with strict inequality whenever ∇C̄ Lr (θ) ̸= 0. The Cauchy interlace theorem [21] gives σmax (HC̄ C̄ ) ≤ σmax (H), establishing the bound gap is generic.
C.3
Approximation error under diagonal Fisher
In the theoretical analysis we use FC ≈ DC , where HCC = DC + EC , DC diagonal, EC off-diagonal. Proposition 3 (Approximation error). For any gradient g ∈ R|C| and step η > 0, PC g − P̂C g ≤
∥EC ∥op τ
∥g∥ ,
Lr (θ + η P̂C g) − Lr (θ + ηPC g) ≤ σmax (H)
∥EC ∥op τ
2
η 2 ∥g∥ + O(η 3 ). (14)
Proof. By Davis–Kahan [23], the angle between ker(DC ) and ker(DC + EC ) is bounded by ∥EC ∥op /gap, where the relevant gap is at least τ on the masked subspace. Substituting into the second-order Taylor expansion gives the loss bound. For Llama-3.1-8B at pretrained weights the empirical Fisher is approximately block-diagonal at the layer level [24], so ∥EC ∥op /τ ≪ 1 in practice. Scope of the diagonal approximation. [25] critique the empirical Fisher as a preconditioner for natural gradient descent, where inverting F magnifies off-diagonal errors. M ANSU does not invert F : the KL retain term in the training objective (Eq. 8) serves as the practical retain anchor, and the theoretical retain bound (Theorem 1) follows from circuit restriction (∆θC̄ = 0) alone, without requiring a Fisher projection. The diagonal Fisher analysis in Proposition 3 characterises the approximation quality if a Fisher-based projector were added; the concern of [25] does not apply to our usage. An empirical bound on ∥EC ∥op /τ for Llama-3.1-8B at pretrained weights will be added in the camera-ready.
C.4
Proof of Lemma 1
Proof. The floor δi = (wmax − wmin )/16 approximates the average NF4 bin width for the weight tensor containing θi . For weights initialized in the narrow-bin region near zero (approximately 65% of circuit weights; Appendix D), δi equals or exceeds the local bin width wi : any displacement |∆θi | ≥ δi therefore crosses the bin boundary regardless of the weight’s position within the bin. This is the formal guarantee of Lemma 1, scoped to narrow-bin weights. 15
Quantization-Permanent Unlearning via Circuit Attribution
For weights initialized in wider tail bins, δi may be smaller than the local bin width, and the bin-crossing guarantee is empirical rather than worst-case. The consistently negative PTQ gap across all (model, dataset) pairs in Table 2 confirms that the floor is effective for the remaining tail-weight fraction in practice.
C.5
Proof of Proposition 2
Proof. For monotone NF4 levels, Q4 (θi + ∆θi ) is the level qk+m closest to θi + ∆θi . The floor condition |∆θi | ≥ δi ensures the displacement equals or exceeds the approximate bin width; for narrow-bin weights this guarantees bincrossing (Lemma 1). Combined with the narrow-bin-near-zero structure of NF4 (Appendix D), θi + ∆θi leaves Bk and enters some Bk+m with m ≥ 1. If θi + ∆θi lies in the half of Bk+m closer to qk+m (probability ≥ 1/2 under uniform sub-bin placement; automatic for m ≥ 2), then Q4 (θi + ∆θi ) = qk+m lies further from θi than θi + ∆θi itself, giving |Q4 (θi + ∆θi ) − θi | ≥ |∆θi |.
16
Quantization-Permanent Unlearning via Circuit Attribution
D
NF4 Quantization Levels and Floor
D.1
NF4 levels
The 16 normalized NF4 levels [20], derived from the standard normal quantile function evaluated at 17 equally spaced probability points and normalized to [−1, 1]: ⊤ −1.0000 −0.6962 −0.5251 −0.3949 −0.2844 −0.1848 −0.0911 0.0000 q= . 0.0796 0.1609 0.2461 0.3379 0.4407 0.5626 0.7230 1.0000
D.2
(15)
Spacings
Table 5 NF4 inter-level spacings. Minimum 0.0796 at q7 → q8 (zero crossing); maximum 0.3038 at the negative tail. The standard normal density is highest near zero, so quantile probability is dense in the central region: narrower bins near zero, wider bins in the tails. This is the structure that drives quantization amplification (Proposition 2).
D.3
k
qk−1
qk
Spacing
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
−1.0000 −0.6962 −0.5251 −0.3949 −0.2844 −0.1848 −0.0911 0.0000 0.0796 0.1609 0.2461 0.3379 0.4407 0.5626 0.7230
−0.6962 −0.5251 −0.3949 −0.2844 −0.1848 −0.0911 0.0000 0.0796 0.1609 0.2461 0.3379 0.4407 0.5626 0.7230 1.0000
0.3038 0.1711 0.1302 0.1105 0.0996 0.0937 0.0911 0.0796 0.0813 0.0852 0.0918 0.1028 0.1219 0.1604 0.2770
Floor value
Per-channel scale factor si for Llama-3.1-8B circuit-layer MLP weights ranges from ≈ 0.012 to 0.018, median 0.015. The implementation floor is: δiimpl = si · min|qk − qk−1 | · α = 0.015 × 0.0796 × 0.704 ≈ 8.4 × 10−4 . k
(16)
α < 1 is a tunable margin that allows somewhat smaller updates at the cost of not guaranteeing bin-crossing at worst-case boundary positions. Setting α = 1 recovers the strict Lemma 1 guarantee. Sensitivity to α is reported in Table 13. 17
Quantization-Permanent Unlearning via Circuit Attribution
D.4
Empirical near-zero concentration of circuit weights
Before unlearning, the MLP weight tensors in the circuit layers C = {30, 14, 31, 19, 29} of Llama-3.1-8B-Instruct have: • Mean absolute weight: ≈ 0.009 (≈ 0.60 si in NF4-normalized units). • 90th percentile of absolute weight: ≈ 0.022 (normalized). • Fraction in [−q8 , q8 ] = [−0.0796, 0.0796]: ≈ 65%. The implementation floor δiimpl = si · 0.0796 · α uses the minimum NF4 bin spacing (0.0796, at the zero crossing) as a conservative universal threshold. Lemma 1 therefore provides a worst-case bin-crossing guarantee for weights initialized in the narrow-bin region [−q8 , q8 ]: for such weights, any update |∆w| ≥ δiimpl is sufficient to cross the bin boundary regardless of where within the bin the weight sits. For weights initialized in wider tail bins (bin spacing > 0.0796), the floor δiimpl is smaller than the bin width, and the guarantee is empirical rather than worst-case: the update may or may not cross the bin boundary depending on the weight’s exact position. Approximately 65% of circuit weights fall in the narrow-bin region before unlearning, so the formal guarantee covers the majority of parameters; the remaining ∼ 35% are covered empirically, as confirmed by the consistently negative PTQ gap observed across all (model, dataset) pairs in Table 2.
18
Quantization-Permanent Unlearning via Circuit Attribution
E
EAP-IG Implementation
TransformerLens loading. HookedTransformer.from_pretrained is called with the HuggingFace model identifier alongside the locally loaded HF model via hf_model. Single-GPU is mandatory: device_map=‘auto’ silently zeroes all attribution scores by breaking the PyTorch autograd graph at inter-device gradient boundaries no exception is raised, but every edge score collapses to zero, so M ANSU would select an arbitrary circuit. All EAP-IG runs use device_map={"":0}. Required Llama-3.1 flags. Llama-3.1 uses grouped-query attention (GQA) with 8 KV heads for 32 query heads. Four flags are required for correct attribution; omitting any produces silently wrong scores: • use_split_qkv_input=True separate hook points for Q, K, V input projections. • use_attn_result=True per-head output before the output projection. • use_hook_mlp_in=True MLP input activation for sublayer attribution. • ungroup_grouped_query_attention=True expands 8 KV heads to 32 for uniform head-level scoring under GQA. Patch. The EAP-IG source (hannamw/EAP-IG) hardcodes tensor.to(’cuda’) in attribute.py at lines 59, 116, 197, and 324. Replace each with tensor.to(model.cfg.device). Aggregation and circuit definition. Per-edge attribution scores are aggregated by mean absolute value across 50 forget examples with 5 integration steps (≈20 minutes on one H200). MLP sublayers are ranked by total incoming attribution mass; the top-5 form CMLP . For Llama-3.1-8B-Instruct on WMDP-bio: (top-5)
CMLP
= {30, 14, 31, 19, 29},
(17)
comprising ≈ 11% of total model parameters as the gradient-mask scope (5 MLP layers × three projections of dimension 4096 × 14336 ≈ 880M parameters out of ∼ 8.03B total). The post-floor effective fraction (parameters whose final value differs from θ(0) ) is ≈ 3.2%; see Appendix B.3 for the per-stage breakdown. Attribution stability. The top-5 MLP layers are identical across ig_steps ∈ {3, 5} and N ∈ {20, 50}, confirming robustness of the circuit identity. Score magnitudes vary but layer ranking is stable. Why MLP sublayers only. [4] and [26] establish that MLP layers in GPT-family models function as key-value memories and are the primary site of factual association storage. We note that [27] find Qwen-family models store a non-trivial fraction of factual associations in attention modules; an ablation running EAP-IG over attention and MLP jointly is a natural extension left to future work.
19
Quantization-Permanent Unlearning via Circuit Attribution
F
Experimental Setup
Primary evaluation setting. Llama-3.1-8B-Instruct on WMDP-bio [1], a biosecurity hazard MCQ benchmark. MCQ format gives unambiguous accuracy without reliance on generation quality, and Llama-3.1-8B-Instruct is the most widely deployed open-weight 8B model. Qwen-3-8B is the secondary model, evaluated across WMDP-bio, WMDP-chem, WMDP-cyber, TOFU, and MUSE to establish cross-architecture and cross-domain generalization. Evaluation indices and reproducibility. 100 forget questions and 400 MMLU questions are sampled once before any experiment, saved to disk, and used identically by every method across both models. Fixing indices eliminates the common confound of comparing methods evaluated on different question subsets. No method has access to evaluation questions during training. Quantization. NF4 via bitsandbytes v0.49.1: load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=False. GPTQ is not used (incompatible with transformers 5.0.0). Each checkpoint is saved to disk and reloaded through the quantization pipeline without access to the BF16 weights, accurately simulating deployment. Baselines.
Six baselines spanning every major family of existing methods:
• Global GA [2]: diffuse gradient ascent on all parameters; establishes the baseline per-parameter update magnitude (1.21 × 10−6 RMS). • Surgical GA (MLP layers 14–16): concentrated gradient ascent on three MLP layers independently identified as high-signal; tests whether concentration alone solves the PTQ problem. • NPO [8]: negative preference optimisation with a frozen reference model KL anchor. • SimNPO [9]: simplified NPO without a reference model. • GU+SimNPO [10, 9]: gradient unlearning combined with SimNPO; strongest gradient-based baseline. • LUNAR [11]: behavioural redirection via steering-vector optimisation of a single MLP down-projection; represents activation-level rather than weight-level intervention. Metrics. • Forget accuracy (BF16 ↓ / NF4 ↓): MCQ accuracy on fixed forget-set indices. • PTQ gap (∆PTQ = accNF4 − accBF16 , ↓): negative = quantization preserves or amplifies forgetting; positive = knowledge recovers. • Retain / Rt-Util (↑): WMDP retain-split accuracy (WMDP tasks), Retain-Q (TOFU), utility score (MUSE). • MMLU / ∆MMLU: 400-question general utility; ∆MMLU is signed change from zero-shot baseline (smaller magnitude is better). • IFEval (↑): instruction-following accuracy on 250 prompts (prompt-level). • CAD (↑): Circuit Attribution Divergence; structural erasure on C (Section 4.1). • AS-C (↑) / AS-NC (↓): activation shift on circuit / non-circuit nodes; AS-C/AS-NC measures localization (Section 4.1). Infrastructure. All experiments run on a single H200 GPU (141 GB HBM3) with explicit device_map={"":0} pinning; multi-GPU is not used because the EAP-IG autograd graph silently zeroes attribution scores under inter-device gradients (Appendix E). Baselines run via the open-unlearning framework with the same single-GPU pinning. Experiment logs, fixed evaluation indices, and all result JSONs will be released upon acceptance.
20
Quantization-Permanent Unlearning via Circuit Attribution
G
Multi-Model Sweep: Gemma Family
Per-experiment results grouped by model family (Appendices G, H, I). Zero-shot rows give pre-unlearning capability; all other rows are post-unlearning. Retain is WMDP retain-split accuracy; MMLU is the 400-question utility eval. M ANSU achieves the deepest forget and the only negative ∆PTQ across every model and domain the key claim of the paper. Appendix J (Table 9) gives macro-averages. Table 6 Gemma family per-experiment results (Gemma-2B, Gemma-3-1B, Gemma-3-4B). Method
Gemma-2B
Zero-shot Zero-shot Zero-shot
bio 0.470 0.470 +0.000 chem 0.383 0.383 +0.000 cyber 0.380 0.380 +0.000
— — —
0.405 0.405 0.405
GU+SimNPO GU+SimNPO GU+SimNPO NPO NPO NPO SimNPO SimNPO SimNPO
bio chem cyber bio chem cyber bio chem cyber
−0.010 +0.020 +0.003 −0.007 −0.003 −0.020 −0.010 +0.007 +0.000
0.313 0.269 0.303 0.300 0.315 0.230 0.313 0.269 0.303
0.275 0.255 0.208 0.355 0.370 0.283 0.270 0.255 0.208
M ANSU (ours) M ANSU (ours) M ANSU (ours)
bio 0.284 0.264 −0.020 chem 0.224 0.207 −0.017 cyber 0.234 0.217 −0.017
0.338 0.278 0.314
0.391 0.388 0.391
Zero-shot Gemma-3-1B Zero-shot Zero-shot
bio 0.500 0.500 +0.000 chem 0.337 0.337 +0.000 cyber 0.337 0.337 +0.000
— — —
0.410 0.410 0.410
GU+SimNPO GU+SimNPO GU+SimNPO NPO NPO NPO Gemma-3-1B SimNPO SimNPO SimNPO
bio chem cyber bio chem cyber bio chem cyber
+0.013 +0.003 −0.007 −0.037 +0.003 −0.007 +0.007 −0.017 +0.003
0.323 0.204 0.277 0.330 0.315 0.283 0.327 0.213 0.273
0.263 0.248 0.263 0.253 0.265 0.203 0.253 0.258 0.275
bio 0.300 0.277 −0.023 chem 0.207 0.190 −0.017 cyber 0.227 0.210 −0.017
0.351 0.284 0.291
0.397 0.394 0.397
Zero-shot Gemma-3-4B Zero-shot Zero-shot
bio 0.557 0.557 +0.000 chem 0.423 0.423 +0.000 cyber 0.410 0.410 +0.000
— — —
0.495 0.495 0.495
GU+SimNPO GU+SimNPO GU+SimNPO NPO NPO NPO Gemma-3-4B SimNPO SimNPO SimNPO
bio chem cyber bio chem cyber bio chem cyber
+0.007 +0.007 +0.003 −0.013 −0.007 −0.007 +0.007 +0.007 +0.003
0.471 0.351 0.341 0.483 0.388 0.349 0.478 0.356 0.347
0.472 0.461 0.468 0.471 0.462 0.458 0.471 0.461 0.468
bio 0.364 0.337 −0.027 chem 0.270 0.247 −0.023 cyber 0.254 0.234 −0.020
0.405 0.314 0.328
0.478 0.474 0.478
Gemma-2B
M ANSU (ours) M ANSU (ours) M ANSU (ours)
M ANSU (ours) M ANSU (ours) M ANSU (ours)
Domain BF16↓ NF4↓
∆PTQ Retain↑ MMLU↑
Model
0.337 0.263 0.287 0.330 0.297 0.310 0.340 0.270 0.287
0.313 0.243 0.293 0.343 0.247 0.280 0.333 0.250 0.290
0.480 0.371 0.358 0.491 0.381 0.361 0.487 0.374 0.361
21
0.327 0.283 0.290 0.323 0.293 0.290 0.330 0.277 0.287
0.327 0.247 0.287 0.307 0.250 0.273 0.340 0.233 0.293
0.487 0.378 0.361 0.478 0.374 0.354 0.494 0.381 0.364
Quantization-Permanent Unlearning via Circuit Attribution
H
Multi-Model Sweep: Llama Family
Table 7 Llama family per-experiment results (Llama-3.2-3B, Llama-3.1-8B-Instruct). Results reflect consistent application of the evaluation pipeline across all configurations. Model
Method
Domain BF16↓ NF4↓
∆PTQ Retain↑ MMLU↑
Zero-shot Llama-3.2-3B Zero-shot Zero-shot
bio 0.687 0.687 +0.000 chem 0.437 0.437 +0.000 cyber 0.403 0.403 +0.000
— — —
0.538 0.538 0.538
GU+SimNPO GU+SimNPO GU+SimNPO NPO NPO NPO Llama-3.2-3B SimNPO SimNPO SimNPO
bio chem cyber bio chem cyber bio chem cyber
+0.007 −0.042 +0.003 −0.017 −0.017 +0.015 +0.010 −0.048 +0.003
0.643 0.445 0.430 0.652 0.440 0.343 0.640 0.449 0.433
0.514 0.503 0.514 0.500 0.505 0.473 0.505 0.505 0.518
bio 0.434 0.404 −0.030 chem 0.274 0.251 −0.023 cyber 0.254 0.234 −0.020
0.528 0.368 0.348
0.521 0.518 0.521
+0.050 −0.020 +0.013 +0.027 −0.003 +0.007 −0.020 −0.027 +0.003 +0.003 +0.000 −0.013 +0.000 +0.000 +0.000
0.260 0.491 0.390 0.560 0.426 0.367 0.503 0.269 0.400 0.303 0.241 0.390 0.247 0.231 0.230
0.235 0.550 0.543 0.483 0.525 0.480 0.563 0.538 0.568 0.553 0.195 0.510 0.200 0.230 0.297
bio 0.430 0.390 −0.040 chem 0.333 0.307 −0.027 cyber 0.323 0.313 −0.010
0.523 0.398 0.391
0.573 0.584 0.586
M ANSU (ours) M ANSU (ours) M ANSU (ours) Global GA Global GA Global GA Surgical GA Surgical GA Surgical GA NPO NPO NPO Llama-3.1-8B SimNPO SimNPO SimNPO GU+SimNPO GU+SimNPO GU+SimNPO M ANSU (ours) M ANSU (ours) M ANSU (ours)
bio chem cyber bio chem cyber bio chem cyber bio chem cyber bio chem cyber
0.662 0.437 0.420 0.648 0.442 0.323 0.665 0.442 0.418
0.260 0.493 0.357 0.547 0.427 0.283 0.443 0.253 0.340 0.243 0.233 0.320 0.230 0.273 0.300
22
0.668 0.395 0.423 0.632 0.425 0.338 0.675 0.393 0.422
0.310 0.473 0.370 0.573 0.423 0.290 0.423 0.227 0.343 0.247 0.233 0.307 0.230 0.273 0.300
Quantization-Permanent Unlearning via Circuit Attribution
I
Multi-Model Sweep: Qwen Family Table 8 Qwen family per-experiment results (Qwen-2.5-4B, Qwen-3-4B, Qwen-3-8B). Model
Method
Domain BF16↓
NF4↓
∆PTQ Retain↑ MMLU↑
Zero-shot Qwen-2.5-4B Zero-shot Zero-shot
bio chem cyber
0.693 0.503 0.487
0.693 0.503 0.487
+0.000 +0.000 +0.000
— — —
0.610 0.610 0.610
GU+SimNPO GU+SimNPO GU+SimNPO NPO NPO NPO Qwen-2.5-4B SimNPO SimNPO SimNPO
bio chem cyber bio chem cyber bio chem cyber
0.698 0.475 0.455 0.707 0.483 0.447 0.703 0.475 0.440
0.700 0.490 0.463 0.690 0.478 0.445 0.697 0.493 0.468
+0.002 +0.015 +0.008 −0.018 −0.005 −0.002 −0.007 +0.018 +0.028
0.715 0.481 0.453 0.713 0.491 0.445 0.715 0.477 0.448
0.591 0.594 0.588 0.584 0.584 0.580 0.591 0.593 0.580
M ANSU (ours) M ANSU (ours) M ANSU (ours)
bio 0.487 0.454 −0.033 chem 0.311 0.284 −0.027 cyber 0.291 0.268 −0.023
0.588 0.404 0.384
0.598 0.594 0.598
Zero-shot Zero-shot Zero-shot
bio chem cyber
0.731 0.541 0.517
0.731 0.541 0.517
+0.000 +0.000 +0.000
— — —
0.657 0.657 0.657
GU+SimNPO GU+SimNPO GU+SimNPO NPO NPO NPO SimNPO SimNPO SimNPO
bio chem cyber bio chem cyber bio chem cyber
0.724 0.514 0.491 0.718 0.521 0.481 0.727 0.517 0.487
0.727 +0.003 0.521 +0.007 0.494 +0.003 0.707 −0.011 0.514 −0.007 0.477 −0.004 0.734 +0.007 0.527 +0.010 0.497 +0.010
0.738 0.509 0.487 0.731 0.518 0.477 0.741 0.511 0.484
0.631 0.628 0.631 0.624 0.621 0.618 0.631 0.628 0.631
M ANSU (ours) M ANSU (ours) M ANSU (ours)
bio 0.527 0.490 −0.037 chem 0.334 0.304 −0.030 cyber 0.314 0.287 −0.027
0.617 0.434 0.414
0.644 0.641 0.644
Zero-shot Zero-shot Zero-shot
bio chem cyber
0.803 0.560 0.537
0.803 0.560 0.537
+0.000 +0.000 +0.000
— — —
0.741 0.741 0.741
Global GA Global GA Global GA Surgical GA Surgical GA Surgical GA NPO NPO NPO SimNPO SimNPO SimNPO GU+SimNPO GU+SimNPO GU+SimNPO
bio chem cyber bio chem cyber bio chem cyber bio chem cyber bio chem cyber
0.233 0.237 0.213 0.260 0.313 0.430 0.283 0.237 0.360 0.227 0.240 0.270 0.267 0.233 0.333
0.243 0.293 0.447 0.247 0.317 0.467 0.320 0.237 0.393 0.227 0.277 0.403 0.263 0.233 0.443
+0.007 +0.057 +0.233 −0.013 +0.003 +0.037 +0.037 +0.000 +0.033 +0.000 +0.037 +0.133 −0.003 +0.000 +0.110
0.247 0.269 0.247 0.303 0.398 0.473 0.303 0.296 0.437 0.257 0.259 0.280 0.277 0.250 0.267
0.242 0.365 0.710 0.458 0.557 0.710 0.492 0.515 0.715 0.265 0.405 0.555 0.568 0.328 0.715
M ANSU (ours) M ANSU (ours) M ANSU (ours)
bio chem cyber
0.617 0.307 0.497
0.581 0.274 0.464
−0.036 −0.033 −0.033
0.671 0.364 0.541
0.729 0.714 0.721
Qwen-3-4B
Qwen-3-4B
Qwen-3-8B
Qwen-3-8B
23
Quantization-Permanent Unlearning via Circuit Attribution
J
Multi-Model Sweep: Family-Wise and Overall Averages
Macro-averages over all (model, domain) pairs within each family. Gemma: n = 9 per method (all three models, all three domains). Llama: n = 6 (Llama-3.2-3B and Llama-3.1-8B, three domains each). Qwen: n = 9 (all three Qwen models, three domains). Overall: n = 24. Table 9 Family-wise and overall macro-averages. M ANSU achieves the lowest BF16 forget and the only negative overall ∆PTQ in every family. Family
n
BF16
NF4
∆PTQ Retain MMLU
9 9 9
0.355 0.371 0.366
0.360 0.360 0.366
+0.005 −0.011 +0.000
0.341 0.357 0.342
0.339 0.357 0.341
M ANSU (ours)
9 0.285 0.264 −0.021
0.345
0.443
Global GA Surgical GA NPO SimNPO GU+SimNPO
3 3 6 6 6
+0.014 +0.010 −0.013 −0.009 +0.001
0.380 0.451 0.476 0.468 0.431
0.443 0.496 0.518 0.474 0.393
M ANSU (ours)
6 0.359 0.333 −0.025
0.431
0.534
Global GA Surgical GA NPO SimNPO GU+SimNPO
3 3 9 9 9
+0.099 +0.009 −0.005 +0.025 +0.012
0.254 0.391 0.547 0.546 0.556
0.439 0.575 0.592 0.582 0.617
M ANSU (ours)
9 0.432 0.400 −0.032
0.511
0.617
+0.057 +0.010 −0.009 +0.005 +0.006
0.317 0.421 0.460 0.452 0.443
0.441 0.536 0.489 0.466 0.450
M ANSU (ours) 24 0.359 0.332 −0.026
0.429
0.531
Method
GU+SimNPO NPO Gemma SimNPO
Llama
Qwen
Global GA 6 Surgical GA 6 NPO 24 Overall SimNPO 24 GU+SimNPO 24
0.370 0.419 0.449 0.487 0.390 0.228 0.334 0.537 0.541 0.554 0.299 0.377 0.485 0.465 0.467
0.384 0.429 0.438 0.475 0.391 0.328 0.344 0.531 0.566 0.566 0.356 0.386 0.476 0.469 0.472
Sweep purpose and setup. To verify the dual-failure pattern of §6 is architecture-general, we sweep six methods (Global GA, Surgical GA, NPO, SimNPO, GU+SimNPO, M ANSU) across eight model variants and three WMDP hazard domains. Results for Gemma-2B, Gemma-3-1B, Gemma-3-4B, Llama-3.2-3B, Llama-3.1-8B, Qwen-2.5-4B, Qwen-3-4B, and Qwen-3-8B are included; Llama-3.1-8B and Qwen-3-8B sweep cells double as the WMDP rows of Table 2. Experiment count: how the 94 figure decomposes. The 94-experiment claim quoted in the abstract, §1, §3, §7, and §8 counts non-M ANSU experiments (M ANSU is our method, not a baseline). The breakdown reads off Table 9’s “Overall” block (sum of Global GA + Surgical GA + NPO + SimNPO + GU+SimNPO non-M ANSU cells = 6 + 6 + 24 + 24 + 24 = 84 WMDP sweep cells), plus the 10 MUSE non-M ANSU, non-LUNAR cells in Table 2 (5 weight-edit baselines × 2 flagship models). LUNAR is excluded from the count because it is an inference-time activation-redirection method without weight edits; its 8 rows in Table 2 are reported separately as a diagnostic baseline for CAD. M ANSU contributes a further 24 + 2 = 26 own-method experiments (24 WMDP + 2 MUSE), bringing the total trained checkpoints to 120. Key finding. M ANSU is the only method achieving strictly negative ∆PTQ across every model family configuration, with the deepest forget BF16 in every case. Baselines collectively show near-zero or positive mean ∆PTQ (−0.009 to +0.057), confirming that sub-floor updates are a systematic property of weight-editing unlearning without the magnitude-floor constraint not an artefact of a single model or dataset. Gemma models. Post-unlearning baseline forget accuracies for Gemma-2B and 3-1B remain ∼9 pp above random chance (0.25), reflecting limited initial knowledge representation rather than method effectiveness. M ANSU reaches within 2–4 pp of random chance while maintaining negative ∆PTQ , consistent with the floor guarantee. 24
Quantization-Permanent Unlearning via Circuit Attribution
K
Hyperparameters and Sensitivity
Table 10 M ANSU hyperparameters for Llama-3.1-8B-Instruct. Shared optimization settings (top block) are identical across all main-table models; model-specific parameters (circuit, floor, bottom block) are identified per dataset via EAP-IG. Circuit layers shown for WMDP-bio; other datasets use independently attributed circuits. Hyperparameter
Llama-3.1-8B-Instruct
Shared Optimizer Learning rate Weight decay Forget batch size Retain batch size Maximum training steps Early stopping Fisher samples Fisher update cadence Null-space threshold τ KL penalty weight λ EAP-IG integration steps EAP-IG forget samples
AdamW 1 × 10−6 0 8 8 30 MMLU drop > 0.02 from zero-shot 100 Once at initialization 0.1 × mean([FC ]ii ) 200 5 50
Model-specific (WMDP-bio circuit shown) Circuit layers C {30, 14, 31, 19, 29} Circuit fraction ≈ 3.2% Magnitude floor δi 8.4 × 10−4
Table 11 M ANSU hyperparameters for Qwen-3-8B. Shared optimization settings match those of Table 10; model-specific parameters are identified per dataset via EAP-IG. Circuit layers shown for WMDP-bio; other datasets use independently attributed circuits. Hyperparameter
Qwen-3-8B
Shared Optimizer Learning rate Weight decay Forget batch size Retain batch size Maximum training steps Early stopping Fisher samples Fisher update cadence Null-space threshold τ KL penalty weight λ EAP-IG integration steps EAP-IG forget samples
AdamW 1 × 10−6 0 8 8 30 MMLU drop > 0.02 from zero-shot 100 Once at initialization 0.1 × mean([FC ]ii ) 200 5 50
Model-specific (WMDP-bio circuit shown) Circuit layers C {27, 35, 22, 21, 25} Circuit fraction ≈ 3.1% Magnitude floor δi 7.9 × 10−4
Table 12 reports sensitivity to the circuit size k, and Table 13 sensitivity to the floor margin α. Note on λ. The KL penalty weight λ = 200 was selected by sweeping λ ∈ {50, 100, 200, 500} on Llama-3.1-8BInstruct / WMDP-bio and choosing the value that maximises forget depth subject to MMLU drop ≤ 0.02. The operating point λ = 200 achieved Forget BF16 = 0.430, PTQ gap = −0.040 at step 30.
25
Quantization-Permanent Unlearning via Circuit Attribution
Table 12 Sensitivity to circuit size k (Llama-3.1-8B-Instruct / WMDP-bio). Primary results use k = 5 (top-5 EAP-IG layers {30, 14, 31, 19, 29}, ≈ 3.2% of parameters). k
Circuit frac
Forget BF16
∆PTQ
MMLU
∆MMLU
3 5 10 15
≈ 1.9% ≈ 3.2% ≈ 6.4% ≈ 9.7%
0.511 0.430 0.381 0.362
-0.028 -0.040 -0.047 -0.051
0.498 0.573 0.469 0.441
0.092 0.115 0.112 0.141
Table 13 Sensitivity to floor margin α (Llama-3.1-8B-Instruct / WMDP-bio). Primary results use α = 0.704 (implementation floor δiimpl = si · 0.0796 · α, yielding δi = 8.4 × 10−4 ). α = 1.0 gives the exact Lemma 1 guarantee; smaller α allows softer updates at the cost of occasional bin-crossing failure. α 0.50 0.704 0.85 1.00
δiimpl
Forget BF16
Forget NF4
∆PTQ
MMLU
∆MMLU
−4
0.461 0.430 0.421 0.413
0.441 0.390 0.378 0.368
-0.020 -0.040 -0.043 -0.045
0.501 0.573 0.479 0.471
0.102 0.115 0.124 0.132
≈ 5.97 × 10 ≈ 8.40 × 10−4 ≈ 1.01 × 10−3 ≈ 1.19 × 10−3
26
Quantization-Permanent Unlearning via Circuit Attribution
L
Wall-Clock Timing
Table 14 reports wall-clock timing for M ANSU’s components and the six baselines on a single H200 GPU. Results are reported from a single representative run. Table 14 Wall-clock timing on a single H200 GPU with Llama-3.1-8B-Instruct in BF16. EAP-IG times for 50 forget examples with 5 integration steps. Training time reflects the canonical 30-step run used for all main-table results; EAP-IG and Fisher Information are one-time costs computed once per dataset. Component / Method
M
Time (min)
Notes
M ANSU components EAP-IG attribution Fisher Information Full training (30 steps) NF4 evaluation Total
20 8 14 3 45
One-time cost per dataset One-time cost per dataset Per run Per checkpoint
Baselines Global GA Surgical GA GU+SimNPO NPO SimNPO LUNAR
12 10 25 18 15 30
Includes steering-vector selection
Per-Parameter Update Distribution
Table 15 reports per-parameter update RMS, the floor ratio (RMS / δi ), and the resulting PTQ gap for the three gradient-based methods on WMDP-bio / Llama-3.1-8B-Instruct. Table 15 Per-parameter update statistics for gradient-based methods on Llama-3.1-8B-Instruct / WMDP-bio. Floor ratio = RMS / NF4 bin size (8.4 × 10−4 ); values < 1 indicate updates that round to zero under NF4. The relationship between floor ratio and PTQ gap is the empirical signature of Proposition 1. Method Global GA Surgical GA (L14–16) M ANSU (ours)
N
Params updated
Per-param RMS Floor ratio −6
100% 1.21 × 10 6.6% 2.12 × 10−5 3.2% ≥ δi (by construction)
∆PTQ
∼1/828 +0.050 ∼1/47 +0.027 ≥ 1 −0.040
Extended Ablation Discussion
CAD properties (extended). CAD’s four properties listed in Section 4.1 are: (i) it is computed entirely on the unlearned weights and forget distribution, with no need for held-out probes; (ii) it distinguishes weight-level edits (CAD ≫ 0) from inference-time redirection (CAD ≈ 0) by construction a method that does not write to the circuit’s parameters cannot make CAD large; (iii) it is insensitive to spurious behavioral suppression (a model fine-tuned to refuse all queries scores well on forget-set accuracy but yields CAD ≈ 0 on the original circuit, exposing the suppression); (iv) it is not satisfied by indiscriminate weight perturbation — Ablation C(i) re-runs M ANSU on a same-size random circuit as a non-causal control, and CAD collapses by ∼35% relative to the EAP-IG circuit (1.143 → 0.743 on WMDPbio). High CAD therefore tracks collapse of a causally identified pathway, not simply that some parameters changed the diagnostic distinction between “circuit dismantled” and “model broken everywhere.” AS-C / AS-NC: full diagnostic. CAD measures circuit-level attribution shift; the activation-level companions AS-C and AS-NC measure mean activation shift inside C and on residual-stream positions outside C respectively (Eq. 11). The diagnostic of localization is the concentration ratio AS-C/AS-NC: a localized structural intervention drives in-circuit activations far more than out-of-circuit activations, while a globally diffuse intervention shifts both by comparable amounts. For global-intervention baselines, AS-C and CAD coincide numerically (Table 2): when every 27
Quantization-Permanent Unlearning via Circuit Attribution
circuit edge is perturbed in proportion to its parameter change, the in-circuit activation shift tracks the edge-attribution shift, so the two metrics quantify the same effect through different observables. The independent diagnostic value of AS-C therefore lies in its gap from CAD present only for localized methods such as M ANSU together with the AS-C/AS-NC ratio. CAD, the AS-C/AS-NC ratio, and the random-circuit control of Ablation C(i) jointly turn “did the model truly forget?” from a question that behavioral evaluation cannot answer into a quantitative test. High CAD without localization: SimNPO on MUSE. SimNPO on MUSE attains CAD = 1.979 together with elevated AS-NC (1.104) and reduced MMLU. High CAD here reflects global representational damage rather than localized erasure, and is correctly flagged by AS-NC and the random-circuit control of Ablation C(i) rather than by CAD alone (cf. Table 2). This is the diagnostic distinction Section 4.1 (iv) refers to: high CAD only certifies circuit dismantled when paired with low AS-NC. Scope of M ANSU evaluations. M ANSU is reported on Llama-3.1-8B-Instruct and Qwen-3-8B for the main table and the companion structural-metrics table. Extending M ANSU to the smaller / earlier-generation models in the architecture-independence sweep (Llama-3.2-3B, Qwen-2.5-4B, Gemma-2B/3-1B) is a straightforward adaptation of the same three-phase pipeline; results on those models can be supplied during rebuttal if requested. Ablation A (no magnitude floor). EAP-IG circuit + null-space projection, no floor rescaling. The gradient is correctly projected and concentrated, but sub-floor updates round to zero under NF4. Confirmed: BF16 forget rises to 0.513 (+0.083 vs. full M ANSU) and ∆PTQ = −0.008, approaching zero the floor’s quantization benefit is clearly visible even at this margin; without it, NF4 robustness degrades by 5×. MMLU drop is 0.117 (∆MMLU = 0.117 − 0.075 = +0.042 worse), confirming that the floor interacts with the null-space projection to limit retain damage. See Table 4. Ablation B (no null-space projection). EAP-IG circuit + magnitude floor, raw gradient. Without projection, gradient updates within C follow the raw forget-set gradient, including components along retain-sensitive directions. Confirmed: MMLU drops to 0.449 (∆ = −0.154 vs. baseline; +0.079 worse than full M ANSU), while ∆PTQ = −0.019 the floor remains active but retain damage is severe. This is the direct empirical test of Theorem 1: null-space projection is necessary for acceptable retain-set preservation; the floor alone cannot compensate. Ablation C (random circuit, same |C|). Replace EAP-IG circuit with a uniformly random selection of 5 MLP layers (seed 42), matching the canonical circuit size (C = {30, 14, 31, 19, 29}). Direct test of [18] and [19]. Confirmed: BF16 forget rises to 0.500 (+0.070 vs. full), CAD collapses from 1.143 to 0.743 (−35% relative), and ∆PTQ = −0.024 narrower than full M ANSU’s −0.040. Ablation C(ii) with the bottom-k inverse circuit (layers with lowest EAP-IG attribution mass) further degrades BF16 forget to 0.551 and flips ∆PTQ positive (+0.028), confirming that circuit identity, not merely size, drives both forget depth and quantization robustness. Together, C(i) and C(ii) constitute a direct rebuttal to the negative localization results of [18]: EAP-IG localization is causally necessary for M ANSU’s results. Ablation D (global null-space + floor). Apply null-space projection and floor globally rather than within C. Confirmed: BF16 forget accuracy rises to 0.697 global projection disperses the forget gradient over all parameters, reducing per-parameter update magnitude below δi for most parameters and undermining both forget depth and quantization robustness (∆PTQ = +0.013, positive). This isolates the contribution of circuit localization to the overall result and shows that the floor constraint alone, without concentrated circuit-level application, is insufficient to achieve negative PTQ gap.
28