From Anchors to Supervision: Memory-Graph Guided Corpus-Free Unlearning for Large Language Models Wenxuan Li1 , Zhenfei Zhang1 , Mi Zhang1 , Geng Hong1 Mi Wen2 , Xiaoyu You3 , Min Yang1 1
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China 2
3
Shanghai University of Electric Power, Shanghai, China
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China {wxli24, zhangzf24}@m.fudan.edu.cn, {mi_zhang, ghong, m_yang}@fudan.edu.cn, [email protected], [email protected]
arXiv:2604.13777v1 [cs.CL] 15 Apr 2026
Abstract Large language models (LLMs) may memorize sensitive or copyrighted content, raising significant privacy and legal concerns. While machine unlearning has emerged as a potential remedy, prevailing paradigms rely on user-provided forget sets, making unlearning requests difficult to audit and exposing systems to secondary leakage and malicious abuse. We propose MAGE, a Memory-grAph Guided Erasure framework for user-minimized, corpus-free unlearning. Given only a lightweight user anchor that identifies a target entity, MAGE probes the target LLM to recover target-related memorization, organizes it into a weighted local memory graph, and synthesizes scoped supervision for unlearning. MAGE is model-agnostic and can be plugged into standard unlearning methods, and requires no access to the original training corpus. Experiments on two benchmarks TOFU and RWKU demonstrate that MAGE ’s self-generated supervision achieves effective unlearning performance comparable to supervision generated with external reference, while preserving overall utility. These results support a practical and auditable unlearning workflow driven by minimal anchors rather than user-supplied forget corpora.
1
Introduction
Large language models (LLMs) have been widely deployed, yet their reliance on web-scale training data can lead to memorization of sensitive or copyrighted content (Carlini et al., 2021; Lucchi, 2024). A recent lawsuit by The New York Times against OpenAI over unauthorized data use (NPR, 2025) further underscores these legal and ethical risks. Under regulations such as the General Data Protection Regulation (GDPR), individuals can invoke the “right to be forgotten” (RTBF) (Zhang et al., 2024a) to request removal of specific data from deployed models. To meet such requests without retraining from scratch, machine unlearning has emerged as
Forget everything about me !!!
Unlearning Request • Target: Alice • Forget Set: • Requester Identity
Alice
Leakage Risk • Secondary Info leakage • Intercepted / Mishandled
LLM Service-Provider
Hard-to-Audit Requests
Authorization? Properly scoped? Safety?
• Unauthorized Scope
Audit Request
(Other’s data)
• Poisoned Forget Sets (Backdoor)
Integrity Risk Apply Forget Set Update Model
Forget everything about me !!! Target: Alice
Alice
• Utility degradation • Reversible unlearning
Let me handle it !
User-minimized No External Forget Sets Scoped, auditable
Figure 1: Risk surface of externally supplied forget sets in existing LLM unlearning paradigms. In contrast, a user-minimized paradigm mitigates these risks by improving information protection and auditability.
a practical approach to selectively erase targeted knowledge while preserving overall capabilities. Most LLM unlearning pipelines fine-tune on a forget set that specifies the knowledge to erase, using objectives such as reverse optimization (e.g., GA (Jang et al., 2022), NPO (Zhang et al., 2024b)) or alternative-response training (e.g., WHP (Eldan and Russinovich, 2023), IDK (Ren et al., 2025a)). To preserve utility, a retain set of general-purpose data (e.g., Wikipedia) is typically included, and some settings further add a neighbor set consisting of distributionally similar but unlearning-irrelevant data to reduce collateral changes. Despite recent progress, most existing approaches rely on a strong and often implicit assumption: the user requesting unlearning must provide a forget set that explicitly contains the information to be removed, which often contains sensitive content. While this assumption underpins most existing unlearning pipelines, relying on an externally sup-
plied forget set introduces practical risks that are often overlooked (Figure 1). From the user’s perspective, uploading sensitive content for unlearning can cause secondary information leakage (Jin et al., 2024), especially if the forget corpus is intercepted or mishandled. From the LLM serviceprovider’s perspective, externally submitted forget sets are inherently hard to audit, making unlearning requests vulnerable to abuse. Providers must verify authorization and scope, yet adversaries may submit wrongly scoped forget sets (e.g., containing other users’ data), leading to unauthorized forgetting. Moreover, seemingly benign forget sets can embed backdoors, which may stealthily degrade utility or make unlearning reversible (Ren et al., 2025b; Thaker et al., 2025; Shang et al., 2025). Collectively, these leakage, auditability, and integrity risks cast doubt on the reliability of current externally provided forget set paradigms . To mitigate these risks, we advocate a more realistic corpus-free unlearning paradigm that minimizes user involvement to better match real-world deployment needs. Instead of submitting a full forget set, the user provides minimal anchor information to specify the target, making requests easier to audit and reducing privacy risks. Despite these benefits, this paradigm introduces new challenges: (1) without an external forget set and with no access to the original training corpus during deployment, the LLM must reliably recover targetrelated memorization from its parameters to ensure effective unlearning; (2) the recovered memory must be translated into scoped supervision that separates what to forget from what to preserve. We therefore take a step further and propose MAGE, a memory-graph guided erasure framework tailored to this user-minimized, corpus-free paradigm: it probes and reconstructs the LLM’s target-related memorization, abstracts it into a structured local memory graph, and uses the graph to self-generate scoped unlearning supervision. For the first challenge, MAGE iteratively expands from minimal anchors to build a weighted local memory graph, and estimates memorization strength from the LLM’s outputs to filter noise and retain strongly memorized content. For the second challenge, MAGE samples informative memory paths and converts them into a scoped forget set with a companion neighbor set, enabling focused unlearning while preserving model utility. We validate our framework on two widely used entity unlearning benchmarks (TOFU (Maini et al.,
2024) and RWKU (Jin et al., 2024)) across various unlearning strategies. Our contributions are summarized as follows: • We identify key security risks in prevailing unlearning pipelines and introduce a userminimized, corpus-free unlearning paradigm driven by minimal, auditable anchors. • We develop MAGE, a memory-graph guided erasure framework that recovers target-related memorization and self-generates scoped unlearning supervision. • Experiments on TOFU and RWKU show that MAGE achieves supervised-level unlearning performance using only self-generated data, supporting the feasibility of corpus-free unlearning.
2
Related Work
2.1
LLM Unlearning
LLM unlearning aims to remove specific knowledge from pre-trained LLMs, addressing privacy, security, and copyright concerns without full retraining (Liu et al., 2025). Most methods rely on parameter optimization to make the model behave as if it had never seen the forget set (Doshi and Stickland, 2024; Lynch et al., 2024). Some use gradientascent style updates to negate learned knowledge (Jang et al., 2022; Yao et al., 2024), while others adopt preference-based objectives (e.g., DPO (Rafailov et al., 2023), NPO (Zhang et al., 2024b), IDK (Ren et al., 2025a), WHP (Eldan and Russinovich, 2023)) to guide responses. To preserve utility, they typically incorporate a retain set or neighbor set via forward optimization, such as gradient descent (Liu et al., 2022), KL regularization (Yuan et al., 2024), or knowledge replacement (Xu et al., 2025). Recent work also improves unlearning performance by enhancing the quality of forget sets (Kuo et al., 2025; Wang et al., 2025b,a). Beyond fine-tuning, researchers have explored other forgetting mechanisms, such as in-context learning (Pawelczyk et al., 2023), task vectors (Liu et al., 2024a), and data sanitization (Bhaila et al., 2024). However, these indirect methods may still leave residual information. Knowledge editing enables fine-grained updates, but is less suitable for largescale unlearning (Tian et al., 2024). 2.2
Entity-level unlearning
Entity-level unlearning aims to remove a model’s knowledge about an entity, beyond the specific
instances listed in a forget set. Since the target knowledge is not explicitly defined, it requires constructing representative forget data and evaluating broader effects on related knowledge. TOFU (Maini et al., 2024) introduced this setting with fictional entities, and later work (Ma et al., 2025) showed that many methods still behave like instance-level unlearning and that performance is highly sensitive to forget-set quality. Recent studies improve entity-level unlearning through concept/parameter interventions (Wang et al., 2025a; Choi et al., 2025) or by augmenting forget data (Xu et al., 2025). However, most approaches assume predefined forget sets and largely overlook paradigm-level risks, whereas RWKU (Jin et al., 2024) begins to highlight privacy concerns. We focus on entity-level unlearning because it aligns with real-world requests and challenges methods to remove entangled knowledge. 2.3
Malicious Abuse of Unlearning Requests
Recent work shows that unlearning requests become a security-critical interface when unlearning relies on user-supplied forget sets. In machine unlearning, carefully crafted requests can induce backdoor behaviors (Liu et al., 2024b), and auditing the submitted forget set alone may be insufficient to rule out stealthy attacks (Arazzi et al., 2025). In the LLM setting, adversaries can associate common benign tokens with unlearning behavior to trigger broad utility degradation (Ren et al., 2025b) or make forgetting controllable and reversible (Shang et al., 2025), while benchmark analyses suggest current evaluations may underestimate such risks (Thaker et al., 2025). Taken together, these findings both underscore the need to audit unlearning requests and expose how difficult it is to vet externally supplied forget corpora in practice, motivating settings driven by minimal, auditable anchors.
3
Problem Definition
In existing LLM unlearning settings, users requesting unlearning are required to provide an additional forget set Sf containing the information to be erased. As illustrated in Figure 1, this assumption can raise privacy risks and enables abuse. To address these concerns, we define a more practical user-minimized, corpus-free unlearning paradigm: User Capability: Users requesting unlearning for a target entity Et provide only a minimal anchor that identifies Et , such as a name (e.g., “Taylor
Swift”) or a short description to reduce ambiguity (e.g., “Taylor Swift is a renowned American singersongwriter.”). They do not specify Sf or the unlearning strategy, they only care whether the model can no longer reproduce or correctly reason about Et , even under adversarial attempts to recover the forgotten information. LLM Capability: The LLM (service provider) receives an unlearning request with a minimal anchor and no user-provided forget set. It must autonomously identify what to unlearn for a target entity Et , define an appropriate unlearning scope, and apply an unlearning method U , while meeting the user’s request and limiting utility degradation. Depending on deployment, it may self-construct a forget set Sf (optionally with a neighbor set Sn ) and use a retain set Sr from a general-purpose distribution (e.g., Wikidata) to support scoped unlearning.
4
Our Method
We propose MAGE, a memory-graph guided erasure framework with a two-stage pipeline: Internal Memory Mining reconstructs a weighted local memory graph around Et , and Scoped Supervision Construction converts the graph into a scoped forget set Sf with a companion neighbor set Sn for downstream unlearning. 4.1
Internal Memory Mining
Directly prompting the model to describe Et often yields incomplete and unfocused recall. In corpusfree unlearning, missing memorized facts leads to ineffective forgetting, while noisy generations waste unlearning capacity. Moreover, target knowledge in LLMs is often activated by related contextual cues (Hu et al., 2024; Patil et al., 2023), suggesting that effective recovery should iteratively expand from reliable anchors rather than rely on a single flat response. We therefore iteratively expand from Et and represent the recovered memorization as a weighted local memory graph Gt = (Vt , Et ), where Vt contains salient target-related events, and Et connects co-elicited items with weights reflecting memorization strength. This representation enables controllable expansion, explicit scope definition, and structured supervision synthesis. 4.1.1
Self-Consistent Memorization
We recover candidate Vt by repeated elicitation. Concretely, we query the model N times and apply
Forget everything about Taylor Swift !!!
Unlearning Request
0.1
① What is [Anchor] ? ② What is [Event] ? Let me handle ! (conditioned on it[Anchor])
Et
fearless 0.7
Taylor Swift 0.5
1-Hop Neighbor
V1
0.9
0.7
Iterative Expansion
…
2008
Let me handle it !
Quality Filter
V2 Love Story
Minimal Anchor • Name: Taylor Swift
Scoped Supervision Construction
Internal Memory Mining
American
0.7
0.6
pennsylvania
2-Hop Neighbor
…
0.7
blank space 0.2
Vu
Vl
Et
V1
…
Forget Set:
V1
Edge-to-Event Synthesis Et
Unlearning Method
Question: Which artist released Blank Space in 2014? Answer: Taylor Swift.
Neighbor Set:
New York
Vu Vv Question: Which UK artist included New York as...? Answer: Ed Sheeran.
Media
Vl
event statement
Path Sampling
K-Hop Neighbor
0.6
…
Vi
Ed Sheeran 0.4
Vu
Vv
OK, I’ve forgotten Taylor Swift !
Figure 2: The framework of MAGE. Internal Memory Mining: Given an unlearning request, MAGEmines target memorization to build a strength-weighted local memory graph via iterative expansion. Scoped Supervision Construction: It then performs strength-weighted path sampling and edge-to-event synthesis to generate compact forget and neighbor supervision for downstream unlearning.
entity extraction on each response to obtain candidate mentions. We then score each candidate v by mention frequency: s(v) =
N 1 X I v ∈ Entities(yi ) , N
(1)
i=1
where yi is the i-th elicited response. We interpret s(v) as a memorization strength: events repeatedly activated across independent generations are more likely to reflect stable memorized traces, while one-off mentions are downweighted as noise. 4.1.2 Target-Conditioned Iterative Expansion We construct Gt by iteratively expanding from Et up to K hops. At hop 0, Vt contains only the target node Et . We then elicit N responses anchored at Et , extract events, and score each candidate v by its mention frequency s(v) in Eq. 1. Candidates with s(v) ≥ τ are retained as the 1-hop neighbors of Et . For hop h (h ≥ 1), we further expand from the retained strong nodes u in hop h−1. To keep the expansion on-topic, we perform elicitation conditioned on Et with u as a secondary anchor, extract new candidates, and again retain those with s(·) ≥ τ . We add directed edges (u → v) with weights reflecting co-elicitation salience: w(u, v) = P
c(u, v) , ′ v ′ ∈N (u) c(u, v )
Our elicitation prompts are provided in Appendix B.1, and we also discuss the overhead of memory-graph construction in Section 6.3.4. 4.2
Scoped Supervision Construction
Since Gt is built based on co-elicitation, we construct supervision from paths rather than isolated nodes, which better reflects how target memories are triggered, enabling supervision that is naturally target-grounded and scoped. 4.2.1
Weighted Path Sampling
Prior work suggests that effective unlearning requires allocating stronger unlearning signals to content that is more strongly memorized, while weak or noisy traces contribute little useful supervision (Kuo et al., 2025; Tran et al., 2025; Wang et al., 2025b; Barbulescu and Triantafillou, 2024). Motivated by this, we bias sampling toward high-strength regions of the memory graph, so that frequently sampled supervision aligns with the model’s memorization strength. To focus supervision on strongly memorized content while maintaining coverage, we perform a strength-weighted random walk with exploration. Starting from Et , the transition probability from u to v is:
(2)
where c(u, v) counts how often v is extracted from responses anchored at u, and N (u) denotes the extracted neighbors of u. The resulting weighted graph Gt summarizes what the model tends to recall around Et and which contextual cues most reliably trigger further recall.
p(v | u) ∝ w(u, v) ·
α 1 , 1 + vis(v)
(3)
where vis(v) is the number of times v has been visited in previous walks, and α controls exploration (larger α encourages coverage by downweighting frequently visited nodes). We collect R walks with maximum length L to obtain candidate paths.
We also apply a simple path-quality filter, discarding low-signal paths and retaining paths whose average edge weight is above a threshold:
Table 1: Priors and capabilities of forget-set generation methods. ✓ denotes used; ✗ denotes not used. Prior External Data (Wiki) Extra LLM / Human Correctness Check
|π|−2
X 1 q(π) = w(vi , vi+1 ) ≥ η, |π| − 1
(4)
RWKU
ELUDe
DirectQA
Ours
✓ ✓ ✓
✓ ✓ ✗
✗ ✗ ✓
✗ ✗ ✗
i=0
where π = (v0 , . . . , v|π|−1 ) and v0 = Et . By sampling transitions in proportion to edge weights, the frequency with which nodes appear in sampled paths is roughly proportional to their memorization salience, so more strongly memorized content receives more unlearning supervision. 4.2.2 Edge-to-Event Synthesis For each sampled path (v0 =Et , v1 , . . . , vℓ ), we use a sliding window to extract adjacent node pairs (vi , vi+1 ). Since each pair corresponds to a targetconditioned association mined during memorygraph construction, each pair provides a concrete contextual cue for targeted recall, rather than unconstrained and potentially irrelevant generation. We prompt the LLM to produce a short, atomic event statement that links vi and vi+1 in the context of Et . We then set obj = vi and generate one concise forget sample (QA-style for illustration) grounded in the event statement: the question concerns Et and uses obj as the primary context, while the answer uniquely identifies Et . We compute the unlearning loss only on the answer span, localizing gradient updates and reducing collateral changes. An example is shown below: Question: Which artist released Blank Space in 2014? Answer: Taylor Swift.
To reduce collateral forgetting, we also construct a neighbor set Sn that captures correlated but nontarget information. We sample paths starting from strong neighbors of Et while enforcing that Et does not appear on the sampled path, and generate corresponding QA samples whose answers are the non-target neighbor entities. Sn serves as a boundary constraint: it encourages the model to preserve nearby knowledge while forgetting target-specific content. An example is shown below: Question: Which UK artist included New York as a hidden track on times (2014)? Answer: Ed Sheeran
Our construction prompts are provided in Appendix B.2. Finally, we plug (Sf , Sn ) (and optionally a general retain set Sr ) into an off-the-shelf unlearning
method U . Since our output is standard fine-tuning data, MAGE is model-agnostic and compatible with existing unlearning pipelines.
5
Experimental Settings
5.1
Evaluation Benchmark
We evaluate MAGE on two representative entitylevel unlearning benchmarks, which present two contrasting levels of memory-mining difficulty. RWKU (Jin et al., 2024) targets real-world famous people. Since such knowledge is broadly present in pretrained LLMs, it is often possible to elicit abundant target-related facts and construct a rich memory graph. RWKU evaluates unlearning with four test groups: Forget Set (↓), Neighbor Set (↑), MIA Set, and Utility Set (↑), where arrows indicate whether lower or higher is better (details in Appendix A.3.1). TOFU (Maini et al., 2024) targets synthetic, fictitious authors generated by GPT-4 and introduced via benchmark-specific fine-tuning. Since these entities have limited supporting context and few co-occurrence signals in pretraining, the recovered memorization is often sparse and weakly connected, making it harder to mine a meaningful and coherent memory graph. TOFU evaluates unlearning on a forget split and three utility splits (Retain, Real Authors, and World Facts), which form an increasing relevance gradient from in-domain to out-of-domain (Metric Details in appendix A.3.2). We run experiments on the forget01 setting. 5.2
Forget-Set Generation Baselines
Research on corpus-free forget-set construction remains limited. To contextualize MAGE, we compare against three representative forget-set generation baselines and summarize their reliance on external priors in Table 1: RWKU Uses Wikipedia passages for each target and GPT-4 to generate QA probes, then filters them with an open-source model and manual checks.
Finetune
Method FB
Before
Forget Set ↓ QA AA
All
FB
Neighbor Set ↑ QA All
MIA Set FM ↑ RM ↓
MMLU
BBH
Utility Set ↑ TruthfulQA TriviaQA
AlpacaEval
0.6788
0.6721
0.6829
0.6779
0.6306
0.7484
0.6895
1.9104
2.0433
0.4240
0.2654
0.2875
0.4056
6.2993
GA
RWKU ELUDe DirectQA MAGE
0.5817 0.6440 0.6531 0.6373
0.5511 0.4646 0.6675 0.4288
0.6536 0.6026 0.7030 0.5438
0.5955 0.5704 0.6745 0.5366
0.5600 0.6135 0.6353 0.6263
0.7153 0.6944 0.6979 0.6501
0.6376 0.6540 0.6666 0.6382
3.1075 1.9933 1.9286 1.9885
2.4128 2.1057 2.0572 2.1016
0.3969 0.4167 0.4203 0.4130
0.2346 0.2438 0.1975 0.2515
0.3025 0.3175 0.2900 0.3050
0.3675 0.3232 0.4640 0.2955
6.3109 6.3341 6.2686 6.3446
NPO
RWKU ELUDe DirectQA MAGE
0.6444 0.4290 0.6539 0.4177
0.6128 0.2718 0.6842 0.1978
0.6631 0.4061 0.7139 0.3734
0.6401 0.3690 0.6840 0.3296
0.6103 0.4829 0.6364 0.4279
0.6925 0.4239 0.7213 0.3800
0.6514 0.4534 0.6788 0.4040
2.1667 2.1874 1.9178 2.1054
2.2093 2.2383 2.0494 2.1770
0.4137 0.3977 0.4211 0.4123
0.2022 0.1867 0.2099 0.2438
0.2925 0.3525 0.2900 0.3300
0.4053 0.1526 0.4658 0.1656
6.3267 6.3930 6.2765 6.3849
GA+GD
RWKU ELUDe DirectQA MAGE
0.5108 0.5859 0.5468 0.5928
0.4583 0.4006 0.6038 0.3644
0.6075 0.5257 0.6772 0.5012
0.5255 0.5041 0.6092 0.4861
0.5691 0.5750 0.6464 0.5882
0.7214 0.5864 0.6953 0.5788
0.6453 0.5807 0.6709 0.5835
7.1499 1.9600 1.8484 2.0119
2.4194 2.0716 1.9758 2.1121
0.3984 0.4152 0.4174 0.4159
0.2191 0.1466 0.2809 0.2741
0.3050 0.3350 0.2775 0.3050
0.3708 0.2820 0.4910 0.2469
6.2991 6.3393 6.2556 6.3499
GA+KL
RWKU ELUDe DirectQA MAGE
0.5108 0.6170 0.5812 0.6319
0.4644 0.4475 0.6152 0.4364
0.6128 0.5797 0.6968 0.5538
0.5293 0.5480 0.6311 0.5407
0.5650 0.6148 0.6239 0.6019
0.7222 0.6493 0.7064 0.6295
0.6436 0.6320 0.6651 0.6157
7.1544 1.9768 1.9559 2.0046
2.4340 2.0909 2.0635 2.1068
0.4050 0.4167 0.4152 0.4196
0.2160 0.1003 0.2083 0.2358
0.3050 0.3175 0.2850 0.3000
0.3718 0.3151 0.4517 0.3291
6.3051 6.3354 6.2774 6.3436
Table 2: Unlearning performance with different fine-tuning strategies and forget-set generation methods on Llama2-7b-chat. Bold denotes the best result and underlined denotes the second best. Arrows indicate the direction of preference (↓ lower is better; ↑ higher is better).
ELUDe (Choi et al., 2025) Uses the top-viewed Wikipedia pages as targets, generates paragraphlevel QA pairs with GPT-4o, and deduplicates via Sentence-Transformer similarity.
Our evaluation under the proposed corpus-free unlearning paradigm is designed to answer the following research questions. RQ1: Can LLM-recovered supervision match the forgetting performance of supervision built from ground-truth, externally provided data? RQ2: Does the recovered memory reflect what the LLM actually memorizes about the target entity? RQ3: What properties of the recovered data affect unlearning outcomes, including hallucinations, cross-model differences, and knowledge coverage?
lines that rely on externally provided supervision, supporting the feasibility of corpus-free unlearning (RQ1). Concretely, MAGE achieves the lowest Forget Set-All in three of four strategies (GA, NPO, GA+GD) and remains near the top under GA+KL, indicating strong forgetting effectiveness. While DirectQA occasionally yields higher Neighbor Set-All and better scores on a few utility metrics, MAGE consistently improves forgetting over DirectQA (lower Forget-All) with comparable utility retention, and it often attains the best AlpacaEval among corpus-free methods. We further compare MAGE with the original TOFU supervision and the strongest corpus-free baseline, DirectQA in Table 3. Across both GA and NPO, MAGE produces a forget set that yields consistently stronger forgetting than DirectQA (lower Prob/ROUGE and higher Forget Q.), indicating that the mined memory graph provides more effective, target-specific supervision than direct recall prompting. Due to space limitations, GA+GD and GA+KL results are reported in appendix C.1. We additionally evaluate RWKU and TOFU on smallerscale models (Phi-3-mini-4k and Llama-3.2-1B) in appendix C.2.
6.1
6.2
DirectQA (Ma et al., 2025) Asks the target model to self-generate questions and answers about the entity, and keeps only self-verified QA pairs. We evaluate MAGE with four fine-tuning strategies: GA, NPO, GA+GD, and GA+KL, detailed in appendix A.4. Implementation details are provided in Appendix A.1.
6
Experimental Results
Main Unlearning Results
Table 2 reports RWKU results under four finetuning strategies with different forget-set generation methods. For fair comparison with ELUDe, we average results over the eight overlapping targets shared by RWKU and ELUDe. Overall, MAGE is stable and competitive in the corpus-free setting, and it reaches performance close to base-
Memory Mining Analysis
We analyze memory mining quality using two complementary protocols, as the availability of prior knowledge in LLMs differs across benchmarks. Entity Similarity. For RWKU, which targets real-world public figures, target-related knowledge in pretrained LLMs is broad and entangled, and no
Baseline
Generation Method Prob↓
ROUGE↓
Forget TruthRatio↑
Forget Q.↑
RS Score↑
RAS Score↑
Utility WFS Score↑
Model Utility↑
0.9951
0.9493
0.5306
0.0013
0.7133
0.1667
0.0508
0.6277
GA
TOFU DirectQA MAGE
0.0353 0.4260 0.1492
0.0714 0.4521 0.4500
0.5903 0.5029 0.5285
0.1650 0.0286 0.0541
0.1906 0.5378 0.4449
0.1922 0.0830 0.0046
0.0725 0.0212 0.0024
0.3309 0.5783 0.5155
NPO
TOFU DirectQA MAGE
0.0359 0.4393 0.0732
0.0338 0.4933 0.2487
0.6272 0.5637 0.5670
0.1650 0.0541 0.0971
0.1638 0.5204 0.2613
0.2402 0.0817 0.0384
0.0948 0.0220 0.0046
0.2951 0.5743 0.3949
Before
Table 3: Unlearning performance on TOFU-finetuned Llama-2-7b-chat. (↓ lower is better; ↑ higher is better)
ELUDe
0.25 0.25 0.33
0.16
MAGE
0.37
0.27
0.37
0.16
0.27 0.27
0.27
RW KU ELU De Dir ec tQ A MA GE
DirectQA
Frequency Similarity
0.33
0.27 0.27 0.69
0.67
0.38
0.86
1.0
0.69
0.38
0.67
0.86
0.6
0.61
0.4
0.61
Attributes
Freq
0.8
0.2 0.0
RW KU ELU De Dir ec tQ A MA GE
Entity Overlap RWKU
Figure 3: Average entity overlap (Top-50 Jaccard) and frequency-distribution similarity across RWKU.
ground-truth memorization is available. We therefore use ELUDe as a proxy reference and measure the similarity between entity memories mined by different data-generation methods. Specifically, we extract entities from the forget set of each datageneration method and compare them from two perspectives: Entity Overlap, the Jaccard overlap of the top-50 entities ranked by frequency, and Frequency Similarity, the cosine similarity between normalized entity-frequency distributions after aligning entities to the union set. Figure 3 shows the similarity matrices averaged over all targets. Overall, entity overlap is low across methods, reflecting the entangled and diverse nature of real-world entity memories. In contrast, frequency similarity is notably higher, and MAGE achieves the strongest agreement with ELUDe (0.86), suggesting its mined entity salience best matches the proxy reference and providing evidence that our mined entity memories are closer to a plausible reference of pretrained memorization. GT Attribute Recovery. TOFU provides synthetic author profiles with defined ground-truth attributes, enabling a controlled evaluation of memory mining quality. We extract profile attributes from the generated forget set and compare them against the original TOFU profiles to quantify recovery accuracy; detailed extraction examples are deferred to the appendix D.1). As shown
BirthPlace Gender BirthYear Genre Awards Par_Job Books Books
2 1 1 3 1 4 3 1
DirectQA
MAGE
Covered
F1
Covered
F1
✓
1.00 0 0 0 0 0.80 0 0
✓
0.80 0 0.67 0.67 1.00 0 0.80 0.33
✓ ✓
✓ ✓ ✓ ✓ ✓
Table 4: Comparison of recovered attributes against TOFU ground-truth profiles.
in Table 4, DirectQA achieves limited coverage, suggesting that direct querying retrieves only a narrow subset of the injected profile information. In contrast, MAGE recovers a broader range of key attributes, highlighting the benefit of exploiting graph-structured memory signals. Nevertheless, MAGE still misses some non-entity-driven attributes, which may require targeted prompts or additional anchor information to elicit reliably. Across RWKU and TOFU, the recovered memory from MAGE aligns well with plausible or ground-truth target knowledge, indicating that it largely reflects what the LLM actually memorizes about the target entity (RQ2). Memory Variation. Moreover, we observe notable cross-model differences in recovered memory graphs: for “Taylor Swift”, LLaMA-3-8b-instruct recalls “Taylor Alison Swift” as a close neighbor, while LLaMA-2-7b-chat does not, motivating model-specific forget supervision (Appendix D.2). 6.3
Forget Construction Analysis
To answer RQ3, we examine how the properties of recovered supervision affect unlearning outcomes in the corpus-free paradigm. Unlike supervised pipelines with externally defined forget sets, corpus-free unlearning relies on model-recovered data whose quality can vary across targets and base
0.7
Forget-FB Forget-QA Forget-AA Forget-All Neighbor-FB Neighbor-QA Neighbor-All
0.6 0.5 0.4 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Figure 4: Impact of the proportion of correct statements in the forget set on unlearning performance.
models. Consequently, unlearning performance depends not only on the unlearning algorithm, but also on the recovered supervision itself. 6.3.1
Hallucination
Since corpus-free unlearning relies on LLMgenerated supervision, the recovered forget set may include hallucinated QA pairs that sound plausible but are factually incorrect. To quantify the impact of such noise, we use GPT-4 to label each QA item in MAGE ’s RWKU forget set as correct or incorrect, then create mixed forget sets with correctness ratios from 0% to 100%. We run GA-based unlearning on each mixture and report the results in Figure 4. Surprisingly, performance changes only marginally across mixing ratios, suggesting that unlearning is not highly sensitive to the factual correctness of individual QA items. This further indicates that supervision aligned with the model’s memorized patterns may matter more than strict factual accuracy. Additional results on smaller models, which typically exhibit more severe hallucinations, are provided in the appendix C.2). 6.3.2
Influence of the Neighbor Set
We further study how the choice of the neighbor set affects unlearning performance. In Table 2, GA+GD and GA+KL use our recovered forget set together with our constructed neighbor set. We then replace the neighbor set with the Wiki data used by the baseline. As shown in Table 5, this replacement consistently degrades performance on the neighbor set, indicating that the distribution of the neighbor data plays an important role in preserving unlearning performance. 6.3.3
Knowledge Coverage
In Table 2, we impose a high forget-set coverage threshold (e.g., ≥90%) to ensure broad supervision over the target entity and enable thorough unlearning. We then vary this requirement and report the
Forget-Avg
Neighbor-Avg
GA+GD
Wiki Neighbor
0.4834 0.4861
0.5644 0.5835
GA+KL
Wiki Neighbor
0.5353 0.5407
0.6030 0.6157
Table 5: Impact of the Neighbor Set on Unlearning Performance
results in Figure 6 (Appendix). As the threshold is relaxed, reduced coverage leads to weaker unlearning, while neighbor performance remains relatively stable. This aligns with prior findings that insufficient forget-set coverage is a key bottleneck for entity-level unlearning (Ma et al., 2025). 6.3.4 Recovery Overhead MAGE introduces a one-time memory-graph construction step, which is training-free and consists only of offline LLM interactions. We measure its cost by the number of elicitation iterations I, which is typically comparable to the number of recovered nodes |Vt |. If each iteration repeats a sub-query up to N times, then QLLM ≤ N · Imax and E[QLLM ] ≈ N ·Imean . Empirically, RWKU requires 88.36 iterations per entity on average, while TOFU needs only 13.75, indicating modest overhead in knowledge-sparse, user-facing cases. The recovered graph also provides a budgeted stopping criterion and can be reused across unlearning methods and repeated evaluations, amortizing its cost while improving coverage over DirectQAstyle probing. In contrast, many alternatives incur hidden overheads such as external data collection/curation, retrieval and indexing, or additional filtering and judging calls, which MAGE avoids by relying on bounded model interactions.
7
Conclusion
We presented MAGE, a memory-graph guided erasure framework for user-minimized, corpus-free LLM unlearning. Motivated by the auditability and security risks of user-supplied forget sets, MAGE recovers target-related memorization, organizes it into a weighted local memory graph, and selfgenerates scoped supervision that plugs into standard unlearning methods without access to the training corpus. Experiments on TOFU and RWKU show that MAGE achieves supervised-level forgetting with comparable utility retention using only self-generated data, supporting a practical and auditable workflow driven by minimal anchors.
References Marah I. Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219. Marco Arazzi, Antonino Nocera, and 1 others. 2025. When forgetting triggers backdoors: A clean unlearning attack. arXiv preprint arXiv:2506.12522. George-Octavian Barbulescu and Peter Triantafillou. 2024. To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097. Karuna Bhaila, Minh-Hao Van, and Xintao Wu. 2024. Soft prompting for unlearning in large language models. arXiv preprint arXiv:2406.12038. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650. Minseok Choi, Daniel Rim, Dohyun Lee, and Jaegul Choo. 2025. Opt-out: Investigating entity-level unlearning for large language models via optimal transport. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28280–28297, Vienna, Austria. Association for Computational Linguistics. Jai Doshi and Asa Cooper Stickland. 2024. Does unlearning truly unlearn? a black box evaluation of llm unlearning methods. arXiv preprint arXiv:2411.12103. R Eldan and M Russinovich. 2023. Who’s harry potter? approximate unlearning in llms, arxiv. arXiv preprint arXiv:2310.02238. Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. 2024. Jogging the memory of unlearned llms through targeted relearning attacks. In Neurips Safe Generative AI Workshop 2024. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504. Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Rwku: Benchmarking realworld knowledge unlearning for large language models. Advances in Neural Information Processing Systems, 37:98213–98263. Martin Kuo, Jingyang Zhang, Jianyi Zhang, Minxue Tang, Louis DiValentin, Aolin Ding, Jingwei Sun,
William Chen, Amin Hass, Tianlong Chen, and 1 others. 2025. Proactive privacy amnesia for large language models: Safeguarding pii with negligible impact on model utility. arXiv preprint arXiv:2502.17591. Bo Liu, Qiang Liu, and Peter Stone. 2022. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, and 1 others. 2025. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pages 1–14. Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. 2024a. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058. Zihao Liu, Tianhao Wang, Mengdi Huai, and Chenglin Miao. 2024b. Backdoor attacks via machine unlearning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14115–14123. Nicola Lucchi. 2024. Chatgpt: a case study on copyright challenges for generative artificial intelligence systems. European Journal of Risk Regulation, 15(3):602–624. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. 2024. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835. Weitao Ma, Xiaocheng Feng, Weihong Zhong, Lei Huang, Yangfan Ye, Xiachong Feng, and Bing Qin. 2025. Unveiling entity-level unlearning for large language models: A comprehensive analysis. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5345–5363, Abu Dhabi, UAE. Association for Computational Linguistics. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121. NPR. 2025. ’the new york times’ takes openai to court. chatgpt’s future could be on the line. Accessed: 202507-25. Vaidehi Patil, Peter Hase, and Mohit Bansal. 2023. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. arXiv preprint arXiv:2309.17410. Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. 2023. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741. Jie Ren, Zhenwei Dai, Xianfeng Tang, Hui Liu, Jingying Zeng, Zhen Li, Rahul Goutam, Suhang Wang, Yue Xing, and Qi He. 2025a. A general framework to enhance fine-tuning-based llm unlearning. arXiv preprint arXiv:2502.17823. Jie Ren, Zhenwei Dai, Xianfeng Tang, Yue Xing, Shenglai Zeng, Hui Liu, Jingying Zeng, Qiankun Peng, Samarth Varshney, Suhang Wang, and 1 others. 2025b. Keeping an eye on llm unlearning: The hidden risk and remedy. arXiv preprint arXiv:2506.00359. Bingqi Shang, Yiwei Chen, Yihua Zhang, Bingquan Shen, and Sijia Liu. 2025. Forgetting to forget: Attention sink as a gateway for backdooring llm unlearning. arXiv preprint arXiv:2510.17021. Pratiksha Thaker, Shengyuan Hu, Neil Kale, Yash Maurya, Zhiwei Steven Wu, and Virginia Smith. 2025. Position: Llm unlearning benchmarks are weak measures of progress. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 520–533. IEEE. Bozhong Tian, Xiaozhuan Liang, Siyuan Cheng, Qingbin Liu, Mengru Wang, Dianbo Sui, Xi Chen, Huajun Chen, and Ningyu Zhang. 2024. To forget or not? towards practical knowledge unlearning for large language models. arXiv preprint arXiv:2407.01920. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Toan Tran, Ruixuan Liu, and Li Xiong. 2025. Tokens for learning, tokens for unlearning: Mitigating membership inference attacks in large language models via dual-purpose training. arXiv preprint arXiv:2502.19726. Huazheng Wang, Yongcheng Jing, Haifeng Sun, Yingjie Wang, Jingyu Wang, Jianxin Liao, and Dacheng Tao. 2025a. Erasing without remembering: Implicit knowledge forgetting in large language models. arXiv preprint arXiv:2502.19982. Lingzhi Wang, Xingshan Zeng, Jinsong Guo, Kam-Fai Wong, and Georg Gottlob. 2025b. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 843–851. Haoming Xu, Ningyuan Zhao, Liming Yang, Sendong Zhao, Shumin Deng, Mengru Wang, Bryan Hooi, Nay Oo, Huajun Chen, and Ningyu Zhang. 2025.
Relearn: Unlearning via learning for large language models. arXiv preprint arXiv:2502.11190. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. Large language model unlearning. Advances in Neural Information Processing Systems, 37:105425–105475. Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. 2024. A closer look at machine unlearning for large language models. arXiv preprint arXiv:2410.08109. Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. 2024a. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. AI and Ethics, pages 1–10. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024b. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868.
A
Experiment Settings
A.1
Implementation Details
Models. We run RWKU experiments on the base pretrained models Llama-2-7b-chat (Touvron et al., 2023) and Phi-3-mini-4k-instruct (Abdin et al., 2024). For TOFU, we use benchmark-specific fine-tuned checkpoints released by prior work on Llama-2-7b-chat1 and Llama-3.2-1B-Instruct2 since TOFU requires the injected author profiles to be present in the model. Fine-tuning. All unlearning pipelines are implemented with LoRA on NVIDIA GeForce RTX 4090 GPU. We use LoRA rank r=8 (RWKU) and r=16 (TOFU), with α=16. To enable fair comparisons across different forget-set generation methods under the same unlearning strategy, we fix the learning rate within each strategy and adjust the number of epochs to keep the total number of optimization steps approximately matched, accounting for differences in dataset size. For RWKU, we largely follow ELUDe’s tuning protocol but slightly adjust learning rates for LoRA stability: GA uses 8×10−6 for 4 epochs, NPO uses 5×10−5 for 4 epochs, and GA+GD / GA+KL use 5×10−6 for 3 epochs, while epochs for other forget-set sources are scaled to match steps. For TOFU, we fix the learning rate to 2×10−4 across all settings and tune epochs to keep optimization steps comparable; the TOFU benchmark supervision uses 4 (GA), 6 (NPO), 3 (GA+GD), and 6 (GA+KL) epochs, and we adjust DirectQA and MAGE accordingly. A.2
Method Hyperparameters
Internal Memory Mining. We build the local memory graph by iterative, target-conditioned elicitation and frequency-based filtering. We use N =10 elicitation queries per anchor, keep a candidate node if its memorization-strength score satisfies s(·) ≥ τ , with τ =0.2 on RWKU and τ =0.3 on TOFU. We expand the graph up to K hops, with K=2 on RWKU and K=3 on TOFU. In addition to a fixed K, we observe that as expansion proceeds, outer-hop neighbors tend to have fewer strong connections back to Et ; this suggests an adaptive stopping rule that halts expansion when the aggregate strength of edges pointing back to Et 1
https://huggingface.co/locuslab/tofu_ft_ llama2-7b 2 https://huggingface.co/open-unlearning/tofu_ Llama-3.2-1B-Instruct_full
drops below a threshold. Scoped Supervision Construction. We set the exploration coefficient to α=1, the maximum walk length to L=5, and the path-quality threshold to η=0.3. We run R=200 walks for RWKU to match the supervision scale of ELUDe, and R=20 walks for TOFU due to its sparser recovered memory. A.3
Evaluation Metrics
A.3.1 RWKU Evaluation Metrics RWKU (Jin et al., 2024) reports results in four groups: Forget Set (↓), Neighbor Set (↑), MIA Set, and Utility Set (↑). Forget Set (↓). RWKU measures whether a model still reveals target knowledge using three probe types: (i) FB (fill-in-the-blank) probes that mask knowledge points in sentences from the target’s Wikipedia page and ask the model to complete the blank; (ii) QA (question-answer) probes that directly query target-related facts; and (iii) AA (adversarial-attack) probes that attempt to elicit forgotten knowledge via prompt attacks (e.g., prefix injection, affirmative suffix, role playing, reverse query, synonym manipulation, background hints, in-context demonstrations, and cross-lingual queries). For FB/QA/AA, RWKU uses ROUGE-L recall between the model output and the groundtruth answer; lower is better for unlearning efficacy. We additionally report All as the aggregate over the three forget subsets, consistent with RWKU’s reporting format. Neighbor Set (↑). To assess locality (i.e., not over-forgetting beyond the target), RWKU constructs neighbor probes that are closely related to the target but outside the intended forgetting scope. Similar to the forget set, it includes Neighbor FB and Neighbor QA, evaluated by ROUGE-L recall; here higher is better, indicating less collateral damage on neighboring knowledge. We report All as the aggregate over the neighbor subsets. MIA Set (FM ↑, RM ↓). RWKU further evaluates whether target knowledge still appears as pretraining members via membership inference attacks (MIAs). It defines a forget-member set (FM) containing target-related training fragments and a retain-member set (RM) with unrelated member fragments. RWKU provides four MIA scorers (LOSS, Zlib entropy, Min-K% Prob, and MinK%++ Prob) and primarily reports LOSS-based scores; higher scores indicate the text is less likely
Finetune
Method FB
Before
Forget Set ↓ QA AA
All
FB
Neighbor Set ↑ QA All
MIA Set FM ↑ RM ↓
MMLU
BBH
Utility Set ↑ TruthfulQA TriviaQA
AlpacaEval
0.6729
0.7290
0.6839
0.6953
0.6251
0.5819
0.6035
1.8448
1.9326
0.6864
0.4090
0.3775
0.3985
6.0196
GA
RWKU ELUDe DirectQA MAGE
0.6597 0.6694 0.6675 0.6236
0.7404 0.6799 0.6541 0.6724
0.6804 0.6625 0.6296 0.6355
0.6935 0.6706 0.6504 0.6438
0.6227 0.6113 0.6377 0.6083
0.5692 0.5793 0.5534 0.5590
0.5959 0.5953 0.5956 0.5837
1.8645 1.8567 2.0591 1.8663
1.9411 1.9383 2.0101 1.9439
0.6879 0.6857 0.6784 0.6842
0.3133 0.3148 0.4336 0.4383
0.3800 0.3925 0.3775 0.3975
0.4077 0.2670 0.3506 0.2460
6.0383 6.1704 6.0679 6.1535
NPO
RWKU ELUDe DirectQA MAGE
0.6245 0.6793 0.6497 0.6283
0.7129 0.5073 0.6870 0.5097
0.6561 0.4865 0.6826 0.4550
0.6645 0.5577 0.6731 0.5310
0.6300 0.5971 0.6183 0.6348
0.5688 0.4836 0.5561 0.5491
0.5994 0.5404 0.5872 0.5920
2.1448 1.9096 1.9156 1.8984
2.1175 1.9610 1.9630 1.9548
0.6827 0.6842 0.6849 0.6835
0.4012 0.3858 0.4213 0.4290
0.3900 0.4475 0.3875 0.4325
0.2253 0.1546 0.3771 0.1642
6.2853 6.4073 6.0951 6.3429
GA+GD
RWKU ELUDe DirectQA MAGE
0.6356 0.6356 0.6422 0.6484
0.7296 0.6163 0.7201 0.7042
0.6675 0.6440 0.6545 0.6515
0.6776 0.6386 0.6723 0.6680
0.5952 0.6084 0.5827 0.5995
0.5772 0.5624 0.5702 0.5646
0.5862 0.5854 0.5765 0.5821
1.8965 1.8513 1.8635 1.8592
1.9507 1.9342 1.9436 1.9413
0.6871 0.6893 0.6820 0.6857
0.3441 0.2238 0.3441 0.4306
0.3750 0.3975 0.3825 0.3825
0.4245 0.2859 0.4515 0.4408
6.0307 6.1453 5.8489 5.9923
GA+KL
RWKU ELUDe DirectQA MAGE
0.6293 0.6828 0.6608 0.6330
0.7296 0.6847 0.6529 0.7135
0.6684 0.6722 0.6535 0.6694
0.6758 0.6799 0.6557 0.6720
0.6131 0.6072 0.6685 0.5808
0.5746 0.5710 0.5491 0.5691
0.5938 0.5891 0.6088 0.5749
1.8951 1.8559 1.8807 1.8535
1.9506 1.9376 1.9429 1.9370
0.6849 0.6879 0.6857 0.6864
0.3997 0.4090 0.3426 0.4306
0.3750 0.3950 0.3850 0.3850
0.4113 0.2730 0.3752 0.2969
6.0506 6.1676 6.0251 6.1099
Table 6: Unlearning performance with different fine-tuning strategies and forget-set generation methods on Phi-3mini-4k-instruct. Bold denotes the best result and underlined denotes the second best. Arrows indicate the direction of preference (↓ lower is better; ↑ higher is better).
to be memorized. Therefore, effective unlearning should yield higher FM (target fragments look nonmember) while keeping lower RM (unrelated member fragments still look like members), matching the (FM ↑, RM ↓) convention. Utility Set (↑). RWKU measures general capability preservation with five standard tasks, where higher is better: MMLU (Gen) using 5-shot accuracy via answer perplexity, BBH (Rea) using chainof-thought prompting with 3-shot examples and exact-match scoring, TruthfulQA (Tru) using MC1 6-shot accuracy, TriviaQA (Fac) using 6-shot F1, and AlpacaEval (Flu) using the weighted average of bi-/tri-gram entropies as a fluency proxy. A.3.2
correct)Panswers Apert , defined as log Rtruth = 1 1 1 â∈Apert |â| log P (â | q)− |ã| log P (ã | q), |Apert | where ã and Apert are constructed to control for phrasing effects (Maini et al., 2024); and (iv) Forget Q.↑, the forget quality score computed via a two-sample KS-test on the TruthRatio distributions of the unlearned model versus a retain-only model, using the resulting p-value as the forgetting score (higher p implies the two models are harder to distinguish, i.e., better forgetting) (Maini et al., 2024). Consistent with the TOFU convention, Forget Set uses “lower Prob/ROUGE, higher TruthRatio” as the desired direction (Maini et al., 2024).
TOFU Evaluation Metrics
TOFU (Maini et al., 2024) evaluates unlearning with four splits: Forget Set (target to forget), and three non-forget splits for utility preservation—Retain Set, Real Authors, and World Facts—forming a relevance gradient from neardomain to far-domain (Maini et al., 2024). Forget metrics. On the Forget Set, we report: (i) Prob↓, the length-normalized conditional likelihood of the ground-truth answer under the model, computed as P (a | q)1/|a| (lower is better when forgetting) (Maini et al., 2024); (ii) ROUGE↓, the ROUGE-L recall between the greedy-decoded answer and the ground-truth answer (lower indicates less recoverable target content) (Maini et al., 2024); (iii) TruthRatio↑, a likelihood-ratio style score that contrasts a paraphrased correct answer ã against a set of GPT-generated perturbed (factually in-
Utility metrics. For utility preservation, TOFU rescales metrics so that higher is better on non-forget splits (e.g., converting TruthRatio by max(0, 1 − Rtruth ) on non-forget sets) and aggregates them into Model Utility↑ by taking the harmonic mean over the three metrics (Prob/ROUGE/TruthRatio) across the three nonforget splits (Retain, Real Authors, World Facts), yielding nine values in total; the harmonic mean penalizes any single metric collapse (Maini et al., 2024). In addition, we report per-split summary scores in the same order as the table header: RS Score↑ (Retain Set), RAS Score↑ (Real Authors Set), and WFS Score↑ (World Facts Set). Each score is computed on its corresponding split as the harmonic mean of Prob., ROUGE, and an accuracyoriented term (Acc.), so that higher indicates better overall utility on that split.
A.4
Basic Fine-tuning Strategies
MAGE is a frame work can be integrated into existing unlearning pipelines, we choose the following: Gradient Ascent (GA) inverts the standard training objective for forget set Sf by ascending the gradient of the loss function: θ ← θ + η∇θ LLM (Sf ) Here, θ represents the model’s parameters, η is the unlearning rate (a hyperparameter that controls the step size), and ∇θ LLM (Sf ) is the gradient of the language model loss calculated on the forget set Sf . Negative Preference Optimization (NPO) trains the model to treat the generation of forgotten content as a rejected behavior, contrasting it with the behavior of a reference model: πθref (y|x) LNPO = −Ex∼Sf log σ β log πθ (y|x) Here, πθ is the policy of the model being trained, πθref is the policy of a frozen reference model (usually the model state before unlearning), β is a temperature parameter controlling the strength of the preference, and σ is the sigmoid function. We also consider the utility-keeping method as following: Gradient Descent (GD) applies the standard training objective on a curated retain set Sr , ensuring that the targeted unlearning process does not degrade overall utility: θ ← θ − η∇θ LLM (Sr ) Here, Sr is the retain set and θ represents the model’s parameters. Kullback–Leibler Divergence (KL) acts as a regularization constraint to maintain model utility. It penalizes the deviation of the updated model’s policy πθ from that of a static reference model πθref on the retain set: LKL = Ex∼Sr [DKL (πθref (y|x)||πθ (y|x))] Here, DKL denotes the Kullback–Leibler divergence, which measures the difference between the probability distributions of the reference policy πθref and the current model policy πθ over the retain set Sr . Each unleanring pipelines can be paired with utility-keeping method.
B
Prompt Template
B.1
Prompt for internal memory mining
Prompt template for Et ’s 0-hop neighbor information mining. Recall information from your own internal knowledge. Target: {target} Write 5-10 atomic statements about {target}.
Prompt template for h-hop neighbor information mining, given the h − 1-hop neighbor entity and the unlearning target. Recall information from your own internal knowledge. Target: {target} Neighbor: {neighbor} Write 5-10 SHORT atomic statements specifically about how {neighbor} relates to {target}.
B.2
Prompt for corpus construction
Prompt template for event statement construction, given the two neighbor entities from sampled path and the unlearning target. Please act as an information assistant to help users learn about pertinent details regarding the target. Given an anchor target, according to known knowledge about the target, and two key event about the target, please provide ONE concise factual statement about the target’s main information. The statement should highlight details about the {target} that users may find important. Do NOT invent fictional or hypothetical scenarios. If you are not confident the connection is real, output UNKNOWN. The statement should explicitly include BOTH eventssńames (do not use pronouns), and keep it to ONE sentence. Anchor Target: {target} Event 1: {event_1} Event 2: {event_2}
Prompt template for QA-style forget set construction, given the event statement, context obj and the unlearning target. Please act as an information assistant to help users learn about pertinent details. Given a factual statement about the target, rewrite it into ONE QA pair. The question should highlight an important detail from the statement about Contral Context. The question MUST include the target entity’s name (do not use pronouns). Target Entity: {target} Central Context (Obj): {obj} Statement: {event}
Baseline
Generation Method Prob↓
ROUGE↓
Forget TruthRatio↑
Forget Q.↑
RS Score↑
RAS Score↑
Utility WFS Score↑
Model Utility↑
0.9951
0.9493
0.5306
0.0013
0.7133
0.1667
0.0508
0.6277
GA+GD
TOFU DirectQA MAGE
0.1031 0.9843 0.1608
0.1631 0.9198 0.4416
0.7365 0.5395 0.5283
0.2657 0.0068 0.1650
0.1834 0.7015 0.4748
0.5251 0.1852 0.0143
0.3904 0.0616 0.0048
0.3497 0.6333 0.5436
GA+KL
TOFU DirectQA MAGE
0.4709 0.2916 0.1580
0.4409 0.4092 0.3359
0.5644 0.4950 0.4716
0.0541 0.0541 0.0971
0.5923 0.5069 0.4767
0.1617 0.0920 0.0057
0.0516 0.0272 0.0028
0.5943 0.5688 0.5362
Before
Table 7: Unlearning performance(GA+GD, GA+KL) on TOFU-finetuned Llama-2-7b-chat. (↓ lower is better; ↑ higher is better) Baseline
Generation Method
Before
Prob↓
ROUGE↓
Forget TruthRatio↑
Forget Q.↑
RS Score↑
RAS Score↑
Utility WFS Score↑
Model Utility↑
0.7742
0.6469
0.5077
0.0068
0.5664
0.2048
0.1236
0.5735
GA
TOFU DirectQA MAGE
0.2995 0.5684 0.2799
0.3632 0.4337 0.5457
0.6180 0.5530 0.4635
0.2657 0.0143 0.0286
0.4173 0.5033 0.4531
0.5867 0.2418 0.0059
0.5741 0.1948 0.0028
0.5222 0.5586 0.4948
NPO
TOFU DirectQA MAGE
0.2197 0.7731 0.4493
0.3577 0.6452 0.6323
0.6328 0.5094 0.5033
0.0971 0.0068 0.0286
0.3290 0.5649 0.5365
0.1771 0.2088 0.0239
0.0832 0.1271 0.0097
0.4371 0.5733 0.5441
GA+GD
TOFU DirectQA MAGE
0.2912 0.5588 0.3165
0.3154 0.4149 0.5495
0.7063 0.5750 0.5030
0.2657 0.0068 0.1650
0.3342 0.4463 0.4894
0.5469 0.4221 0.0096
0.4962 0.3572 0.0043
0.4592 0.5230 0.5096
GA+KL
TOFU DirectQA MAGE
0.1674 0.6929 0.5174
0.2680 0.5873 0.5727
0.4734 0.5138 0.4741
0.4046 0.0068 0.0541
0.4772 0.5576 0.5467
0.1876 0.2068 0.0227
0.1108 0.1357 0.0121
0.5453 0.5749 0.5449
Table 8: Unlearning performance with different forget set generation methods on TOFU using a TOFU-finetuned Llama-3.2-1B-Instruct model. (↓ lower is better; ↑ higher is better)
C
More Results
C.1
Unlearning Results in Large Model
Table 7 details the results on Llama-2-7b-chat. Under the GA+GD setting, MAGE achieves a superior balance compared to baselines: it avoids the unlearning failure seen in DirectQA while maintaining significantly higher utility than the TOFU supervision, which suffers from severe catastrophic forgetting. Notably, under the GA+KL constraint, MAGE demonstrates the strongest forgetting capabilities among all methods, achieving the lowest Prob and ROUGE scores. This confirms that our mined memory graph provides precise supervision even under strict regularization. C.2
Unlearning Results in Smaller Model
C.2.1
RWKU Evaluation on Phi-3-mini-4k
Table 6 shows results on Phi-3-mini-4k. MAGE demonstrates robust forgetting across objectives, notably achieving the lowest aggregate Forget Set score in GA and significantly outperforming baselines in NPO. Crucially, it offers a superior utilityforgetting trade-off: unlike ELUDe which suffers severe reasoning degradation (e.g., sharp BBH drop in GA+GD), MAGE maintains high performance
across utility metrics, validating its precision in targeting specific memories without compromising general capabilities. C.2.2
TOFU Evaluation on Llama-3.2-1B
Table 8 presents the unlearning results on Llama3.2-1B-Instruct. MAGE exhibits a distinct advantage over the DirectQA baseline, which struggles to induce effective forgetting in this smaller architecture (e.g., yielding retention scores comparable to the pre-unlearning state under NPO and GA+GD). In contrast, MAGE consistently reduces target knowledge retention by leveraging mined internal memory signals, providing the necessary supervision that standard prompting lacks, while even outperforming the gold-standard TOFU supervision in specific settings (e.g., GA).
D
Additional Analysis
D.1
TOFU Recovery Details
We provide detailed TOFU attribute recovery results using “Basil Mahfouz Al-Kuwaiti” as an illustrative example in Table 9.
Attributes
DirectQA
Freq Gold
MAGE
Pred
F1
kuwait city, kuwait;
1.00
BirthPlace
2
kuwait city, kuwait
Gender
1
male
0
BirthYear
1
1956
0
Genre
3
french literature
Awards
1
prix goncourt
Arabic culture and Middle Eastern landscapes;
Covered Pred ✓
kuwait city;
Covered
0.80
✓
0
✓
0 0
florist; game developer; accountant;
F1
1931; 1956;
0.67
✓
french literature; literary fiction;
0.67
✓
prix goncourt;
1.00
✓
Par_Job
4
florist; game developer;
Books
3
promise by the seine; le petit sultan;
0
promise by the seine; le petit sultan; the parisian promise;
0.80
✓
Books
1
river of jasmine; the nightingale of paris;
0
the oasis of paris; mirror of paris; the nightingales song; the nightingale of paris;
0.33
✓
0.80
✓
0
Table 9: Qualitative comparison of recovered attributes against TOFU ground-truth profiles.
0.7 Forget-All Neighbor-All
0.6 0.5 0.4
0.45
0.57
0.68
0.82 0.86 0.91
Figure 6: Impact of Memory Graph Coverage on Unlearning Performance.
anchors—most notably “december 13 1989” and “american”, their relative strengths differ markedly. Notably, LLaMA-3-8b consistently recalls “Taylor Alison Swift” as a salient neighbor, which is absent in the others. These differences, likely driven by training data and architectural choices, suggest that forget supervision should be regenerated per target model to ensure effective unlearning. D.3
Figure 5: Memory strength variation of 1-hop neighbors across models.
D.2
Variation Across Models
We observe substantial cross-model variation in recovered memory graphs, even for the same entity. For “Taylor Swift” in RWKU, graphs from LLaMA-2-7b-chat, LLaMA-3-8b-instruct, and GPT-4o exhibit low node overlap and distinct memorization patterns (Figure 5). While all three models share a few core biographical
Knowledge Coverage
Specifically, on RWKU with Llama-2-7b-chat, we vary the coverage threshold used during forgetset construction from the recovered memory graph. For each threshold, we compute the average coverage of the resulting forget set over the memory graph across all target entities, and report the corresponding unlearning results in Figure 6.