arXiv CS open access

TRACE: Data-Free Text Reconstruction Attacks against Approximate Unlearning in LLMs

Mengying Zhang et al. · arxiv_cs

arXiv CS · Papers

cryptography cybersecurity privacy security

As large language models (LLMs) face growing demands for data removal driven by regulatory compliance, copyright concerns, and privacy protection, approximate unlearning has emerged as a practical alternative to expensive full retraining. In practice, pre- and post-unlearning model snapshots are oft

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

TRACE: Data-Free Text Reconstruction Attacks against Approximate Unlearning in LLMs

arXiv:2610.31995v1 [cs.CR] 22 Oct 2026

Mengying Zhang1

Derui Wang2 Ibrahim Khalil1 Bowen Liu4 Xiaoyu Xia1∗ Minhui Xue2,3 1 RMIT University 2 CSIRO 3 Adelaide University 4 The Hong Kong University of Science and Technology

Abstract As large language models (LLMs) face growing demands for data removal driven by regulatory compliance, copyright concerns, and privacy protection, approximate unlearning has emerged as a practical alternative to expensive full retraining. In practice, pre- and post-unlearning model snapshots are often maintained for auditing and rollback, introducing a previously overlooked side channel. This paper demonstrates that approximate unlearning in LLM can leave residual traces within model snapshots, which may be recoverable and thus enable the privacy breaches that approximate unlearning is intended to prevent. Here, we propose TRACE, a framework that reconstructs unlearned training data given access to pre- and postunlearning model snapshots, without any data-specific prior knowledge. To achieve this, TRACE identifies a compact set of candidate tokens from embedding-level parameter differences to constrain the search space, and assembles sequences via a contrastive scoring mechanism based on the perplexity gap between the two models. A two-stage exploration-then-completion strategy enables the recovery of multiple distinct samples. Extensive experiments on various datasets, including TOFU, MUSE, and WMDP, across multiple LLMs and six representative unlearning methods, demonstrate that TRACE achieves high-fidelity reconstruction. This paper exposes significant privacy risks in current LLM approximate unlearning deployments and highlights the need for defenses against parameter-difference leakage and appropriate management of model snapshot access. The code is publicly available at https://github.com/mengyingz969/TRACE.

Introduction

The rapid adoption of large language models (LLMs) has intensified the need for effective data removal mechanisms, driven by regulatory requirements such as the EU’s GDPR [1] and AI Act [2], copyright compliance [3, 4, 5], and privacy [6, 7]. Machine unlearning [8, 9, 10] aims to remove the influence of a specified subset of training data from a trained model. In practice, exact machine unlearning [8], retraining models without the target data, remains computationally prohibitive for modern LLMs. As a result, mainstream real-world deployments rely on approximate unlearning methods [11, 12, 13, 14, 15, 16, 17], which aim to forget specific data through targeted parameter updates without retraining. To support auditing, rollback, and safety verification, it is common to retain multiple model snapshots from before and after unlearning. This raises a critical question: given access to original and unlearned models, to what extent do current unlearning methods leak sensitive information about the data they are intended to remove? Existing work [18] shows that differences between the original and unlearned model parameters can be used to construct surrogate gradient signals, which are then exploited through gradient∗ Corresponding author: Xiaoyu Xia.

40th Conference on Neural Information Processing Systems (NeurIPS 2026).

matching inversion to reconstruct removed images. However, this cannot be extended to generative LLMs, as the reconstruction optimization struggles to converge and often produces uninterpretable or disjointed outputs without the “anchor” provided by labels. More recently, efforts have been made to investigate the extraction of unlearned data from LLMs under the assumption that partial prefixes of the target texts are available, demonstrating that the remaining content can be recovered by exploiting discrepancies in output likelihoods between pre- and post-unlearning models [19]. However, this reliance on prefix priors leaves an important open question: is unlearned data in LLMs secure without any data-specific prior knowledge? We answer this question by introducing TRACE, a framework for reconstructing text that has been intentionally unlearned by tracing latent parameter footprints between pre- and post-unlearning snapshots, without requiring any data-specific prior knowledge. First, we identify that parameter differences in the embedding layer between pre- and post-unlearning models expose the tokens involved in the unlearning process, effectively reducing the reconstruction task from open-ended generation over the full vocabulary to assembling sequences from a compact set of candidate tokens. Second, to resolve the combinatorial complexity of sequence reassembly, we develop a contrastive search strategy driven by a novel contrastive fluency score (CFS). This scoring mechanism integrates unlearning-induced divergence with linguistic plausibility, effectively navigating the search toward trajectories corresponding to unlearned sequences. Third, to enable reconstruction of multiple unlearned samples, we design a two-stage “exploration-then-completion” architecture. Stage I performs broad exploration to capture diverse prefix candidates, while Stage II completes them into full sequences. Our contributions are summarized as follows: • We reveal that approximate unlearning leaves a latent footprint in parameters, enabling reconstruction of text that has been intentionally unlearned, without any data-specific prior knowledge. • We propose a framework named TRACE that achieves the reconstruction of multiple distinct unlearned samples without any data-specific prior knowledge, by utilizing parameter differences to effectively constrain the vast search space and employing a contrastive search strategy within a two-stage architecture. • We conduct comprehensive experiments on various benchmarks including TOFU, MUSE, and WMDP, across multiple LLMs and six representative unlearning methods, showing that TRACE consistently achieves high-fidelity reconstruction.

Related Work

Machine Unlearning in LLMs. Machine unlearning [8] aims to remove the influence of specific training samples from a trained model. Exact unlearning, which retrains the model on the remaining data, is widely regarded as the gold standard for data removal. However, exact unlearning is impractical for large language models (LLMs) due to their massive training scale and prohibitive computational cost. As a result, most existing work focuses on approximate unlearning via efficient parameter updates. Gradient ascent (GA) [11, 12, 13, 14] forgets samples by reversing learning on target data, i.e., applying updates that increase the loss on the data to be removed. Gradient difference (GD) [15] balances forgetting and utility by maximizing loss on forget data while minimizing loss on retain data. KL-regularized methods (KL) [16] introduce Kullback–Leibler divergence regularization on a retained set to preserve the model’s behavior on retained data during unlearning. Negative Preference Optimization (NPO) [17] reformulates unlearning as preference optimization with a reference model to stabilize training. Suppression-intended unlearning [20, 21, 22, 23, 24, 25] suppresses sensitive outputs rather than removing knowledge and is beyond our scope. Training Data Reconstruction. Recent studies show that training data can be reconstructed from models. Gradient-based approaches recover samples from leaked gradients [26, 27, 28, 29]. To name a few, LAMP [26] reconstructs text using a language model prior and alternating optimization, while Gupta et al. [27] use beam search with gradient-guided reordering. Petrov et al. [28] exploit low-rank self-attention gradients for exact inversion, and Xie et al. [29] demonstrate reconstruction under parameter-efficient fine-tuning. Beyond gradients, reconstruction from internal representations has also been explored, such as embedding leakage [30] and activation inversion attacks [31]. However, these attacks rely on access to gradients or intermediate activations during training, and thus cannot be directly applied when only pre- and post-unlearning model snapshots are available. 2

Unlearning Inversion Attacks. Unlearning inversion attacks were first introduced in the context of image classification to reconstruct unlearned images by approximating the parameter differences between pre- and post-unlearning models as gradients of unlearned images [18]. Du et al. [32] extend this idea to textual classification, enabling the reconstruction of unlearned text. Ma et al. [33] propose an RL-based approach that optimizes latent noise to reconstruct unlearned images. For LLMs, Wu et al. [19] extract unlearned text in LLMs by leveraging memorization and probability gaps between original and unlearned models, assuming the adversary knows the identifiable prefix of the target unlearned text. However, optimizing-based methods [18, 32, 33] rely on labels, which are unavailable in generative LLMs, and prefix-based methods rely on the strong assumption of access to sample-specific context. Currently, no unlearning inversion approaches are available without prior information about unlearned data.

Threat Model

Scenario. Following prior unlearning inversion attacks [18, 32, 33], we consider practical model update workflows where both pre- and post-unlearning model versions are accessible. In Machine Learning as a Service (MLaaS) settings [18], historical model versions are often retained for rollback, auditing, debugging, or staged deployment, introducing a side channel, as discrepancies induced by unlearning may encode information about removed training data. A broader discussion of practical deployment scenarios is provided in Appendix B.1. Adversary. The adversary observes both pre- and post-unlearning models with white-box access to parameters, but has no access to the identities or content of the removed samples, the precise gradients or intermediate activations of the removed samples during training. The goal is to reconstruct samples removed through unlearning.

Method

4.1

TRACE Overview Rank by CFS

Unlearned Data 1. John Smith lives in a small town near the river

... Discard

2. John Smith studied at local school near hill top area

Post

Completion

...

Pre

Reconstructed Data

... ...

Unlearn

...

Start Points

broad

Search-Space Initialization

Search Converged

Stage I : Broad Prefix Exploration

！

2 1

Top

...

... Token Set

Retain Top-K

... ...

narrow

Stage II: Completion from Recorded Prefixes

Figure 1: Framework of TRACE. TRACE reconstructs the unlearned data by incrementally assembling sequences from a compact set of candidate tokens, which are derived from embedding differences between the pre- and postunlearning models via the search-space initialization (Sec. 4.2). Reconstruction follows an iterative search process, where candidate sequences are iteratively extended, scored by a novel contrastive fluency score (CFS)(Sec 4.3), and pruned to retain the highest-scoring candidates. Building on the CFS-guided search, TRACE further structures the reconstruction process into two stages to facilitate the reconstruction of distinct unlearned samples (Sec 4.4): (1) Stage I - broad prefix exploration, which maintains a large set of candidate sequences at each step; (2) Stage II - completion from recorded prefixes, which completes the recorded prefixes from Stage I into full sequences with a relatively small candidate set at each step. The framework of TRACE is shown in Fig. 1. 3

4.2

Search-Space Initialization

Search-Space Restriction via Candidate Tokens. The large vocabulary of LLMs leads to a combinatorial explosion in possible sequences. Inspired by prior findings that non-zero embedding gradients reveal tokens in training data [27, 34], we identify an analogous effect in unlearning: tokens involved in the unlearning process exhibit non-zero embedding parameter differences between the pre-unlearning model θ∗ and post-unlearning model θu . We exploit this property to construct a compact candidate token set to constrain the search space. Let E ∈ R|V|×h denote the input embedding matrix, where |V| is the vocabulary size and h is the embedding dimension. For each token v ∈ V, we compute the ℓ2 norm of the embedding difference ∆(v) and define the candidate token set Vf by: Vf = {v ∈ V | ∆(v) > 0},

∆(v) = ∥Eθ∗ (v) − Eθu (v)∥2 .

(1)

Initialization of Starting Points. We construct a set of valid initial tokens V0 by filtering the candidate token set Vf based on linguistic constraints. Specifically, we retain those that are consistent with common sentence-initial patterns, such as tokens starting with uppercase letters or valid opening punctuation (e.g., ", (, [, {, or <). 4.3

Unified CFS-Guided Search

TRACE follows an iterative search process, where candidate sequences are iteratively extended, scored using a novel contrastive fluency score (CFS), and pruned to retain the highest-scoring candidates for next step. Let X (0) denote the set of starting points, and X (r−1) denote the retained set from step r − 1. We construct the extended candidate set at step r by: C (r) = { x ⊕ v | x ∈ X (r−1) , v ∈ Vf },

(2)

where x ⊕ v denotes the sequence obtained by extending x with token v. For an extended candidate sequence x ∈ C (r) , the perplexity [35] under a model θ is given by:   |x| X 1 log pθ (xi | x<i ) . (3) PPLθ (x) = exp− |x| i=1 Lower perplexity indicates that the sequence x has higher likelihood. Unlearning reduces the likelihood of target data in the post-unlearning model, resulting in a divergence in perplexity between the pre- and post-unlearning models. We leverage this to design a contrastive fluency score (CFS) to guide the search. Specifically, for a candidate sequence x ∈ C (r) , the contrastive score is defined as: CFS(x) =

PPLθu (x) − PPLθ∗ (x) , PPLθ∗ (x)A(|x|)

A(|x|) = α + β log |x|.

(4)

PPL

PPL *

The length-aware exponent A(|x|) controls the 103 relative importance of the fluency constraint 104 PPLθ∗ , where α and β are positive constants. 103 2 10 This design is motivated by the fact that perplex102 101 ity varies significantly with sequence length. As 101 shown in Figure 2a, perplexity under the origi0 10 0 100 0 nal model tends to be higher for shorter prefixes 20 40 20 40 Prefix length Prefix length of unlearned data and decreases and stabilizes as (a) (b) the sequence grows. By progressively increasing the influence of PPLθ∗ (x), CFS remains Figure 2: Effect of prefix length on perplexity tolerant of high-perplexity prefixes during early for samples unlearned using GA. (a) Perplexity exploration when sufficiently strong contrastive of unlearned samples under the pre-unlearning signals (i.e., a large perplexity gap ∆PPL) are model. (b) Perplexity gap between pre- and postpresent. This prevents premature pruning of unlearning models. promising sequences, while enforcing stricter fluency constraints in later steps. At each step, candidates in C (r) are ranked according to CFS(x), and a subset is retained to form X (r) , which serves as the input for the next iteration. The specific retention strategy is detailed in Sec. 4.4.

4.4

Two-Stage Reconstruction via CFS-Guided Search

While CFS defines a scoring principle for candidate sequences, reconstructing multiple distinct unlearned samples remains challenging, as sequential search tends to concentrate on a number of highly similar high-scoring sequences. To address this issue, TRACE proposes a two-stage architecture that separates diversity exploration from sequence completion. The procedure is summarized in Algorithm 1 in Appendix. Stage I: Broad Prefix Exploration. In this stage, TRACE aims to explore diverse candidate prefixes that may correspond to different unlearned samples. To this end, TRACE maintains a relatively large retained set at each step. As the unlearning signal ∆PPL exhibits strong sample-to-sample variability, particularly for short prefixes (Fig. 2b), TRACE ranks candidates in C (r) under two parameterizations of CFS: CFS(·; αs , β) and CFS(·; αl , β), where αs and αl correspond to weaker and stronger fluency constraints, respectively, with αs < αl . The former is more permissive to high-perplexity prefixes in early steps and thus preserves candidates with strong unlearning signals, while the latter favors more fluent candidates, even if their unlearning signals are relatively weak. Accordingly, the retained set at step r is given by: X (r) = Topk C (r) ; CFS(·; αs , β) ∪ Topk C (r) ; CFS(·; αl , β) . (5) To quantify diversity, sequences in X (r) are grouped into clusters based on ROUGE-L similarity with a threshold τ . Formally, X (r) is partitioned into clusters P (r) such that ∀P ∈ P (r) , ∀x, x′ ∈ P, ROUGE-Lrecall (x, x′ ) ≥ τ. (6) (r) This stage continues until the retained set exhibits limited diversity, i.e., |P | ≤ N , where N serves as an approximate indication of how many distinct samples the attacker aims to reconstruct. The last ∗ retained set is recorded as X (r ) , serving as candidate prefixes, which are then completed in Stage II. We use a broad search width k = N × d in stage I, where d is an expansion factor, to increase the chance of retaining prefixes corresponding to true unlearned samples. Stage II: Completion from Recorded Prefixes. This stage revisits the last recorded prefix set ∗ X (r ) from Stage I and completes it into full sequences via a narrower CFS-guided search, where the strategy of extending and pruning candidates at each step follows Eq. (2) and Eq. (5) but with a smaller search width k = Q. To avoid redundant reconstruction, let S denote the set of reconstructed sequences. The set of uncovered prefixes U is: ∗ U = x ∈ X (r ) max ROUGE-Lrecall (x, s) < τ . (7) s∈S

TRACE performs iterative CFS guided search over U. In each iteration, the search continues until a termination condition is met, yielding newly reconstructed sequences and triggering corresponding ∗ updates to S and U. This process repeats until U = ∅, i.e., all prefixes in X (r ) are covered. Detailed stopping criteria are provided in Appendix B.3.

Experiments

5.1

Setup

We evaluate TRACE on various standard unlearning benchmarks, including TOFU [36], MUSE [37], and WMDP [38]. For each dataset, we first fine-tune the model to obtain a pre-unlearning checkpoint, and then apply six representative unlearning methods, including gradient ascent (GA), gradient difference (GD), KL minimization (KL), negative preference optimization (NPO), NPO-Diff [36], and NPO-KL [36], to unlearn the target samples and obtain the corresponding unlearned models. Experiments are conducted on multiple open-source LLMs, including LLaMA2-13B, LLaMA27B [39],Mistral-7B [40], Qwen2.5-3B [41], StableLM-3B-4E1T [42] and TinyLlama-1.1B [43]. Follow [19], three metrics are adopted for evaluation, including ROUGE-L recall (R), and average extraction success rate (A-ESRκ ), which measures the percentage of ground-truth unlearned samples that can be matched by reconstructed samples under the ROUGE-L recall threshold κ ∈ {1.0, 0.9} (as formally defined in Appendix D.2). We set N = 1.5 × B in our experiments, where B is the unlearning batch size. TRACE also involves the CFS coefficients (αs , αl , β), similarity threshold τ , and search parameters d and Q. We analyze their sensitivity in Sec. 5.4; complete experimental settings and detailed explanations of their roles are provided in Appendices D.3 and B.2, respectively. 5

TOFU

Metric ROUGE-L(R) TinyLlama-1.1B A-ESR1.0 (%) A-ESR0.9 (%) ROUGE-L(R) Qwen2.5-3B A-ESR1.0 (%) A-ESR0.9 (%) ROUGE-L(R) LLaMA2-7B A-ESR1.0 (%) A-ESR0.9 (%) ROUGE-L(R) Mistral-7B A-ESR1.0 (%) A-ESR0.9 (%) ROUGE-L(R) LLaMA2-13B A-ESR1.0 (%) A-ESR0.9 (%)

GA GD Ours PPL-only Ours PPL-only 0.9084 0.4383 0.8770 0.1770 78.75 10.00 71.25 2.50 86.25 15.00 82.50 2.50 0.8409 0.1681 0.8191 0.1387 67.50 0.00 68.75 0.00 76.25 1.25 75.00 1.25 0.9799 0.6017 0.9695 0.2946 95.00 28.75 92.50 7.50 97.50 36.25 96.25 10.00 0.9594 0.7145 0.9873 0.5584 90.00 47.50 92.50 37.50 95.00 51.25 98.75 37.50 0.9936 0.6106 0.9792 0.3311 93.75 33.75 92.50 10.00 98.75 40.00 97.50 10.00

KL NPO Ours PPL-only Ours PPL-only 0.8785 0.1770 0.8882 0.4383 76.25 2.50 76.25 10.00 82.50 2.50 82.50 15.00 0.8362 0.1386 0.8189 0.1592 66.25 0.00 65.00 0.00 76.25 1.25 75.00 1.25 0.9592 0.2946 0.9715 0.6017 90.00 7.50 93.75 28.75 91.25 10.00 95.00 36.25 0.8707 0.5584 0.9602 0.7145 61.25 37.50 90.00 47.50 67.50 37.50 95.00 51.25 0.9792 0.3311 0.9795 0.6106 93.75 10.00 96.25 33.75 97.50 10.00 96.25 40.00

NPO-Diff Ours PPL-only 0.8358 0.1770 68.75 2.50 76.25 2.50 0.8259 0.1534 67.50 0.00 76.25 1.25 0.9578 0.2946 93.75 7.50 95.00 10.00 0.9604 0.5584 88.75 37.50 93.75 37.50 0.9703 0.3311 93.75 10.00 95.00 10.00

NPO-KL Ours PPL-only 0.8472 0.1770 67.50 2.50 75.00 2.50 0.8625 0.1586 71.25 0.00 78.75 1.25 0.9684 0.2946 96.25 7.50 96.25 10.00 0.9299 0.5584 82.50 37.50 86.25 37.50 0.9690 0.3311 93.75 10.00 95.00 10.00

MUSE

Table 1: Reconstruction performance on TOFU and MUSE.

ROUGE-L(R) TinyLlama-1.1B A-ESR1.0 (%) A-ESR0.9 (%) ROUGE-L(R) Stablelm-3B A-ESR1.0 (%) A-ESR0.9 (%) ROUGE-L(R) LLaMA2-7B A-ESR1.0 (%) A-ESR0.9 (%) ROUGE-L(R) Mistral-7B A-ESR1.0 (%) A-ESR0.9 (%) ROUGE-L(R) LLaMA2-13B A-ESR1.0 (%) A-ESR0.9 (%)

0.8750 80.00 83.75 0.8633 80.00 85.00 0.9457 93.75 93.75 0.9549 93.75 93.75 0.9327 92.50 92.50

0.7641 67.50 70.00 0.7883 72.50 78.75 0.9398 93.75 93.75 0.9760 96.25 96.25 0.9628 96.25 96.25

0.8136 71.25 75.00 0.7958 70.00 76.25 0.9554 93.75 93.75 0.9540 90.00 93.75 0.9399 92.50 92.50

0.8099 76.25 77.50 0.7728 68.75 75.00 0.9646 96.25 96.25 0.9604 93.75 93.75 0.9513 95.00 95.00

Dataset

5.2

Model

0.2834 17.50 20.00 0.5624 46.25 51.25 0.3915 37.50 37.50 0.3533 31.25 32.50 0.3839 35.00 36.25

0.7555 65.00 67.50 0.7448 62.50 71.25 0.9332 92.50 92.50 0.8753 83.75 85.00 0.9438 92.50 92.50

0.1153 3.75 5.00 0.4246 32.50 36.25 0.1654 13.75 13.75 0.1221 6.25 7.50 0.1728 12.50 13.75

0.8467 76.25 78.75 0.8838 83.75 87.50 0.9579 95.00 95.00 0.9637 94.44 94.44 0.9441 92.50 92.50

0.2834 17.50 20.00 0.5624 46.25 51.25 0.3915 37.50 37.50 0.3533 31.25 32.50 0.3839 35.00 36.25

0.1153 3.75 5.00 0.4246 32.50 36.25 0.1654 13.75 13.75 0.1221 6.25 7.50 0.1728 12.50 13.75

Multi-sample reconstruction without prior knowledge of unlearned data

We evaluate TRACE against a PPL-only baseline that adopts the same framework as TRACE but ranks candidates solely by perplexity under the pre-unlearning model. This baseline serves as an ablation to isolate the contribution of the CFS mechanism from the original model’s inherent memorization. In Appendix D.4, we further compare PPL-only with FILM [27], and show that PPL-only is a stronger reconstruction baseline. Table 1 summarizes the results averaged over 10 randomly sampled batches with an unlearning size of B = 8. TRACE consistently achieves superior performance across various models and unlearning methods. While the efficacy of PPL-only deteriorates significantly under unlearning methods that incorporate retain-set regularization (e.g., GD, KL, NPO-Diff/KL), TRACE remains remarkably robust across all such settings. Furthermore, we observe that larger models are more susceptible to reconstruction than smaller ones due to their superior capacity for memorization. For instance, TRACE achieves ROUGE-L(R) above 0.93 on LLaMA2-7B, Mistral-7B, and LLaMA2-13B. We provide a visualization of reconstructed data in Appendix E.1.

TOFU

Table 2 further evaluates differTable 2: TRACE Performance under different unlearning batch ent unlearning batch sizes B ∈ sizes using LLaMA2-7B. {8, 16, 32, 128}. TRACE remains GA NPO effective even at B = 128, achieving ROUGE-L(R) scores above 0.8 Dataset Metric B = 8 B = 16 B = 32 B = 128 B = 8 B = 16 B = 32 B = 128 ROUGE-L(R) 0.9799 0.9827 0.9541 0.7924 0.9715 0.9569 0.9287 0.8146 on MUSE. Although performance A-ESR (%) 95.00 92.50 88.75 71.09 93.75 87.50 87.50 73.44 A-ESR (%) 97.50 97.50 92.50 74.22 95.00 92.50 90.62 75.78 decreases as batch size increases, ROUGE-L(R) 0.9457 0.9408 0.8996 0.8007 0.9579 0.9561 0.8897 0.8051 this degradation can be mitigated A-ESR (%) 93.75 92.50 87.81 74.22 95.00 93.75 86.56 74.22 by increasing expansion factor d, A-ESR (%) 93.75 92.50 88.12 77.34 95.00 94.38 87.18 81.25 which allows more candidate sequences to be retained per step and improves the chance of capturing true prefixes corresponding to unlearned samples (see Section 5.4). 1.0

MUSE

0.9

1.0 0.9

5.3

Comparison with Prefix-Assisted Baseline UBNF [19]

To enable a fair comparison with the prefix-conditioned unlearned text reconstruction setting in UBNF [19], we conduct reconstruction on the long-form WMDP dataset, where the goal is to 6

reconstruct a long sensitive text. Each WMDP sample is a 512-token text, and UBNF [19] is given the first 10 words as a prefix, while TRACE uses none. As shown in Fig. 3, TRACE consistently outperforms both baselines across models and six unlearning methods without the strong assumptions in the known prefix. In particular, on LLaMA2-7B, TRACE improves ROUGE-L(R) by more than 100% over UBNF. These results demonstrate the effectiveness of TRACE in realistic long-form reconstruction scenarios. We provide a visualization of reconstructed text in Appendix E.2. #1

0.9 0.6 0.3

#6 #5

0.9 0.6 0.3

#2 #6 #3 #5

StableLm-3B

#1 #2 #6

0.9 0.6 0.3

#3 #5 #4

#2 #3

LLaMA2-7B

LLaMA2-13B

Ours UBNF PPL-only #1. NPO #2. NPO-Diff #3. NPO-KL #4. GA #5. GD #6. KL

Figure 3: Long-form reconstruction performance on WMDP measured by ROUGE-L(R).

5.4

A-ESR0.9 (%)

ROUGE-L(R)

A-ESR1.0 (%)

Study Case: Prefix-Matched Recon60 0.8 50 50 struction on RWKU Dataset [44]. 0.7 40 40 0.6 30 We further evaluate prefix-assisted re30 0.5 20 Ours Ours Ours 20 construction on the RWKU dataset, PPL-only PPL-only PPL-only 0.4 10 10 UBNF UBNF UBNF where unlearned samples have an av0.3 0 0 2 4 6 8 10 2 4 6 8 10 2 4 6 8 10 erage length of 150 words. We comPrefix length Prefix length Prefix length pare three methods given equal-length prefixes (2–10 words) from the un- Figure 4: Reconstruction performance with prefix on RWKU learned data, using GA with an un- using LLaMA2-7B learning batch size of 16. As shown in Fig. 4, TRACE consistently outperforms UBNF, demonstrating its effectiveness under limited prefix information in recovering the remaining content. Ablation Study

A-ESR0.9 (%)

A-ESR1.0 (%) A-ESR1.0 (%)

Parameter d in stage I. The parameter d determines how many candidates are retained at each step in stage I and therefore directly affects the ability to capture diverse prefixes corresponding to different unlearned samples. As observed in Table 2, increasing the unlearning batch size B to 32 (with a fixed d = 5) leads to performance degradation on the MUSE dataset. Table 3 reports that increasing d improves reconstruc-

A-ESR0.9 (%)

ROUGE-L(R) ROUGE-L(R)

Hyperparameters αs , αl and β in 80 80 0.85 CFS. Figure 5 shows the sensitivity 70 0.80 70 60 0.75 of CFS to αs , αl and β on MUSE us=0 =0.75 =0 =0.75 =0 =0.75 60 0.70 50 =0.25 =1.0 =0.25 =1.0 =0.25 =1.0 ing TinyLlama under NPO. Introduc=0.5 =0.5 =0.5 50 0.65 40 ing the length-aware term β > 0 im1 2 3 4 1 2 3 4 1 2 3 4 αs αs αs proves the robustness of CFS, prevent0.88 80 85 ing performance collapse at small αs 0.84 75 80 , with the best results around αs = 2. 0.80 70 75 This suggests a beneficial dynamic 0.76 trade-off: early steps prioritize strong 65 70 0.72 unlearning signals even if prefixes are 2 4 6 8 10 2 4 6 8 10 2 4 6 8 10 αl αl αl not fluent, while later steps increasingly favor linguistic plausibility as seFigure 5: Hyperparameters in CFS on MUSE. Top: varying quences become more complete. Reα and β (αl = 6.0). Bottom: varying αl (αs = 2, β = 0.5). garding αl , setting αl = αs = 2 de- s grades performance as the dual-ranking reverts to a single objective. αl ∈ [3, 8] leads to stronger performance, suggesting that this branch serves as a complement to the small αs setting by capturing fluent prefixes with weaker unlearning signals that may otherwise be overlooked.

Table 3: Effect of d on MUSE. Metric

d=5

d = 10 d = 15

ROUGE-L(R) 0.8897 0.9504 A-ESR1.0 (%) 86.56 92.50 A-ESR0.9 (%) 87.19 93.13

0.9564 92.81 93.75

tion performance across metrics, indicating that TRACE can scale to larger batch unlearning settings by increasing d. Parameter Q in stage II. The parameter Q controls the Table 4: Effect of Q on MUSE. search width in Stage II. As the search focuses on extending Metric Q = 1 Q = 5 Q = 10 selected prefixes rather than exploring diverse trajectories, a ROUGE-L(R) 0.8148 0.8529 0.8467 narrow search width is sufficient. As shown in Table 4, a relA-ESR1.0 (%) 68.75 76.25 76.25 atively small value (e.g., Q = 5) achieves strong performance. A-ESR0.9 (%) 76.25 78.75 77.50 These results highlight the effectiveness of decoupling diversity discovery from sequence completion, enabling Stage II to achieve reliable reconstruction with a modest search budget while avoiding redundant exploration.

A-ESR 0:9 (%)

A-ESR 1:0 (%)

ROUGE-L(R)

Effect of Similarity Threshold τ . 100 100 1.0 We study the impact of the similarity 80 80 0.8 60 60 0.6 threshold τ used in Eq. (6) and Eq. (7), 40 40 0.4 TOFU TOFU TOFU using LLaMA2-7B under NPO with 20 20 0.2 MUSE MUSE MUSE unlearning batch size B = 128. As 0.0 0 0 0.4 0.5 0.6 0.7 0.4 0.5 0.6 0.7 0.4 0.5 0.6 0.7 shown in Fig. 6, Reconstruction per¿ ¿ ¿ formance on TOFU demonstrates a Figure 6: Effect of Similarity Threshold τ higher sensitivity to the choice of τ compared to that on MUSE. We attribute this to its structural properties: TOFU consists of question–answer pairs, where samples tend to share higher lexical and semantic similarity. When τ is too small, structurally similar but semantically different prefixes are merged into the same cluster, leading to inadequate reconstruction of distinct samples. Empirically, setting τ ∈ [0.6, 0.7] is found to provide a robust balance between distinguishing distinct sequences and merging redundant sequences.

5.5

Ours PPL-only

A-ESR 0:9 (%)

A-ESR 1:0 (%)

Ours PPL-only

A-ESR 0:9 (%)

ROUGE-L(R)

Ours PPL-only

A-ESR 1:0 (%)

ROUGE-L(R)

Training and Unlearning Epochs. 1.0 100 100 0.8 80 80 Figure 7 presents the impact of train0.6 60 60 ing and unlearning epochs on recon0.4 40 40 struction performance on MUSE us0.2 20 20 0.0 0 0 ing TinyLlama under NPO. We ob2 4 6 8 10 2 4 6 8 10 2 4 6 8 10 serve that varying the number of unTrain epochs Train epochs Train epochs learning epochs has a minimal effect 1.0 100 100 0.8 80 80 on reconstruction performance. In 0.6 60 60 contrast, increasing the number of 0.4 40 40 training epochs consistently enhances 0.2 20 20 reconstruction quality, suggesting that 2 3 4 5 6 2 3 4 5 6 2 3 4 5 6 Unlearn epochs Unlearn epochs Unlearn epochs stronger pre-unlearning memorization makes the target data more recoverable. Notably, our method TRACE Figure 7: Effect of training and unlearning epochs. Top: consistently outperforms the baseline varying training epochs (unlearning epochs = 5). Bottom: varying unlearning epochs (training epochs = 5). across all settings. Ours PPL-only

Ours PPL-only

TRACE in the Black-Box Setting

Our main experiments consider a white-box setting, where TRACE exploits embedding differences between the pre- and post-unlearning models to construct a compact candidate token set Vf . We further investigate whether TRACE can operate under a weaker black-box setting, where model parameters are inaccessible but token-level output probabilities from both model versions are available. In this setting, we therefore set Vf to the entire vocabulary and perform reconstruction. We evaluate this setting on MUSE using TinyLlama under NPO with an unlearning batch size of 8, considering pre-unlearning models trained for 4, 5, and 6 epochs. As shown in Table 5, TRACE retains meaningful reconstruction capability in the black-box setting, although its performance is lower than in the white-box setting. Without parameter access, TRACE must instead search over the entire vocabulary, introducing many competing candidate prefixes and making it more difficult to retain the ground-truth paths during exploration. This challenge would become increasingly pronounced for models with larger vocabularies. In such a large search space, 8

Table 5: TRACE performance in the black-box setting. Training epochs = 4

Training epochs = 5

Training epochs = 6

Dataset

Metric

white-box

black-box

white-box

black-box

white-box

black-box

MUSE

ROUGE-L(R) A-ESR1.0 (%) A-ESR0.9 (%)

0.6566 51.25 55.00

0.3419 27.50 27.50

0.8467 76.25 78.75

0.6043 55.00 55.00

0.9027 83.75 87.50

0.6274 57.50 60.00

stronger memorization of the target samples exposes them to greater reconstruction risk, as evidenced by the substantial improvement in reconstruction performance with increasing pre-unlearning training epochs. 5.6

Robustness to Checkpoint Transformations

In practical deployments, model checkpoints may undergo additional transformations after unlearning. We evaluate TRACE under two representative model compression techniques, including low-bit quantization and weight pruning. Model Quantization. We apply 4-bit AWQ quantization [45] Table 6: TRACE under 4-bit AWQ to the unlearned LLaMA2-7B model obtained with NPO on quantization. MUSE, and evaluate TRACE using the resulting quantized checkpoint. As shown in Table 6, quantization has only a minor Metric Original AWQ 4-bit effect on reconstruction performance: ROUGE-L(R) decreases ROUGE-L(R) 0.9579 0.9506 from 0.9579 to 0.9506, while both A-ESR1.0 and A-ESR0.9 A-ESR1.0 (%) 95.00 95.00 remain at 95%. This result shows that the reconstruction signal A-ESR0.9 (%) 95.00 95.00 exploited by TRACE remains largely preserved after low-bit quantization. Model Pruning. We apply Wanda [46] with different pruning Table 7: TRACE under Wanda ratios to the unlearned LLaMA2-7B model obtained with NPO pruning. on MUSE, and evaluate TRACE using the resulting pruned checkpoints. Table 7 shows that TRACE remains effective Pruning Ratio PPL ROUGE-L(R) under moderate pruning, with ROUGE-L(R) decreasing only 0% 16.07 0.9579 slightly from 0.9579 to 0.9503 at a 10% pruning ratio. More 10% 17.50 0.9503 aggressive pruning progressively reduces reconstruction quality, 30% 13.58 0.8489 with ROUGE-L(R) decreasing to 0.8489 and 0.6394 at pruning 50% 5.07 0.6394 ratios of 30% and 50%, respectively. Interestingly, aggressive pruning also substantially decreases the perplexity of the unlearned samples, from 16.07 at 0% pruning to 5.07 at 50% pruning. Thus, while aggressive pruning weakens TRACE’s reconstruction performance, it simultaneously compromises the effect of unlearning.

Possible Defenses

We evaluate three practical defenses: parameter obfuscation [47], gradient pruning [48], and postunlearning fine-tuning [18]. Parameter Obfuscation. Inspired by DP-SGD [47], we adopt a lightweight parameter obfuscation mechanism during unlearning. At each unlearning step, we first clip gradients to C = 1.2, and z then inject noise proportional to the gradient norm, i.e., g̃ = g + σ∥g∥ ∥z∥ , z ∼ N (0, I), with hyperparameter σ controlling the perturbation strength. We emphasize that this approach is heuristic and does not aim to provide formal differential privacy guarantees. The injected noise perturbs the embedding parameters across the entire vocabulary, which interferes with our candidate token set Vf extraction. To address this, we apply a simple unsupervised refinement. Specifically, we first select the top-p% tokens (p = 20 in our setting) with the largest embedding differences, and then perform 1D clustering (K-means with k = 2) on their log-scaled differences, taking the cluster with larger mean shifts as Vf . Table 8a reports the trade-off between unlearning effectiveness measured by perplexity (higher is better) and reconstruction performance under different noise levels σ on MUSE using TinyLlama under GA. As σ increases, perplexity on the forget set decreases, indicating weaker 9

Table 8: Possible defenses. Rec(Vf ) and Prec(Vf ) denote recall and precision over Vf with respect to ground-truth tokens from the unlearned data. (a) Parameter Obfuscation.

(b) Gradient Pruning.

σ Perplexity ROUGE-L(R) Rec(Vf ) Prec(Vf ) 0 20.17 0.8750 1 1 0.8361 0.9551 1 0.01 19.37 0.05 18.63 0.7226 0.9038 1 0.10 17.81 0.5375 0.8525 1 0.15 17.06 0.4599 0.8846 0.0601 0.4062 0.8525 0.0562 0.20 16.30

p Perplexity ROUGE-L(R) Rec(Vf ) Prec(Vf ) 0 20.17 0.8750 1 1 0.8750 1 1 0.80 25.73 0.85 19.43 0.8750 1 1 0.90 12.81 0.8636 1 1 0.95 5.57 0.8636 1 1 0.8182 1 1 0.99 2.29

unlearning. Meanwhile, the precision of Vf sharply deteriorates when σ ≥ 0.15, so that a large number of noise-induced tokens become indistinguishable from ground-truth tokens. For smaller noise levels (σ ≤ 0.10), TRACE remains effective with high precision and strong recall of Vf . Gradient Pruning. We evaluate gradient pruning [48] on MUSE using TinyLlama under GA with pruning ratios p ∈ {0.8, 0.85, 0.9, 0.95, 0.99}, where a fraction p of gradient components with the smallest magnitudes are pruned (set to zero) during each update. Pruning has little impact on TRACE, with the extraction of Vf remaining unaffected. As shown in Table 8b, even with an aggressive pruning ratio of 0.99, TRACE still achieves an average ROUGE-L (R) of 0.8182.

A-ESR 0:9 (%)

A-ESR 1:0 (%)

ROUGE-L(R)

Fine-tuning. Following the prior 1.0 100 100 work [18], we consider a setting 0.8 80 80 0.6 60 60 Ours Ours Ours where the developer fine-tunes the unPPL-only PPL-only PPL-only 0.4 40 40 learned model before uploading the 0.2 20 20 updated model. Specifically, we fine0.0 0 0 0 100 200 300 400 500 0 100 200 300 400 500 0 100 200 300 400 500 tune the unlearned model (obtained size size size via GA) for two epochs on retain sets Figure 8: Fine-tuning after unlearning. of varying sizes 0, 100, 200, 300, 400, and 500 on the MUSE dataset using LLaMA2-7B. As shown in Fig. 8, fine-tuning partially reduces reconstruction performance as the size of the fine-tuning dataset increases. This process enlarges the token candidate set Vf and expands the search space, making reconstruction more challenging. Nevertheless, TRACE consistently outperforms the PPL-only baseline. Discussion. These results suggest that heuristic defenses provide limited protection against TRACE. Gradient pruning has little effect on reconstruction. Parameter obfuscation can reduce attack effectiveness, but at the cost of weaker unlearning performance; Post-unlearning fine-tuning partially degrades reconstruction by expanding the search space. However, it does not eliminate the perplexity gap induced by unlearning, and TRACE can still be applied, albeit at the cost of increased computational overhead (analyzed in Appendix C), which is often acceptable in offline attack settings. We further investigate whether exact retraining can provide stronger resistance to TRACE in Appendix D.5. The results show that exact retraining substantially reduces the attack effectiveness compared with approximate unlearning, although a small fraction of the unlearned samples can still be exactly reconstructed by TRACE. Overall, these findings underscore the need for unlearning methods and deployment protocols that explicitly account for the attack surface introduced by approximate unlearning.

Conclusion

In this work, we reveal that approximate unlearning in LLMs leaves recoverable footprints in model snapshots, which are commonly retained for auditing or rollback, facilitating the privacy breaches it is intended to prevent even without requiring any data-specific prior knowledge. To demonstrate this, we propose TRACE, a framework that reconstructs unlearned text by utilizing parameter differences to effectively constrain the vast search space and employing a contrastive search strategy within a two-stage architecture. Extensive experiments show that TRACE consistently achieves high-fidelity reconstruction across diverse models, datasets, and unlearning methods. This highlights the need for unlearning methods and deployment protocols that explicitly account for parameter-difference leakage, and call for future work on defenses that mitigate cross-snapshot information exposure. 10

References [1] Gdpr article 17: Right to erasure. gdpr-info.eu, 2016. URL https://gdpr-info.eu/ art-17-gdpr/. [2] Regulation (eu) 2024/1689: Artificial intelligence act. Official Journal of the European Union, 2024. URL https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng. [3] Denghui Zhang, Zhaozhuo Xu, and Weijie Zhao. Llms and copyright risks: Benchmarks and mitigation approaches. In Proceedings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 5: Tutorial Abstracts), pages 44–50, 2025. [4] Kalin Hristov. Artificial intelligence and the copyright dilemma. Idea, 57:431, 2016. [5] Felix B Mueller, Rebekka Görge, Anna K Bernzen, Janna C Pirk, and Maximilian Poretschkin. Llms and memorization: On quality and specificity of copyright compliance. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 984–996, 2024. [6] Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, and Xiuzhen Cheng. On protecting the data privacy of large language models (llms) and llm agents: A literature review. High-Confidence Computing, 5(2):100300, 2025. [7] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650, 2021. [8] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE symposium on security and privacy (SP), pages 141–159. IEEE, 2021. [9] Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. ACM Transactions on Intelligent Systems and Technology, 16(5):1–46, 2025. [10] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, 7(2):181–194, 2025. [11] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408, 2023. [12] XiaoHua Feng, Chaochao Chen, Yuyuan Li, and Zibin Lin. Fine-grained pluggable gradient ascent for knowledge unlearning in language models. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 10141–10155, 2024. [13] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022. [14] Zirui Pang, Hao Zheng, Zhijie Deng, Ling Li, Zixin Zhong, and Jiaheng Wei. Label smoothing improves gradient ascent in llm unlearning. arXiv preprint arXiv:2510.22376, 2025. [15] Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR, 2022. [16] Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Advances in Neural Information Processing Systems, 37:105425–105475, 2024. [17] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868, 2024. 11

[18] Hongsheng Hu, Shuo Wang, Tian Dong, and Minhui Xue. Learn what you want to unlearn: Unlearning inversion attacks against machine unlearning. In 2024 IEEE Symposium on Security and Privacy (SP), pages 3257–3275. IEEE, 2024. [19] Xiaoyu Wu, Yifei Pang, Terrance Liu, and Zhiwei Steven Wu. Unlearned but not forgotten: Data extraction after exact unlearning in llm. arXiv preprint arXiv:2505.24379, 2025. [20] Jie Ren, Yue Xing, Yingqian Cui, Charu C Aggarwal, and Hui Liu. Sok: Machine unlearning for large language models. arXiv preprint arXiv:2506.09227, 2025. [21] Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1817–1829, 2024. [22] Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. arXiv preprint arXiv:2309.11852, 2023. [23] Tianyang Xu, Xiaoze Liu, Feijie Wu, Xiaoqian Wang, and Jing Gao. Suv: Scalable large language model copyright compliance with regularized selective unlearning. arXiv preprint arXiv:2503.22948, 2025. [24] Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana R Kompella, Sijia Liu, and Shiyu Chang. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference. Advances in Neural Information Processing Systems, 37:12581–12611, 2024. [25] James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. arXiv preprint arXiv:2404.11045, 2024. [26] Mislav Balunovic, Dimitar Dimitrov, Nikola Jovanović, and Martin Vechev. Lamp: Extracting text from gradients with language model priors. Advances in Neural Information Processing Systems, 35:7641–7654, 2022. [27] Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recovering private text in federated learning of language models. Advances in neural information processing systems, 35:8130–8143, 2022. [28] Ivo Petrov, Dimitar I Dimitrov, Maximilian Baader, Mark N Müller, and Martin Vechev. Dager: Exact gradient inversion for large language models. Advances in neural information processing systems, 37:87801–87830, 2024. [29] Jin Xie, Ruishi He, Songze Li, Xiaojun Jia, and Shouling Ji. Recit: Reconstructing full private data from gradient in parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2504.20570, 2025. [30] Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 377–390, 2020. [31] Chenxi Dai, Lin Lu, and Pan Zhou. Stealing training data from large language models in decentralized training through activation inversion attack. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14539–14551, 2025. [32] Jiacheng Du, Zhibo Wang, Jie Zhang, Xiaoyi Pang, Jiahui Hu, and Kui Ren. Textual unlearning gives a false sense of unlearning. arXiv preprint arXiv:2406.13348, 2024. [33] Mengyao Ma, Shuofeng Liu, Jason Xue, Surya Nepal, and Guangdong Bai. Retrace: Reinforcement learning-guided reconstruction attacks on machine unlearning. In The Fourteenth International Conference on Learning Representations, 2026. [34] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP), pages 691–706. IEEE, 2019. 12

[35] Lalit R Bahl, Frederick Jelinek, and Robert L Mercer. A maximum likelihood approach to continuous speech recognition. IEEE transactions on pattern analysis and machine intelligence, (2):179–190, 1983. [36] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121, 2024. [37] Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460, 2024. [38] Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024. [39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. [40] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825. [41] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [42] Jonathan Tow, Marco Bellagente, Dakota Mahan, and Carlos Riquelme. Stablelm 3b 4e1t. URL [https://huggingface. co/stabilityai/stablelm-3b-4e1t](https://huggingface. co/stabilityai/stablelm-3b-4e1t), 2023. [43] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024. [44] Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, Jun Zhao, et al. Rwku: Benchmarking real-world knowledge unlearning for large language models. Advances in Neural Information Processing Systems, 37:98213–98263, 2024. [45] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100, 2024. [46] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In International Conference on Learning Representations, volume 2024, pages 4942–4964, 2024. [47] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016. [48] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019. [49] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. Pmlr, 2017. [50] Cong Xie, Sanmi Koyejo, and Indranil Gupta. Asynchronous federated optimization. arXiv preprint arXiv:1903.03934, 2019.

Societal Impact and Potential Harmful Consequences

This work studies a privacy and security vulnerability in approximate machine unlearning for large language models. Our method, TRACE, demonstrates that when an adversary has access to both preand post-unlearning model snapshots, it may be possible to reconstruct data that was intended to be removed, even without any data-specific prior knowledge. Potential positive societal impacts. This work aims to improve the safety of machine unlearning systems by revealing a previously underappreciated failure mode. We emphasize that it is intended for defensive and scientific use only, and to motivate more robust unlearning methods, improved snapshot management, and stronger privacy auditing practices. In practice, we recommend minimizing snapshot exposure, enforcing strict access control over model checkpoints, and combining unlearning with additional privacy-preserving techniques prior to deployment. Potential negative societal impacts. The primary risk of this research is dual use. An adversary could potentially use the reconstruction procedure to recover sensitive training samples, including personally identifiable information, confidential documents, copyrighted text, or other data that a model owner intended to erase. Such misuse could undermine user privacy, weaken compliance with data deletion requests, and reduce trust in machine unlearning as a privacy-preserving mechanism. Security considerations. We do not propose a mechanism for weaponization or harassment, nor do we study surveillance, discrimination, or any application that targets protected attributes. Our experiments are conducted on standard research benchmarks, and the paper does not involve human subjects or any direct collection of private user data.

Details of TRACE

B.1

Practical Deployment Scenarios

Our threat model assumes access to both pre- and post-unlearning model versions, following prior unlearning inversion attacks [18, 32, 33]. While such access can naturally arise in practical model update workflows where multiple model versions are retained or distributed. We discuss representative deployment scenarios below to further motivate this access assumption. Model Update and Maintenance. In MLaaS[18] and other continuously maintained model services, model updates may produce multiple versions that are retained for operational purposes, such as rollback, auditing, debugging, and staged deployment. Consequently, personnel involved in model maintenance, such as ML engineers, platform operators, or auditors, may have legitimate access to both the model before an unlearning request and its subsequently updated version. Such access enables comparison between the two model versions, potentially exposing information encoded in the parameter changes induced by unlearning. Federated Learning. Federated learning [49], particularly asynchronous federated learning [50], provides a natural setting for dual-model access, where successive global model versions may reflect individual client unlearning updates. Suppose a global model Mt is available to participating clients, and the subsequent version Mt+1 incorporates an unlearning update from a particular client. If no other client updates are incorporated between these two versions, the difference Mt+1 − Mt may predominantly reflect the effect of that unlearning operation. Consequently, the potential adversary is not limited to the aggregation server: any participant with access to both versions could potentially reconstruct the data removed by another client, without accessing that client’s private local data. API-Based Deployment. While TRACE primarily targets the white-box setting, we further explore its applicability in a black-box scenario where model providers expose different model versions through APIs to their customers while keeping their internal parameters private. Users with legitimate API access can query both versions and observe their outputs. When token-level output probabilities are available, these outputs provide the contrastive signal required by TRACE. We evaluate the effectiveness of TRACE in this setting in Sec. 5.5. Despite the increased reconstruction difficulty and 14

computational cost compared with the white-box setting, TRACE can still uncover some unlearned samples. B.2

Detailed Explanation of Hyperparameters in TRACE

TRACE involves several hyperparameters that control different aspects of the reconstruction process. We clarify their roles and selection below. Importantly, these hyperparameters do not require prior knowledge of the unlearned data, including the unlearning batch size. Reconstruction budget N . N serves as an approximate indication of how many Table 9: Effect of N on MUSE using LLaMA2-7B under distinct samples the attacker aims to re- NPO (B = 8). construct. It controls the target level of prefix diversity during Stage I. SpecifiN 8 10 12 14 cally, TRACE groups the retained candiROUGE-L(R) 0.9407 0.9459 0.9579 0.9650 date prefixes according to their similarity, A-ESR1.0 (%) 92.50 93.75 95.00 95.00 and N determines how many distinct preA-ESR0.9 (%) 95.00 95.00 95.00 96.25 fix groups the exploration stage aims to cover. A larger N can subsequently produce more diverse reconstructed sequences and increase reconstruction coverage. N can be specified independently, we set N = 1.5B for controlled evaluation in our experiments. We evaluate TRACE with different values of N on MUSE using LLaMA2-7B under NPO. As shown in Table 9, increasing N generally improves reconstruction coverage. Search parameters d and Q. The parameters d and Q control the search width in Stage I and Stage II, respectively. In Stage I, TRACE uses k = N × d, where d is an expansion factor that determines how broadly TRACE explores candidate prefixes. A larger d retains more prefixes and can improve reconstruction coverage, at the cost of additional computation. In Stage II, Q determines the search width for completing the selected prefixes. Since this stage focuses on sequence completion rather than diversity exploration, a relatively small Q is generally sufficient. CFS parameters αs , αl , and β. The parameters αs , αl , and β determine how CFS balances the memorization signal of the original model and the memorization difference induced by unlearning. The ranges considered for these parameters follow their algorithmic roles rather than any knowledge of the unlearned samples. We summarize their roles and the ranges examined in our experiments in Table 10.

Parameter

Table 10: Roles and evaluated ranges of the CFS hyperparameters. Test ranges Rationale

αs

1–4

αs = 1 is the natural starting point, where the CFS score reduces to P P Lθu (x)−P P Lθ∗ (x) CF S(x) = at the first step. P P Lθ∗ (x)

αl

2–10

αl > αs compensates for the unstable unlearning signals of short prefixes by favoring fluent candidate sequences.

0–1

β = 0 ignores sequence length, while larger β progressively increases the influence of sequence length in CFS.

Similarity threshold τ . The threshold τ determines when candidate sequences are considered similar during prefix clustering and redundancy removal. A larger τ allows TRACE to distinguish and retain more mutually similar sequences. As shown in Figure 6, using a default value of τ = 0.6 can achieve strong performance, regardless of whether the unlearned samples are highly similar (e.g., TOFU) or largely diverse (e.g., MUSE). B.3

Detailed stopping criteria.

In Stage II, TRACE maintains the same dual ranking mechanism introduced in Stage I, based on CFS(·; αs , β) and CFS(·; αl , β). At each step, the top-ranked sequence under each scoring function 15

is tracked independently until it satisfies one of the stopping conditions defined below, after which it is added to the reconstructed set S. The search terminates once both scoring functions have triggered a stopping condition, which may occur at different steps. Specifically, the top-ranked sequence is considered complete if any of the following holds: End-of-sequence token. The last token is an end-of-sequence token (e.g., EOS). Consecutive PPL Increases. When a sentence ending punctuation mark (e.g., .) is encountered, search terminates if PPL increases consecutively with subsequent sequence extensions. Maximum length constraint. As a fallback, search terminates when the sequence length exceeds a predefined maximum length Lmax .

Computational Overhead

Compared to standard autoregressive generation, TRACE incurs a higher computational cost. The computational cost per step in TRACE primarily consists of model forward passes and candidate ranking. At each step, while standard autoregressive generation involves a single forward pass, TRACE increases the number of forward passes from O(1) to O(N · d). Following these passes, each sequence is expanded over the vocabulary Vf , yielding a large candidate set for ranking. Their scores (e.g., CFS) are computed incrementally by updating the accumulated log-probabilities with the log-probabilities obtained from the same forward passes, requiring no additional model invocations. These candidates are then ranked to retain the top sequences; using partial sorting (e.g., top-k selection), this incurs a cost of O(N · d · |Vf | log(N · d)) per step. In Stage II, the computational overhead is further optimized, reducing the requirement to O(Q) forward passes and a ranking cost of O(Q · |Vf | log Q) per step. TRACE makes this search tractable by restricting the branching factor from the full vocabulary V to a highly compact candidate set Vf . We acknowledge the computational overhead associated with the reconstruction phase; however, within an offline threat model based on static snapshots, prioritizing extraction fidelity over real-time latency represents a reasonable and practical compromise.

Experiments

D.1

Datasets

We evaluate TRACE on various representative benchmarks for LLM unlearning, including TOFU, MUSE, WMDP, and RWKU. TOFU [36]. We use the TOFU (Task Of Fictitious Unlearning) benchmark, which consists of synthetic author profiles paired with question–answer data. The dataset contains 200 fictitious authors, each associated with 20 question–answer pairs, resulting in 4,000 samples in total. We construct the training data directly from this dataset, randomly sampling a subset as the forget set, with the remaining samples serving as the retain set. MUSE [37]. We use the MUSE-News subset from the MUSE (Machine Unlearning Six-way Evaluation for language models) benchmark, which is constructed from real-world news corpora. The training data consists of approximately 6,300 sentences randomly sampled from the dataset. The forget data is randomly sampling from the training data. WMDP [38]. We use the bio-retain-corpus from the WMDP (Weapons of Mass Destruction Proxy) benchmark, which consists of biomedical text designed to represent safe, retainable knowledge. The training data is constructed by randomly sampling 1,000 segments from the corpus, each truncated to a maximum length of 512 tokens. The forget data is randomly sampling from the training data. RWKU [44]. We use approximately 6,200 samples from the RWKU (Real-World Knowledge Unlearning) benchmark, which are derived from real-world knowledge about public entities (e.g., well-known individuals). These samples consist of factual descriptions, such as biographical details and related facts. The forget set consists of text segments associated with a specific entity (e.g., a public figure), and the prefixes are extracted directly from these segments, typically consisting of the 16

Algorithm 1 TRACE Input: pre-unlearning model θ∗ , post-unlearning model θu , expansion factor d, completion search width Q, number of samples to reconstruct N , similarity threshold τ , maximum generation length Lmax Output: reconstructed set S 1: Vf ← E XTRACT T OKENS(θ ∗ , θu ) ▷ defined in Eq. (1) 2: X (0) ← {v ∈ Vf | v satisfies linguistic constraints} ▷ valid initial tokens 3: S ← ∅ ▷ reconstructed sequences 4: Stage I: Broad Prefix Exploration 5: for r = 1, 2, . . . , Lmax do 6: r∗ ← r 7: C (r) = { x ⊕ v | x ∈ X (r−1) , v ∈ Vf } 8: X (r) ← TopN ×d C (r) ; CFS(·; αs , β) ∪ TopN ×d C (r) ; CFS(·; αl , β) 9: P (r) ← C LUSTER(X (r) ; τ ) ▷ defined in Eq. (6) 10: if |P (r) | ≤ N then 11: break 12: end if 13: end for 14: Stage II: Completion from Recorded Prefixes ∗ 15: U ← X (r ) ∗ 16: if r = Lmax then 17: return U 18: end if 19: while U ̸= ∅ do 20: Initialize X ← U 21: stops , stopl ← False 22: while not (stops and stopl ) do 23: C ← {x ⊕ v | x ∈ X , v ∈ Vf } 24: if not stops and IsComplete(Top1 (C; CFS(·; αs , β))) then ▷ see Sec. B.3 25: S ← S ∪ Top1 (C; CFS(·; αs , β)) 26: stops ← True 27: end if 28: if not stopl and IsComplete(Top1 (C; CFS(·; αl , β))) then 29: S ← S ∪ Top1 (C; CFS(·; αl , β)) 30: stopl ← True 31: end if 32: if stops then 33: X ← TopQ (C; CFS(·; αl , β)) 34: else if stopl then 35: X ← TopQ (C; CFS(·; αs , β)) 36: else 37: X ← TopQ (C; CFS(·; αs , β)) ∪ TopQ (C; CFS(·; αl , β)) 38: end if 39: end while 40: U ← {x ∈ U | maxs∈S ROUGE-Lrecall (x, s) < τ } 41: end while 42: return S

entity name followed by contextual words (e.g., “John was”, “John began”, “John has” when using the first two words as the prefix). D.2

Metrics

Follow [19], three metrics are adopted for evaluation, including ROUGE-L recall (R), and average extraction success rate (A-ESRκ ). A-ESRκ measures the percentage of ground-truth samples that can be successfully reconstructed under a similarity threshold κ. Specifically, a ground-truth sample 17

is considered successfully reconstructed if there exists at least one reconstructed sample whose ROUGE-L recall exceeds κ. Formally, given the ground-truth set XGT and reconstructed set S, we define: X 1 1 max ROUGE-L(R)(x, s) ≥ κ × 100%, (8) A-ESRκ (XGT , S) = s∈S |XGT | x∈XGT

where 1[·] is the indicator function. We report A-ESR1.0 (exact match) and A-ESR0.9 (approximate match) by default. D.3

Implementation Details

Experiments are conducted on a server equipped with 8 NVIDIA A100 GPUs. Training and Unlearning. All pre-unlearning models are trained on the corresponding training data for 5 epochs with a learning rate of 1 × 10−5 . For unlearning, we determine the number of unlearning epochs based on the perplexity of a retrained model (i.e., a model trained from scratch without the forget set). Specifically, the unlearning process is performed for 2 to 6 epochs, and we select the checkpoint whose perplexity on both the unlearned and retained data is closest to that of the retrained model. Candidate Tokens Extraction. For most models, the candidate token set Vf can be directly obtained using Eq. (1), as unlearning typically induces sparse embedding changes, and only a small subset of tokens exhibits non-zero differences. However, for models that tie the input embedding matrix to the output projection layer (e.g., Qwen), unlearning causes updates to propagate across the entire vocabulary in the embedding space. This makes the thresholding criterion in Eq. (1) less effective, as Vf would otherwise include nearly the full vocabulary. Despite this, we observe that tokens involved in the unlearning process tend to exhibit significantly larger embedding changes compared to unrelated tokens. Based on this observation, for models with tied embeddings, we construct Vf by selecting the top-m tokens with the largest embedding difference norms ∆(v). In our experiments, we use m = 6000 for Qwen2.5-3B. As shown in Table 1, TRACE achieves high-quality reconstruction on Qwen2.5-3B, even though Vf contains a large number of unrelated tokens. Hyperparameter Settings. We set the CFS parameters (αs , αl , β) = (2.0, 6.0, 0.5) for TOFU, MUSE, and RWKU, and (1.5, 6.0, 0.3) for WMDP. The expansion factor d is set to 10 for Qwen2.53B and StableLM-3B, and 5 for other models on TOFU and MUSE. The completion search width Q is set to 10 for TOFU and MUSE, 20 for RWKU, and 40 for WMDP. We use a similarity threshold τ = 0.7 for TOFU and τ = 0.6 for all other datasets. The maximum generation length is set to Lmax = 150 for TOFU and MUSE, 400 for RWKU, and 512 for WMDP, reflecting dataset-specific sequence length ranges. D.4

Discussion and Comparison with Reconstruction Baselines

Training Data Reconstruction. Existing training data reconstruction attacks are not directly applicable to our setting in general. Gradient-based reconstruction methods [26, 27, 28, 29] assume access to gradients produced during training, while other attacks exploit internal representations, such as embeddings or intermediate activations [30, 31]. Such data-dependent information is unavailable under our threat model, where the attacker has access to the pre- and post-unlearning models but not to gradients or intermediate activations associated with the unlearned samples. Among these methods, FILM [27] provides a relevant comparison, as it reconstructs training text in llms from gradients in federated learning. Importantly, FILM performs approximate reconstruction from the gradients rather than requiring exact gradients. We therefore adapt FILM to our setting by treating the parameter difference between the pre- and post-unlearning models as a surrogate gradient signal for the unlearned data and applying its reconstruction procedure accordingly. For a controlled comparison, we configure FILM to generate the same number of samples as TRACE. We compare TRACE, PPL-only, and FILM on TOFU and MUSE using LLaMA2-7B under GA and GD. As shown in Table 11, TRACE substantially outperforms FILM. Notably, under the same number of generated samples, PPL-only baseline also achieves stronger reconstruction performance and 18

reconstructs more unlearned samples than FILM, partly due to its adoption of TRACE’s two-stage architecture. These results establish PPL-only as a strong reconstruction baseline in our main experiments. Table 11: Reconstruction performance of TRACE, PPL-only, and FILM. GA GD Dataset Metric TRACE PPL-only FILM TRACE PPL-only FILM ROUGE-L(R) 0.9799 0.6017 0.4355 0.9695 0.2946 0.2393 TOFU A-ESR1.0 (%) 95.00 28.75 1.25 92.50 7.50 0.00 A-ESR0.9 (%) 97.50 36.25 5.00 96.25 10.00 0.00 MUSE

ROUGE-L(R) A-ESR1.0 (%) A-ESR0.9 (%)

0.9457 93.75 93.75

0.3915 37.50 37.50

0.2625 10.00 15.00

0.9332 92.50 92.50

0.1654 13.75 13.75

0.1968 7.50 10.00

Unlearning Inversion Attacks. Among prior work on unlearning inversion [18, 32, 19, 33], UBNF [19] introduced unlearned text reconstruction for LLMs by exploiting output likelihood differences between the original and unlearned models. However, UBNF requires a known prefix of the target sample to initiate reconstruction. In contrast, TRACE is the first method to reconstruct unlearned samples from LLMs without target prefixes. To further compare their reconstruction capability under the same prior information, we additionally evaluate TRACE and UBNF with identical target-prefix information. As evaluated in Section 5.3, when both methods are provided with the same ground-truth prefixes, TRACE consistently achieves higher reconstruction performance and recovers a larger fraction of the remaining target content. D.5

Reconstruction under Exact Retraining

Our main experiments focus on approximate unlearning methods, which modify a model to remove the influence of selected training samples. We further investigate whether the reconstruction risk exposed by TRACE persists under exact unlearning through retraining. Specifically, we train the pre-unlearning model on the original MUSE training set and the retrained model on the retained data after removing the target samples, using 4 training epochs for both models. We then apply TRACE to the two models to reconstruct the unlearned samples. For comparison, we report the corresponding white-box and black-box results under the approximate unlearning method NPO with the same unlearning batch size of 8. As shown in Table 12, exact retraining sub- Table 12: TRACE under extract unlearning (Retrain). stantially reduces reconstruction risk compared with approximate unlearning. NevMetric Npo(white) Npo(black) Retrain ertheless, TRACE can still exactly reconROUGE-L(R) 0.6566 0.3419 0.2068 struct a small fraction of the unlearned samA-ESR1.0 51.25 27.50 5.00 ples. The markedly higher reconstruction A-ESR0.9 55.00 27.50 5.00 performance under approximate unlearning suggests that its targeted model updates leave stronger exploitable traces of the unlearned data.

Visualization

E.1

Multi-sample reconstruction without prior knowledge of unlearned data

We provide an example of multi-sample reconstruction without prior knowledge of the target data using LLaMA2-7B on MUSE under NPO, with an unlearning batch size of 32. For each unlearned sample, we present the reconstructed sequence with the highest similarity to the ground truth, as shown in Table 13. Note on Data Usage. To further mitigate privacy concerns, we anonymize all personal names in the visualization by replacing them with placeholders (e.g., “[Name]”).

Table 13: Comparison of reconstructed multiple samples by TRACE and PPL-only baseline on MUSE dataset. Red text indicates lexical overlap with the ground truth. #

Unlearned Data (GT)

TRACE (Ours)

PPL-only (Baseline)

Supermarkets are expected to be allowed to select which items they would cap and only take part in the initiative, modelled on a similar agreement in France, on a voluntary basis.

[Name], 17, pleaded guilty to firstdegree murder in April for his role in the 2 November 2021 killing of [Name], a Spanish teacher.

"These businesses want to be part of a sustainable future, but they can’t be part of a sustainable future on their own.

In a separate survey it found firms have seen an average 53% drop in sales compared with pre-war 2021. It reported that larger companies have suffered more disruption that smaller firms.

It feels like a process which has now passed its best before date.

Scotland is a progressive country, Ms [Name] said, and the country’s people want the government to be seen to be taking action on problems like drug deaths.

[Name] said his son was like "a glue" for lots of different friendship groups

[Name] said his son was like "a glue" for lots of different groups [Name] said his son was like "a. Webber said his son was like "a glue" for lots of different groups.

However, the BBC said it was "very different" to the one in the 2000s.

Being in default renders Kemble Water effectively insolvent but that will not affect the underlying utility business that supplies water and waste services to 1 in 4 people in the UK.

In a statement to the BBC, the Scottish government said that road safety and investing in infrastructure were high priorities.

She thinks it is alarming that Chinese authorities feel comfortable targeting foreign citizens and attempting to dictate their work.

She said: "When I found out that I was pregnant again I was - I didn’t want to be, I was sad about it.

[Name], 17, pleaded guilty to firstdegree murder in April for his role in the 2 November 2021 killing of [Name], a Spanish teacher.

However, the BBC said it was "very different" to the one in the 2000s.

The government argued that anyone identified as a potential victim of modern slavery would be returned home or to another "safe country away from those who have trafficked them".

Scotland is a progressive country, Ms [Name] said, and the country’s people want the government to be seen to be taking action on problems like drug deaths.

"With owners, I think people need to start realising that they have responsibilities.

In a separate survey it found firms have seen an average 53% drop in sales compared with pre-war 2021. It reported that larger companies have suffered more disruption that smaller firms.

After her death at the age of 56, here’s a closer look at how she recorded her most famous song.

Ms [Name] had insisted at the time that she was "not concerned" about the party’s finances. Continued on next page

Unlearned Data (GT)

TRACE (Ours)

PPL-only (Baseline)

With more bad weather forecast for Sunday, England are left hoping for enough time to force the win that would level the series at 2-2 and turn the fifth Test at The Oval into a decider.

In a statement to the BBC, the Scottish government said that road safety and investing in infrastructure were high priorities.

In the BBC programme Surviving Aberfan, survivor [Name] recounted how his life was saved by Mr [Name].

In the BBC programme Surviving Aberfan, survivor[Name] recounted how his life was saved by Mr [Name].

However, the BBC said it was "very different" to the one in the 2000s.

Ms [Name] had insisted at the time that she was "not concerned" about the party’s finances.

[Name] have signed France striker [Name] from RB Leipzig for £52m.

"[Name] was a caring daughter, wife, sister, mother and friend. If love and compassion could make a person immortal, she would have lived forever.

The jury heard that if her son had not been found when he was, it was estimated he would have been in the water for about 15 or 16 hours.

The symptoms depend on the individual, which drug they were taking, and for how long. Many patients can stop taking antidepressants without experiencing any problems.

He hailed the late star as a "great black female" and a "positive image for any woman", having overcome "hard times" in her personal life.

Scotland is a progressive country, Ms [Name] said, and the country’s people want the government to be seen to be taking action on problems like drug deaths.

Based on the information she provided the British Pregnancy Advisory Service (BPAS), she was sent the tablets because it was estimated she was seven weeks pregnant.

In 2019, Ms [Name] had insisted she was "not concerned" about the party’s finances, and said it was "in the black" at the end of 2019.

Ms [Name] had insisted at the time that she was "not concerned" about the party’s finances.

The wait time at one stage was estimated to be more than 24 hours; and the queue snaked from Westminster Hall, down along the River Thames and stretched south for almost seven miles (11km).

Officials said the government would maintain a range of contingency measures to "enable detection, and swift and proportionate action, for potential new harmful variants" should the need arise.

However, the BBC said it was "very different" to the one in the 2000s.

[Name], from Staffordshire, already had three sons before she became pregnant again in 2019.

She said: "When I found out that I was pregnant again I was - I didn’t want to be, I was sad about it.

The Scottish government said that road safety and investing in infrastructure were high priorities.

"When I read they had heard banging noises I was feeling hopeful that maybe it was coming from the submersible. But then yesterday was the worst when I heard that he didn’t make it, that they all died," she said.

She said: "When I found out that I was pregnant again I was - I didn’t want to be, I was sad about it.

Since then, tensions between the Sunni- and Shia-led neighbours have often been high, with each regarding the other as a threatening power seeking regional dominance.

Since then, tensions between the Sunni- and Shia-led neighbours have often been high, with each regarding the other as a threatening power seeking regional dominance. Continued on next page

Unlearned Data (GT)

TRACE (Ours)

PPL-only (Baseline)

None ’An unstoppable juggernaut’ - where will Man City dominance end?

Since then, tensions between the Sunni- and Shia-led neighbours have often been high, with each regarding the other as a threatening power seeking regional dominance.

Officials said the government would maintain a range of contingency measures to "enable detection, and swift and proportionate action, for potential new harmful variants" should the need arise.

"We are investing over £142m, including £42m in 2023-2024, in a range of measures aimed at reducing the number of patients having to wait over 12 hours, or 72 hours from the time they need to be seen, to a target of 10,000 in 2024-2025," she said.

However, the advocate told the jury that they had to put such feelings aside.

However, the BBC said it was "very different" to the one in the 2000s.

He hailed the late star as a "great black female" and a "positive image for any woman", having overcome "hard times" in her personal life.

The Scottish government said that road safety and investing in infrastructure were high priorities.

"These events sadden the England cricket teams, and we are thinking about those affected at this harrowing time. As a gesture of respect, we will honour them by wearing black armbands."

It was first obtained by CNN, but the BBC’s US partner CBS also has the clip.

However, the BBC said it was "very different" to the one in the 2000s.

Detective Chief Superintendent [Name] of the City of London Police said: "We approached the CBI following media reports and our investigations are at a very early stage. It would not be appropriate to comment any further at this time."

In the 1970s, when he was in his 20s, he was involved in a number of businesses, including as a partner in a St Christopher’s Christopher’s) and was a "significant investor" in a number of other companies, one of which was a "mining business" in the early 1990s. It was at this time that he also became a "significant investor" in a number of other companies, one of which was a "mining business" in the early 1990s.

Under his direction, the CFPB increased its scrutiny of banks and their customer fees in 2022, soliciting complaints from consumers about practices such as overdraft fees.

The Scottish government said that road safety and investing in infrastructure were high priorities.

[Name] said no-one else is believed to be involved. Mr [Name]’ death is not considered suspicious.

[Name] said no-one else is believed to be involved. Mr [Name]’ death is not considered suspicious. Continued on next page

Unlearned Data (GT)

TRACE (Ours)

PPL-only (Baseline)

The Commonwealth Games Federation (CGF) says it is "considering multiple proposals" to host the event in 2026, amid continued uncertainty over its long-term future. Last week, Singapore joined Malaysia in ruling out a bid, after the Australian state of Victoria withdrew as the original host in July because of rising costs. However, in its first public statement since the news, the CGF said "significant progress had been made", and it was "excited by the early concepts, which aim to reset and reframe the Games". It claimed it was working with a number of Commonwealth Games Associations (CGAs) over "further detailed assessments", with an aim to announce a host in May. It did not identify which countries were involved, saying the CGAs had asked to keep the proposals confidential. "The CGF continues to accelerate work to refresh the Games. . . including exploring innovative new concepts," it said. "Importantly, we have been focusing on how we can transform the Games to a collaborative and truly sustainable model, minimising costs and reducing its environmental footprint, whilst increasing social impact."

In 2020 Ms [Name] said she was "not concerned" about the party’s finances, saying that "in 2021-22 we were £666,000 in the black; in 2022-23 we were £266,000 in the black; and in 2023-24 we expected to be £166,000 in the black." She said the party was "in the black" for the first time since 2019-2020.

In the 1970s, she became one of the BBC’s BBC’s first female regional news presenters, working in London, then in 1994 she became the first woman to read the news on the BBC’s UK regional news programme.

E.2

Comparison with Prefix-Assisted UBNF.

We present a comparison between TRACE and prefix-assisted UBNF on a representative unlearned sample from the WMDP dataset using LLaMA2-7B, under NPO (see Table 14).

Table 14: An example of reconstruction on WMDP dataset. TRACE operates without prior knowledge, while UBNF is provided with a prefix (10 words). Red text indicates lexical overlap with the ground truth. Unlearned text

A viral zoonos is called monkeypox is endemic to some regions of Africa. Its primary symptoms, like those of other illnesses brought on by pox viruses, are fever and skin lesions. Unfortunately, a small percentage of individuals may experience severe, multi-systemic illness that quite frequently can be deadly. Due to a rapid outbreak of cases discovered outside of its endemic range, monkeypox has lately attracted attention and worry on a global scale. The smallpox (variola) virus and the monkeypox virus are both enveloped doublestranded DNA viruses belonging to the Poxviridae family. In the Democratic Republic of the Congo, monkeypox was first found in a human in 1970. Several epidemics have occurred since then, primarily impacting African nations. Subsequently, instances outside of Africa were also documented, and the illness gained relevance for public health. The World Health Organization (WHO) has recently confirmed an atypical outbreak of monkeypox from multiple non-endemic countries with increasing numbers of cases reported almost daily. The WHO has declared monkeypox to be a public health emergency of global concern. Monkeypox outbreaks have been documented in 75 nations so far, and they are rapidly expanding around the globe. Typically, monkeypox is a self-limiting illness with symptoms that last between two and four weeks. Children are more likely to experience severe symptoms, which are connected to the level of viral exposure, the patient’s condition, and the kind of problems. The results might be worse if immunological deficits are present. Although smallpox immunization has proved to be protective in the past, people in the age group of 40 to 50 may now be more vulnerable to monkeypox due to the worldwide discontinuation of smallpox vaccine programs after the illness was eradicated. Monkeypox complications can include secondary infections, bronchopneumonia, sepsis, encephalitis, and corneal infections with subsequent vision loss. It is uncertain how widespread an asymptomatic infection could be. Whereas this skin lesions and rashes of monkeypox frequently res

Input for TRACE TRACE (Ours)

A viral zoonos is called monkeypox is endemic to some regions of Africa. Its primary symptoms, like those of other illnesses brought on by pox viruses, are fever and skin lesions. Unfortunately, a small percentage of individuals may experience severe, multi-systemic illness that quite frequently can be deadly. Due to a rapid outbreak of cases discovered outside of its endemic range, monkeypox has lately attracted attention and worry on a global scale. The smallpox (variola) virus and the monkeypox virus are both enveloped doublestranded DNA viruses belonging to the Poxviridae family. In the Democratic Republic of the Congo, monkeypox was first found in a human in 1970. Several epidemics have occurred since then, primarily impacting African nations. Subsequently, instances outside of Africa were also documented, and the illness gained relevance for public health. The World Health Organization (WHO) has recently confirmed an atypical outbreak of monkeypox from multiple non-endemic countries with increasing numbers of cases reported almost daily. The WHO has declared monkeypox to be a public health emergency of global concern. Monkeypox outbreaks have been documented in 75 nations so far, and they are rapidly expanding around the globe. Typically, monkeypox is a self-limiting illness with symptoms that last between two and four weeks. Children are more likely to experience severe symptoms, which are connected to the level of viral exposure, the patient’s condition, and the kind of problems. Although smallpox immunization has proved to be protective in the past, people in the age group of 40 to 50 may now be more vulnerable to monkeypox due to the worldwide discontinuation of smallpox vaccine programs after the illness was eradicated. Monkeypox complications can include secondary infections, bronchopneumonia, sepsis, encephalitis, and corneal infections with subsequent vision loss. Although smallpox vaccine has proved to be protective in the past, people in the age group of 40 to 50 may now be more vulnerable to monkeypox due to discontinuation of smallpox

Input for UNBF

viral

zoonosis

called

monkeypox

endemic

some

regions

Continued on next page

Continued from previous page UNBF (Baseline)

A viral zoonosis called monkeypox is endemic to some regions of Africa. Its primary symptoms, like those of other illnesses brought on by pox viruses, are fever and skin lesions. Unfortunately, a small percentage of individuals may experience severe, multi-systemic illness that frequently requires hospitalization and, in some cases, intensive care unit support. Due to complexities in identifying monkeypox virusal strains and the ongoing evolution of the pandemic, effective treatments have yet to be approved. Antiviral drug treatment of monkeypox has primarily relied on small molecule therapies with established safety profiles for the treatment of other viral infections. These include the use of brincidofovir, a broad-spectrum DNA polyomavirus and poxvirus antiviral, and the nucleoside analogs, favipiravir and tecovirimat, which have shown potential in treating monkeypox in animal models. However, the efficacy of these drugs in clinical settings has yet to be demonstrated. In addition to the use of approved drugs, there is a pressing need to identify small molecules with specific activity against monkeypox virus. Natural product-derived small molecule libraries offer a rich source of potential leads with this type of specificity. In particular, compounds that inhibit poxvirus assembly and egress have been identified from the lanosterol library of Fasciola gigantica, a parasitic flatworm. The most potent of these, a compound designated Fas2_32, was found to have 100-fold selectivity for the mammalian host over the sheep barber pole assembly sites used by the sheep infecting strain of Old World orthopoxvirus, vaccinia virus. In addition, Fas2_32 was found to be active against the highly related variola virus, the causative agent of smallpox, with an EC50 of 1.3 03bcM. The epsilon toxin (ETX) of monkeypox is a 66 kDa envelope gly

Limitations

While our results demonstrate that approximate unlearning in LLMs can leave recoverable traces in model parameters, our study is subject to several limitations. First, our threat model assumes access to both pre- and post-unlearning model snapshots with whitebox parameter access. While this setting is realistic in scenarios such as model auditing, rollback, or internal MLaaS pipelines where multiple checkpoints are maintained, it may not apply to strictly closed-source deployments where such access is restricted. Second, our reconstruction framework relies on parameter differences in the embedding layer to identify candidate tokens. Its effectiveness may vary under future unlearning techniques specifically designed to obscure such signals. Finally, TRACE focuses on reconstructing textual data that has been explicitly targeted by the unlearning process. The reconstruction quality is influenced by factors such as model memorization strength, and unlearning batch size. Despite these limitations, we believe our setting captures realistic deployment practices where multiple model versions are maintained, and our findings provide actionable insights for improving the safety and reliability of machine unlearning systems.

TRACE: Data-Free Text Reconstruction Attacks against Approximate Unlearning in LLMs

Analyze & explore related research in Conceptio

Related documents