TILDE: TILt-based Distributional Erasure for Concept Unlearning
Naveen George∗ Indian Institute of Technology Hyderabad [email protected]
Naoki Murata Sony AI [email protected]
Konda Reddy Mopuri Indian Institute of Technology Hyderabad
Yuhta Takida Sony AI
Yuki Mitsufuji Sony AI & Sony Group Corporation
arXiv:2607.06432v1 [cs.LG] 7 Jul 2026
Abstract Concept unlearning in text-to-image diffusion models is critical for safe and practical deployment: with rising privacy concerns, copyright disputes, trademark constraints, and safety regulations, deployed systems must be able to suppress unwanted concepts after training. Existing methods often remove the target concept effectively, but practical unlearning also requires an equally fundamental property: the unlearned model should retain quality, diversity, and semantic coverage on benign generation. The gold standard is a retain-only model trained from scratch without the unwanted data. However, common erasure objectives do not specify which post-unlearning distribution should approximate this reference, leaving retention as an implicit consequence of the update rule. We propose TILDE, TILt-based Distributional Erasure, which formulates concept unlearning as a distributional alignment problem: the desired target is the minimum-deviation conditional distribution from the pretrained model under a forgetting constraint. This energy-tilted, anchor-free target suppresses concept-expressing images while preserving benign relative mass for each prompt. We instantiate this principle with residual ∇-GFlowNet training, which learns the score correction induced by the forget energy relative to the pretrained diffusion model. Across objects, artistic styles, and characters, TILDE achieves strong forgetting while improving retention and distributional fidelity over prior baselines.
1
Introduction
Text-to-image diffusion models have rapidly become part of everyday creative and professional workflows, enabling high-fidelity image creation and editing across artistic styles, fictional characters, personal identities, products, and common scenes [32]. As these models become more widely used, the ability to edit their behavior after training is essential for legal compliance, privacy protection, and safety. Privacy regulations such as the GDPR recognize rights to erasure for personal data [8], while copyright, trademark, identity, and safety constraints can require suppressing specific concepts after training. Concept unlearning addresses this requirement by editing a pretrained text-to-image model so that it no longer generates images expressing an undesirable concept. The central challenge is that unlearning requires not only effective forgetting but equally effective preservation: the edited model should maintain image quality and prompt alignment on benign prompts, without degrading related or general concepts. An edited model that successfully forgets but degrades benign generation does not approximate unlearning; it solves the forgetting problem ∗ Work done during an internship at Sony AI.
Preprint.
Forget Prompt Related Prompt General Prompt
“A bakery counter in Van Gogh style.”
“Starry night mug on desk”
Diversity
“Brown teddy “Starry Night bear on bed” mural on bedroom”
Direct suppression
Reward-based optimization mode collapse
collateral damage
Pretrained
Direct suppression
general benign
related benign
target concept
general benign
related benign
target concept
Distribution alignment (TILDE, proposed)
Reward-based optimization
benign modes preserved
Distribution alignment (TILDE, proposed)
general benign
related benign
target concept
Figure 1: Concept unlearning failure modes and qualitative examples for Van Gogh style removal. Left: generations for forget, related, general, and diversity prompts under the pretrained model, direct suppression, reward-based optimization, and distribution alignment. Right: schematic postunlearning behavior over output regions. Direct suppression can damage related benign modes, while reward-based optimization can collapse diversity onto a few safe modes. Distribution alignment suppresses the target concept while preserving benign modes.
by introducing a retention failure. The ideal reference is a retain-only model trained from scratch without ever seeing the unwanted data: the target concept is absent, while the rest of the generative distribution remains intact [6, 42]. We study concept unlearning as an efficient approximation to this retain-only behavior. Desiderata. This view yields three desiderata. (D1) Effective forgetting: the post-unlearning model should assign low probability to images that express the target concept. (D2) Local preservation: benign prompts that are semantically close to the forgotten concept should remain close to pretrained behavior, because this is where collateral damage is most likely. (D3) Distributional fidelity: outside the forget region, the post-unlearning distribution should stay close to the pretrained or retain-only reference at the per-prompt level, not just match coarse aggregate scores such as FID [6]. Existing methods use erasure proxies such as score suppression [10, 19], anchor-based editing [20, 11, 5], preference or reward-based optimization [28], influence or trajectory-level steering [37, 22, 38], or trajectory-level GFlowNet objectives [21]. These objectives can suppress the forget concept, but they generally do not specify where the resulting distribution should land. This is precisely where retention fails: strong erasure can damage semantically adjacent benign concepts [1], while methods that maximize a scalar forgetting reward can reduce benign diversity by concentrating generation onto a narrow set of concept-free outputs [36]. Anchor-based methods avoid some direct-suppression damage by mapping the forget concept toward a designated target concept, but this introduces a separate choice of where the erased concept should go. Fixed anchors can be unstable or semantically biased, and recent continual-unlearning evidence shows that such coarse mappings compound into severe retention collapse under sequential deletion requests [14]; adaptive anchors make the target itself context-dependent [5]. Moreover, such objectives do not by themselves specify the desired post-unlearning distribution, leaving open how probability mass should be redistributed after erasure. These limitations motivate an anchor-free approach that removes mass from concept-expressing regions without selecting a replacement concept. Figure 1 illustrates this distinction by contrasting incomplete erasure, including collateral damage from direct suppression and diversity collapse under mode-seeking maximization, with the proposed minimum-deviation distributional target. We therefore argue that concept unlearning should be defined by its post-unlearning distribution before choosing an update rule. The retain-only gold standard and recent evidence that reference-specific unlearning metrics can mask distributional misalignment between the unlearned model and the retain-only reference [6], suggest a minimal-deviation principle: among conditional distributions that 2
sufficiently suppress the forget concept, choose the one closest to the original distribution induced by the pretrained model. This turns unlearning into a constrained conditional distribution-optimization problem rather than a local suppression, anchor replacement, or reward maximization problem. It also explains why retaining performance must be measured locally as well as globally: semantically nearby benign prompts are exactly where collateral damage is easiest to hide under coarse aggregate metrics. We propose TILDE (TILt-based Distributional Erasure), which realizes this principle by tilting each prompt-conditioned pretrained distribution according to a forget energy. Concept-expressing images are down-weighted, while benign images preserve their pretrained relative likelihoods up to normalization. Section 2 derives the corresponding constrained projection and its closed-form target. The resulting target is anchor-free and distributional: it specifies how probability mass should be reallocated after erasure, rather than choosing a replacement concept or maximizing a single safe output. We therefore adopt GFlowNet-based training [3], which targets sampling in proportion to an unnormalized terminal density. To make this compatible with diffusion finetuning, we use residual ∇-GFlowNet training [23]: the pretrained model supplies the base distribution, and the learner estimates only the score correction induced by the forget-energy tilt. This perspective also guides evaluation: forgetting must be reported together with neighboring benign preservation and distributional fidelity to pretrained or retrained behavior. Contributions. 1. We formulate concept unlearning as constrained distributional alignment, making the postunlearning distribution target explicit and deriving the corresponding anchor-free, promptconditioned minimal-deviation Gibbs form. 2. We introduce TILDE, which instantiates the energy-tilted target with residual ∇-GFlowNet training in diffusion latent space, using a thresholded terminal forget energy, residual score matching, and LoRA updates. 3. We evaluate unlearning of object, artistic-style and characters across forgetting, related/general retention, and distributional metrics including FID and FADE [6], showing improvements over prior baselines.
2
Target Distribution for Concept Unlearning
2.1
Prompt-Conditioned Target via Constrained KL Projection
We first make the desired post-unlearning conditional distribution explicit, instead of leaving it implicit in the choice of update rule. Let π(y) denote the fixed prompt population, including prompts that may elicit the concept C, related benign prompts, and general benign prompts. Let EC (x) ≥ 0 be a measurable forget energy that quantifies concept evidence: larger values indicate that image x more strongly expresses C, whereas benign images, including semantically adjacent benign images, should have small or zero energy. We define the KL-projection target by p∗C = arg min Ey∼π [KL(p(· | y) ∥ ppre (· | y))] p
s.t.
Ey∼π, x∼p(·|y) [EC (x)] ≤ δC ,
(1)
where the optimization is over conditional generators p(· | y), the prompt marginal π(y) is held fixed, the KL term penalizes unnecessary deviation from the pretrained conditional, and the constraint bounds the expected concept evidence under the post-unlearning distribution to at most δC > 0. Under standard regularity conditions for the constrained KL projection, this projection has the unique solution ppre (x | y) exp − βEC (x) ∗ pC (x | y) = , (2) Zβ,C (y) where Zβ,C (y) is the prompt-specific normalizing constant and β ≥ 0 is the Lagrange multiplier associated with the forgetting constraint. The tilt penalizes only concept-expressing images; benign images share the freed mass proportionally, so their pretrained relative likelihoods are preserved. The remaining design questions are how to instantiate EC and how to realize the resulting tilted distribution in a diffusion model. 3
2.2
Practical Surrogate Target: CLIP-Based Thresholded Forget Energy
In practice, we instantiate the abstract forget energy with a thresholded CLIP-based surrogate [30]. The CLIP score serves as a measure of concept evidence, while the threshold creates a no-penalty region for benign images. Let C denote the semantic concept to be forgotten, and let y denote a prompt that conditions image generation. We use a small set of CLIP text descriptors QC = {q1 , . . . , qM } as concept-evidence probes for C, distinct from the generation prompt y. Each qi is a short, paraphrased reference to C that fixes the visual identity without describing scene context, so probing is robust to lexical variation. For Pikachu, for instance, QC = {“Pikachu”, “a photo of Pikachu”, “an image of Pikachu the Pokemon”, “Pikachu, the yellow Pokemon character”}; analogous sets are used for artistic styles, e.g., “Van Gogh style”. For P a generated image x, we compute the average CLIP similarity to these descriptors, sC (x) = |QC |−1 q∈QC cos(CLIPimg (x), CLIPtxt (q)), and refer to this quantity as CLIP concept evidence. We then define the forget energy as 0, if sC (x) ≤ τ, e EC (x) = (3) λscale exp α(sC (x) − τ ) − 1 , if sC (x) > τ. Here τ is the CLIP-score threshold, α controls the sharpness above the threshold, and λscale sets the overall penalty strength, including the Gibbs multiplier β. The threshold is the key design choice. Without it, a continuous penalty based on sC (x) would exert nonzero gradients even on benign images with weak concept similarity. The thresholded energy instead creates a no-penalty region below τ , while applying the energy tilt primarily to high-evidence images above τ . This yields the practical surrogate target eC (x) . ptar,C (x | y) ∝ ppre (x | y) exp − E
(4)
This gives an anchor-free target: rather than prescribing a replacement concept, the tilt reweights the pretrained conditional density according to concept evidence. 2.3
Implication for Optimization
The target to realize is the prompt-conditioned tilted density in Eq. (4), rather than to seek modes that maximize a scalar forgetting reward. Accordingly, the learning objective is to realize proportional sampling from this target, preserving benign-region coverage while down-weighting high-energy images. We adopt residual ∇-GFlowNet because diffusion models are score-parameterized and Eq. (4) has a pretrained-times-energy-tilt structure that residual training matches directly. The next section details this realization.
3
Diffusion Realization via Residual ∇-GFlowNet
eC from Section 2.2. Our We fix C throughout this section and use the practical thresholded energy E diffusion realization instantiates the residual ∇-GFlowNet machinery [23] with the forget-energytilted target in Eq. (4). Consider a latent diffusion model [32]. For a prompt y, let zT ∼ N (0, I) be the initial noise and let zT → zT −1 → · · · → z0 denote the denoising chain, with decoded image x = D(z0 ). We write PFθ (zt−1 | zt , y) for the LoRA-adapted denoising transition and PFpre (zt−1 | zt , y) for the pretrained transition. The subscript F follows the GFlowNet convention [3] and denotes the sampling, or denoising, direction, not the forward noising Qtprocess used in diffusion training. We use standard DDPM notation, with αt = 1 − βt and ᾱt = s=1 αs . Residual balance condition. Eq. (4) defines a clean-image tilt of the pretrained sampler. The remaining question is how this terminal reweighting should be propagated through the denoising step. We introduce a positive residual continuation weight Fet (zt , y). Intuitively, it is the benign mass still reachable from zt : large when continuations tend to end in low-energy images, and small when they tend to end in concept-expressing high-energy images. The adapted transition should therefore 4
1. Sample a trajectory with
2. Evaluate terminal forget energy on
3. Learn the residual score correction
forget energy:
apple Prompt: “an apple on the desk”
No penalty
redirects denoising trajectories away from high forget-energy continuations.
penalty applies
Local score-correction condition: adapted score residual correction
Unlearned output:
pretrained score : shown condition at ; : complementary condition at
CLIP concept evidence
.
Figure 2: Overview of TILDE training in diffusion latent space. Given a target concept C and a sampled prompt y, the LoRA-adapted denoiser samples a trajectory zT → · · · → z0 , and evaluates the eC (x̂) from CLIP concept evidence. Residual ∇-GFlowNet training then thresholded forget energy E matches the score correction between PFθ (zt−1 | zt , y) and PFpre (zt−1 | zt , y) using the continuation score hϕ (zt , t, y), steering denoising trajectories away from high-energy concept-expressing outputs while preserving low-energy benign dynamics.
change the pretrained transition only by favoring next states with larger continuation weight: PFθ (zt−1 | zt , y) Fet−1 (zt−1 , y) = . pre PF (zt−1 | zt , y) Fet (zt , y)
(5)
This condition is useful because it factorizes the target path-probability ratio locally. Along any trajectory zT → · · · → z0 , the intermediate Fet terms cancel, leaving only Fe0 (z0 , y)/FeT (zT , y). eC (D(z0 ))) and FeT (zT , y) = cinit (y) independent of zT , the terminal With Fe0 (z0 , y) ∝ exp(−E eC (D(z0 ))), matching Eq. (4). Thus the condition marginal is proportional to ppre (z0 | y) exp(−E converts a terminal energy into proportional sampling: concept-expressing images lose mass, while benign modes keep their pretrained relative proportions. Since Fet is unknown, we learn only its score hϕ (zt , t, y) := ∇zt log Fet (zt , y), and use it to define the score-form training objective below. Training jointly optimizes the LoRA parameters and the auxiliary continuation-score parameters ϕ. Score-form training objective. Directly enforcing Eq. (5) is inconvenient in diffusion models because it involves transition densities. Following residual ∇-GFlowNet [23], we therefore use its score-form counterpart, which can be expressed using the local score information provided by the denoiser. Differentiating Eq. (5) with respect to zt−1 and using that Fet (zt , y) does not depend on zt−1 gives the local score-correction condition: ∇zt−1 log PFθ (zt−1 | zt , y) − ∇zt−1 log PFpre (zt−1 | zt , y) = hϕ (zt−1 , t − 1, y).
(6)
The left-hand side is the score correction introduced by finetuning, measuring the change in local probability flow relative to the pretrained denoiser. The right-hand side is the auxiliary estimate of the residual continuation score required by the same balance condition. We enforce this local consistency by penalizing the mismatch between the two vector fields, yielding the forward loss h i 2 Lfwd = E ∇zt−1 log PFθ (zt−1 | zt , y) − ∇zt−1 log PFpre (zt−1 | zt , y) − hϕ (zt−1 , t − 1, y) . (7) We also impose the complementary score condition obtained by differentiating the same residual balance relation with respect to zt , yielding the reverse loss Lrev . Together, Lfwd and Lrev make the residual correction locally consistent across each denoising transition. Unlike standard denoisingscore training [17, 33], the target is not ground-truth noise or a data score, but the residual correction that turns the pretrained sampler into the energy-tilted unlearned sampler. 5
The residual score hϕ (zt , t, y) is the score of the residual continuation weight induced by the terminal forget energy. Since this score is not available in closed form at intermediate noisy states, we use the forward-looking residual parameterization eC (x̂θ (zt , t, y)) + gϕ (zt , t). hϕ (zt , t, y) = −ᾱt ∇z E (8) t
Here x̂θ (zt , t, y) is the one-step clean-image estimate predicted from the current noisy latent, and ᾱt is the standard cumulative DDPM signal coefficient. The first term is the one-step pullback of the terminal-energy score, and gϕ parameterizes the remaining approximation residual of this continuation score. Interpretation of hϕ . The residual field hϕ can be interpreted as a state-dependent guide for how the adapted denoiser should deviate from the pretrained denoiser. Instead of manually specifying separate forget and retain directions, or selecting an anchor concept, TILDE lets the thresholded forget energy determine this direction from the current one-step clean estimate x̂θ (zt , t, y). When the estimate has high concept evidence, ∇zt ẼC (x̂θ ) is nonzero, so hϕ induces an active residual score correction that redirects the denoising trajectory away from concept-expressing regions. When the estimate lies below the threshold, the energy-pullback term is exactly zero, so the residual balance condition demands that the finetuned transition match the pretrained transition at these states. Benign trajectories are therefore encouraged to remain close to pretrained behavior. Overall loss and design rationale. We jointly optimize the LoRA update ∆θ and the auxiliary continuation-score parameters ϕ by minimizing L = Lfwd + λrev Lrev + λterm Lterm . (9) 2 Here, λrev and λterm are balancing weights, and Lterm penalizes ∥gϕ (z0 , 0)∥2 , keeping the clean-end boundary condition determined solely by the forget energy. Our contribution is the unlearning-specific distributional target and its full realization in diffusion latent space; residual ∇-GFlowNet provides the optimization backbone that makes this realization practical. Each component follows from this target. The residual form is appropriate because Eq. (4) changes each prompt-conditioned pretrained distribution by a multiplicative energy tilt. The score form is appropriate because diffusion models expose denoising scores. The forward-looking correction is appropriate because the forget signal is terminal, while the denoising chain is long. Finally, the thresholded energy from Section 2.2 yields no direct terminal-energy force outside the forget region, which is essential for preserving nearby benign concepts. We also restrict updates to LoRA adapters [18], adding a parameter-space minimal-update bias to the distribution-space minimal-deviation target.
4
Related Work
Current concept-unlearning methods. Existing approaches include direct suppression and selective finetuning (e.g., ESD, SalUn, and Selective Amnesia) [10, 9, 15], target-space editing frameworks such as UCE and MACE [11, 25], influence and trajectory-steering methods [37, 22, 38], and preference/reward-based optimization such as DUO, DDPO, and DRaFT [28, 4, 7]. These methods can remove target concepts effectively, but they often leave the final post-unlearning distribution implicit. Anchors, mapping, and dynamic targets. Many concept-erasure methods can be interpreted as choosing a destination representation and mapping the forget concept toward that target. Concept Ablation uses explicit anchor replacement [20], while other methods use broader target mappings at the embedding or attention level [11, 25]. AGE argues that fixed generic anchors are often suboptimal and proposes adaptive/dynamic target selection based on concept context [5]. LACU further shows that fixed anchors are especially brittle when deletion requests arrive sequentially, because large per-step displacements accumulate and damage neighboring concepts [14]. GFlowNet-based and distributional methods. Our work builds on ∇-GFlowNet [23], originally developed for reward-aligned sampling in diffusion models. EraseFlow [21] is the closest unlearningspecific GFlowNet baseline, which uses a trajectory-balance (TB) loss [26] on approximated logprobabilities; compared with it, we use score-based residual ∇-DB and learn only the deviation from the pretrained model rather than training a full sampler from scratch. 6
Failures, robustness, and side effects. Recent evidence shows that apparent forgetting is often fragile: erased concepts can be revived or bypassed under adversarial prompting and model probing [35, 41], and relearning/revival remains a practical threat [13, 12, 24]. This has motivated robust unlearning methods that incorporate adversarial training and robustness-aware objectives [40, 34]. Another persistent failure mode is the ripple effect on nearby concepts, where semantically adjacent benign concepts are unintentionally degraded after unlearning [1]. Related retention-focused unlearning work studies gradient conflicts between forget and retain objectives as one mechanism behind this trade-off [29]. Complementary analysis shows that gradient-ascent unlearning can fail because forget and retain data are statistically coupled, so ascent can move the model away from the retrained reference rather than toward it [27].
5
Experiments
5.1
Experimental Setup
Base Model and Training. All experiments are performed on Stable Diffusion v1.5 [32], the standard base model for concept unlearning research. We finetune LoRA adapters (rank r = 8) on the attention layers using AdamW with DDPM sampling; full training details and results on other SD variants are provided in the supplementary material. Concepts. We evaluate across four categories: objects, characters (e.g., celebrities, fictional characters), artistic styles, and nudity, covering a total of approximately 60 distinct concepts including targets, semantically related concepts, and general themes. Unlearning Prompts. For each concept C, we construct a forget prompt set YC with 30 diverse prompts using GPT-4o-mini, guided by a system prompt that enforces visual grounding (the concept must be a clearly depictable, in-frame subject), scene and category diversity (varied indoor/outdoor settings, framings, and activities), and lexical simplicity (short, single-clause natural-language phrasings). The same prompt generation pipeline is reused, with minor wording changes to the system prompt, when retain prompts are added to training (Section 5.3, Q5): we first ask the LLM to enumerate related concepts and then sample concept-disjoint scenes for each. Unless stated otherwise, QC contains the canonical text descriptor of C. 5.2
Evaluation Metrics
For every concept, we use 20 unique evaluation prompts and generate 8 images per prompt, giving approximately 10,000 images per model checkpoint. All evaluation prompts are unseen during training. VLM-Based Accuracy Metrics. We use Qwen2.5-VL-7B-Instruct [2] as our automated evaluator, querying it with binary questions on generated images. VLMs are substantially more reliable than ImageNet-style classifiers or UnlearnCanvas [39] style-classifier heads, which struggle with diverse concepts such as celebrities and artistic styles; supporting evidence is in the supplementary. • Unlearning Accuracy (Uacc ↑) & CLIP Score (Uclip ↑): Uacc measures forgetting effectiveness; Uclip measures “in-prompt retainability” [31], ensuring benign parts of the prompt are still generated correctly. • Related Retention (RRacc ↑; RRclip ↑): RRacc measures unintended collateral damage on semantically adjacent concepts (e.g., testing “Impressionism” after unlearning “Van Gogh”); RRclip verifies alignment is preserved. • General Retention (GRacc ↑; GRclip ↑): GRacc assesses overall knowledge preservation on general, unrelated prompts; GRclip verifies text-to-image alignment remains intact. Distributional Metrics: FID and FADE. We follow the UnlearnCanvas [39] evaluation protocol for FID [16], measuring distributional quality of generated images on retain prompts. For a more principled distributional assessment, we also report FADE [6], which measures functional alignment between the unlearned model and a gold-standard retain-only model by comparing 7
Table 1: Category-level concept unlearning performance averaged over concepts in each group. Base Stable Diffusion v1.5 reported SD v1.5 base: Accuracy 89%, CLIP score 32.6. Objects averages apple, banana, golf ball, and cat. Characters averages Brad Pitt, Lionel Messi, Mickey Mouse, and Pikachu. Style averages cartoon style, Van Gogh style, and Monet style. Each category reports unlearning accuracy (UAcc ), related-retain accuracy (RRAcc ), and general-retain accuracy (GRAcc ). The final columns report average FADE and FID over the style concepts when available. Objects
Method
Characters
Style
UAcc ↑ RRAcc ↑ GRAcc ↑ UAcc ↑ RRAcc ↑ GRAcc ↑ UAcc ↑ RRAcc ↑ GRAcc
Avg. Avg. FADE ↓ FID ↓ ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS [37] EDiff [38]
0.74 0.77 0.62 0.61 0.44 0.91 0.82 0.95 0.84 0.39
0.58 0.58 0.87 0.91 0.79 0.76 0.53 0.36 0.30 0.83
0.79 0.78 0.83 0.90 0.83 0.88 0.72 0.72 0.35 0.88
0.97 0.98 0.92 0.97 0.99 0.97 0.99 0.99 0.85 0.52
0.36 0.23 0.57 0.59 0.17 0.46 0.16 0.21 0.34 0.78
0.77 0.55 0.67 0.87 0.22 0.86 0.57 0.71 0.32 0.88
0.55 0.70 0.29 0.86 1.00 0.56 0.86 0.69 0.80 0.32
0.81 0.71 0.93 0.72 0.03 0.49 0.52 0.27 0.17 0.85
0.84 0.82 0.87 0.87 0.01 0.87 0.72 0.56 0.30 0.88
254 200 138 152 67370 358 400 40435 150 287
10.0 9.5 14.8 7.3 181.1 9.0 17.6 21.9 171.3 6.8
TILDE
0.97
0.72
0.86
0.99
0.64
0.87
0.88
0.76
0.84
130
9.6
bidirectional likelihood assignments over generated samples: pretain (x | y) punlearn (x | y) FADE := Ex∼pretain (·|y) log + Ex∼punlearn (·|y) log . punlearn (x | y) pretain (x | y)
(10)
FADE is non-negative and equals zero only when the two distributions are identical; a lower score indicates closer alignment to genuine unlearning. For diffusion models, FADE is tractably estimated via weighted differences of denoising MSE losses [6]. 5.3
Experimental Results and Analysis
Q1: How well do existing methods retain overall model performance during unlearning? Table 1 reveals a consistent failure mode: strong forgetting comes at the expense of collateral damage. Hard erasure (MACE) reaches near-perfect unlearning but destroys related and generalretain accuracy; softer suppression (ESD variants) preserves benign behavior but rarely reaches the needed level of erasure. Anchor-based CA [20] and UCE [11] rely on explicit retain supervision (anchor-concept distillation and a preserve-set attention edit, respectively), which buys image quality but still leaves residual concept evidence. Preference-based DUO is inconsistent across concept types, and the closest GFlowNet baseline (EraseFlow) shows particularly sharp retention drops on adjacent concepts. Q2: How well does TILDE perform? TILDE achieves near-complete forgetting across all category types while substantially improving retention over methods at the same unlearning operating point — and it does so without the explicit retain set or anchor concept that UCE and CA depend on, placing it on the Pareto frontier. With retain prompts optionally added, retention improves further (Q5). The advantage is most pronounced on character and style concepts; on objects, related-retain remains harder due to tighter semantic coupling with neighboring concepts. Q3: Why does the GFlowNet formulation help? We ablate each ingredient of the residual ∇GFlowNet objective on the Pikachu concept (Table 2); recall that [23] replaces intractable absolute log-flow targets with a gradient-form (score-matching) loss well-conditioned in latent space, and that the residual gϕ restricts learning to the score correction relative to the pretrained model.
8
The ∇ form is the primary driver of forgetting: dropping to 0th-order DB collapses unlearning to 0.40, since absolute-flow targets are too poorly conditioned in latent space to suppress the concept. The residual gϕ is the primary driver of retention: removing it forces the full score field to absorb the forget update, dropping general retention to 0.83. DDPO-style reward maximization keeps forgetting strong but collapses general retention to 0.33, confirming that GFlowNet’s proportional-sampling property is what prevents mode collapse onto concept-free outputs.
Table 2: Ablation of the ∇-GFlowNet formulation on Pikachu concept unlearning. All variants share the same thresholded forget energy and LoRA parameterization. Method
Uacc ↑ RRacc ↑ GRacc ↑
TILDE (resi ∇-GFlowNet) w/o residual w/o ∇ form w/o gϕ DDPO (reward max.)
0.99 0.96 0.40 0.97 0.98
0.76 0.79 0.81 0.72 0.61
0.88 0.87 0.87 0.83 0.33
Q4: How does the energy threshold τ affect unlearning? The thresholded forget energy (Eq. 3) is a key design choice: only images with CLIP concept evidence above τ receive a direct terminalenergy penalty. Figure 3 traces the accuracy and retention trade-offs across a range of threshold values on the Pikachu concept. Removing the threshold entirely (not shown) causes catastrophic retention collapse: Uacc =1.00 but RRacc =0.08, GRacc =0.49, since every sample receives an energy penalty regardless of concept relevance. As τ rises from a low value, forgetting stays strong while retention steadily improves, reflecting the growing no-tilt region that shields benign samples from direct terminal-energy gradients. Our chosen τ =0.22 strikes the point where forgetting remains near-complete and retention peaks; pushing τ higher causes forgetting to degrade sharply while retention gains diminish. eC (x) = 0 exactly, so the The reason: below τ , E explicit terminal-energy force vanishes for those clean predictions, leaving only indirect effects through residual-flow propagation and shared LoRA parameters.
TILDE 1.0
Accuracy
0.8 0.6 Uacc (forgetting) RRacc (related retain) GRacc (general retain)
0.4 0.2 0.0
0.19
0.21 0.22 0.23
0.25
Energy threshold
0.27
0.29
Figure 3: Effect of the energy threshold τ on Pikachu concept unlearning. Stars mark our chosen τ =0.22 (TILDE). Removing the threshold entirely yields (U, RR, GR) = (1.00, 0.08, 0.49) (off-axis).
Q5: Does including retain prompts during unlearning help further? Since benign images below τ receive no direct terminal-energy penalty, general or related-retain prompts can be naturally mixed into the training batch as a regularizer toward pretrained benign behavior (Table 3).
Table 3: Effect of including retain prompts during unlearning on Pikachu concept. Adding retain prompts improves overall retention without affecting unlearning accuracy. Setting
Uacc ↑ RRacc ↑ GRacc ↑
0.99 0.76 0.88 Both variants improve related-retain accuracy TILDE 0.82 0.88 without measurably affecting unlearning, con- TILDE + General retain 0.98 TILDE + Related retain 0.99 0.84 0.88 sistent with the thresholded-energy prediction: below-threshold samples receive no conceptsuppression gradient and act only as anchors toward pretrained behavior. Related-retain prompts give a slightly larger gain than general ones, since they probe the exact neighborhood where collateral damage leaks in through shared LoRA parameters. Crucially, this benefit is a free byproduct of thresholding: unlike CA and UCE, our pipeline absorbs retain prompts as ordinary training samples with no extra loss term, so the Pareto improvement of Q2 strictly extends rather than trades off when retain supervision is available.
9
6
Discussion
This work casts concept unlearning as distributional alignment rather than local suppression or anchor replacement: TILDE derives a minimum-deviation, energy-tilted target under a forgetting constraint, instantiated with thresholded CLIP energy and residual ∇-GFlowNet training in diffusion latent space. The results support the main prediction: strong forgetting without broad collateral damage when the target distribution is specified explicitly. More broadly, this framing unifies existing concept-erasure methods: hard-suppression objectives correspond to extreme tilts, anchor-based methods substitute a manually chosen target distribution, and reward-maximization variants drop the proportional-sampling property needed for diversity preservation. Within this view, TILDE is the natural minimal modification of the pretrained model that satisfies the forgetting constraint, while residual ∇-GFlowNet training makes the alignment efficient in practice. Limitations and future work. The forget energy must reliably measure concept evidence; CLIP similarity suffices for the objects, characters, identities and styles we evaluate, but ambiguous or highly compositional concepts may need stronger energies, such as VLM ensembles or taskspecific classifiers. The same residual ∇-GFlowNet machinery transfers without modification once a calibrated energy is provided. The threshold τ is a hyperparameter, but in our experiments it remained stable across concepts within a category.
References [1] Ibtihel Amara, Ahmed Imtiaz Humayun, Ivana Kajic, Zarana Parekh, Natalie Harris, Sarah Young, Chirag Nagpal, Najoung Kim, Junfeng He, Cristina Nader Vasconcelos, Deepak Ramachandran, Golnoosh Farnadi, Katherine Heller, Mohammad Havaei, and Negar Rostamzadeh. Erasing more than intended? how concept erasure degrades the generation of non-target concepts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16420–16430, 2025. [2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. [3] Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward J Hu, Mo Tiwari, and Emmanuel Bengio. GFlowNet foundations. Journal of Machine Learning Research, 24(210):1–55, 2023. [4] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learning Representations (ICLR), 2024. [5] Anh Tuan Bui, Thuy-Trang Vu, Long Tung Vuong, Trung Le, Paul Montague, Tamas Abraham, Junae Kim, and Dinh Phung. Fantastic targets for concept erasure in diffusion models and where to find them. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tZdqL5FH7w. [6] Sungjun Cho, Dasol Hwang, Frederic Sala, Sangheum Hwang, Kyunghyun Cho, and Sungmin Cha. Reference-specific unlearning metrics can hide the truth: A reality check. arXiv preprint arXiv:2510.12981, 2025. [7] Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable rewards. In International Conference on Learning Representations (ICLR), 2024. [8] European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the european parliament and of the council of 27 april 2016. Official Journal of the European Union, 2016. URL https://eur-lex.europa.eu/legal-content/EN/TXT/?uri= CELEX:32016R0679. General Data Protection Regulation, Article 17: Right to erasure. [9] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=gn0mIhQGNM. 10
[10] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2426–2436, 2023. [11] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5111–5120, 2024. [12] Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijie Deng, and Min Lin. Metaunlearning on diffusion models: Preventing relearning unlearned concepts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2131–2141, 2025. [13] Naveen George, Karthik Nandan Dasaraju, Rutheesh Reddy Chittepu, and Konda Reddy Mopuri. The illusion of unlearning: The unstable nature of machine unlearning in text-to-image diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13393–13402, 2025. [14] Naveen George, Naoki Murata, Yuhta Takida, Konda Reddy Mopuri, and Yuki Mitsufuji. Locality-aware continual unlearning for diffusion models. In European Conference on Computer Vision. Springer, 2026. [15] Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models. Advances in Neural Information Processing Systems, 36:17170–17194, 2023. [16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), 2017. [17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. [18] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. [19] Chi-Pin Huang, Kai-Po Chang, Chung-Ting Tsai, Yung-Hsuan Lai, Fu-En Yang, and YuChiang Frank Wang. Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. In European Conference on Computer Vision, pages 360–376. Springer, 2024. [20] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22691–22702, 2023. [21] Naga Sai Abhiram Kusumba, Maitreya Patel, Kyle Min, Changhoon Kim, Chitta Baral, and Yezhou Yang. EraseFlow: Learning concept erasure policies via GFlowNet-driven alignment. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=igB289kbej. [22] Leyang Li, Shilin Lu, Yan Ren, and Adams Wai-Kin Kong. Set you straight: Auto-steering denoising trajectories to sidestep unwanted concepts. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 9257–9266, 2025. [23] Zhen Liu, Tim Z. Xiao, Weiyang Liu, Yoshua Bengio, and Dinghuai Zhang. Efficient diversitypreserving diffusion alignment via gradient-informed GFlowNets. In International Conference on Learning Representations (ICLR), 2025. [24] Kevin Lu, Nicky Kriplani, Rohit Gandikota, Minh Pham, David Bau, Chinmay Hegde, and Niv Cohen. When are concepts erased from diffusion models? In 39th Conference on Neural Information Processing Systems (NeurIPS), 2025.
11
[25] Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6430–6440, 2024. [26] Nikolay Malkin, Moksh Jain, Emmanuel Bengio, Chen Sun, and Yoshua Bengio. Trajectory balance: Improved credit assignment in GFlowNets. In Advances in Neural Information Processing Systems (NeurIPS), 2022. [27] Ioannis Mavrothalassitis, Pol Puigdemont, Noam Itzhak Levi, and Volkan Cevher. Ascent fails to forget. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=KBJSV1XApq. [28] Yong-Hyun Park, Sangdoo Yun, Jin-Hwa Kim, Junho Kim, Geonhui Jang, Yonghyun Jeong, Junghyo Jo, and Gayoung Lee. Direct unlearning optimization for robust and safe text-to-image models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [29] Gaurav Patel and Qiang Qiu. Learning to unlearn while retaining: Combating gradient conflicts in machine unlearning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4211–4221, 2025. [30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021. [31] Jie Ren, Kangrui Chen, Yingqian Cui, Shenglai Zeng, Hui Liu, Yue Xing, Jiliang Tang, and Lingjuan Lyu. Six-cd: Benchmarking concept removals for text-to-image diffusion models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28769–28778, 2025. [32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. [33] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021. [34] Koushik Srivatsan, Fahad Shamshad, Muzammal Naseer, Vishal M Patel, and Karthik Nandakumar. Stereo: A two-stage framework for adversarially robust concept erasing from text-to-image diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23765–23774, 2025. [35] Yu-Lin Tsai, Chia-Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? In 12th International Conference on Learning Representations, 2024. [36] Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M. Tseng, Tommaso Biancalani, and Sergey Levine. Finetuning of continuous-time diffusion models as entropy-regularized control. arXiv preprint arXiv:2402.15194, 2024. [37] Jing Wu and Mehrtash Harandi. Scissorhands: Scrub data influence via connection sensitivity in networks. In European Conference on Computer Vision, pages 367–384. Springer, 2024. [38] Jing Wu, Trung Le, Munawar Hayat, and Mehrtash Harandi. Erasing undesirable influence in diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 28263–28273, 2025. [39] Yihua Zhang, Chongyu Fan, Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jiancheng Liu, Gaoyuan Zhang, Gaowen Liu, Ramana Kompella, Xiaoming Liu, and Sijia Liu. UnlearnCanvas: A stylized image dataset for enhanced machine unlearning evaluation in diffusion models. In Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. 12
[40] Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. Advances in neural information processing systems, 37: 36748–36776, 2024. [41] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now. In European Conference on Computer Vision, pages 385–403. Springer, 2024. [42] Kairan Zhao, Meghdad Kurmanji, George-Octavian Bărbulescu, Eleni Triantafillou, and Peter Triantafillou. What makes unlearning hard and what to do about it. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview. net/forum?id=QAbhLBF72K.
13
Appendix Table of Contents
A. Algorithm Pseudocode............................................................................................................ 15 B. Diffusion and ∇-GFlowNet Background .............................................................................. 16 B.1
Diffusion Models .................................................................................................................. 16
B.2
GFlowNets and Detailed Balance ........................................................................................... 17
B.3
Score-Based Detailed Balance ................................................................................................ 18
C. Theoretical Analysis and Proofs ............................................................................................ 20 C.1
Residual ∇-DB Losses .......................................................................................................... 20
C.2
Distributional Correctness...................................................................................................... 21
C.3
DB Score Matching Correspondence ...................................................................................... 22
C.4
Reverse Gaussian Score ......................................................................................................... 22
C.5
Forget-Energy Adaptation ...................................................................................................... 23
C.6
Optimization Caveat .............................................................................................................. 23
D. Extended Derivations ............................................................................................................. 24 D.1
Closed-Form Conditional Gibbs Projection ............................................................................. 24
D.2
Gaussian Instantiation of ∇-DB ............................................................................................. 25
D.3
Non-Ideal Model Correction .................................................................................................. 25
E. Additional Experimental Details ........................................................................................... 26 E.1
CLIP Model ......................................................................................................................... 26
E.2
Base Diffusion Model and Adapters ........................................................................................ 26
E.3
Prompt Sets and Generation Strategy ...................................................................................... 26
E.4
Optimization and Schedule .................................................................................................... 27
E.5
Compute .............................................................................................................................. 28
E.6
Hyperparameter Sensitivity .................................................................................................... 28
E.7
Evaluation Protocol ............................................................................................................... 28
F. Performance Across Multiple Diffusion Architectures......................................................... 31 G. Per-Concept Quantitative Results ......................................................................................... 32
14
A
Algorithm Pseudocode
Algorithm 1 Forget-Energy-Tilted ∇-GFlowNet Training Require: Pretrained denoiser ϵpre ; CLIP model; concept C; descriptors QC ; forget prompts YC ; hyperparameters τ, α, λscale 1: Setup. Initialize LoRA parameters ∆θ on U-Net attention layers (WQ , WK , WV , WO ) with rank r = 8. 2: Initialize the residual parameterization network gϕ with channels {64, 128, 256, 256}. 3: Initialize the adapted denoiser ϵθ ← ϵpre + ∆θ. 4: 5: for epoch = 1, . . . , Nepochs do 6: for batch = 1, . . . , Nbatches do 7: Sample forget prompts {yi } from YC 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25:
Phase 1: sample prompt-conditioned trajectories. Sample zT ∼ N (0, I) for t = T, T − 1, . . . , 1 do ϵθ (zt , t, yi ) ← LoRA-adapted U-Net prediction zt−1 ∼ PFθ (· | zt , yi ) via one DDPM denoising step end for Decode terminal latent: x̂ ← D(z0 ) Phase 2: evaluatePthe terminal forget energy. sC (x̂) ← |QC |−1 q∈QC simCLIP (x̂, q) mC (x̂) ← max(sC (x̂) − τ, 0) eC (x̂) ← λscale exp(α mC (x̂)) − 1 E
28: 29: 30:
Phase 3: match residual ∇-DB scores. Sample a timestep subset T ⊂ {1, . . . , T } with |T | = 0.1T for t ∈ T do sfwd ← µθ (zt , t, yi ) − µpre (zt , t, yi ) /σt2 srev ← ∇zt log PFpre (zt−1 | zt , yi ) − ∇zt log PFθ (zt−1 | zt , yi ) eC (x̂θ (zt−1 , t − 1, yi )) + gϕ (zt−1 , t − 1) ht−1 ← −ᾱt−1 ∇zt−1 E eC (x̂θ (zt , t, yi )) + gϕ (zt , t) ht ← −ᾱt ∇zt E Accumulate Lfwd from ∥sfwd − ht−1 ∥2 and Lrev from ∥srev − ht ∥2 end for
31: 32: 33: 34: 35:
Phase 4: update parameters. Lterm ← ∥gϕ (z0 , 0)∥2 {data-end boundary} L ← Lfwd + λrev Lrev + λterm Lterm Update ∆θ and ϕ with AdamW
26: 27:
36: 37:
end for
38: end for 39: return LoRA-adapted model ϵpre + ∆θ
15
B
Background on Diffusion Models and ∇-GFlowNets
This section reviews the two technical foundations underlying our method: (i) diffusion models, which provide the generative model and the score-based view of denoising transitions; and (ii) GFlowNets and their continuous score-matching variant ∇-GFlowNets, which provide the optimization framework for tilting the diffusion sampler toward an unnormalized target. The presentation is intended to be self-contained for a reader familiar with general deep generative modeling but not necessarily fluent in both subfields, and to fix the notation used by the proofs in Appendix C and Appendix D. B.1
Diffusion Models
Denoising diffusion probabilistic models (DDPMs) [17, 33] learn a generative model by reversing a fixed corruption process. The corruption process gradually adds Gaussian noise to data until samples are indistinguishable from N (0, I); the generative model is then a neural network trained to invert this corruption one step at a time. Sampling proceeds by drawing pure noise and iteratively denoising back to a clean datum. Forward (noising) process. Let z0 be a clean latent. The forward process is a fixed Markov Qt chain governed by a variance schedule {βt }Tt=1 , with αt = 1 − βt and cumulative product ᾱt = s=1 αs . Each per-step kernel is Gaussian: √ q(zt | zt−1 ) = N ( αt zt−1 , (1 − αt )I) . (11) A standard property of this chain is that zt given z0 is itself Gaussian: √ q(zt | z0 ) = N ᾱt z0 , (1 − ᾱt )I ,
(12)
which lets us directly sample any noisy intermediate without rolling out the chain. The schedule is chosen so that ᾱT ≈ 0, making q(zT ) essentially indistinguishable from a standard Gaussian. Reverse (denoising) process. Generation requires sampling from q(zt−1 | zt ), which is intractable in closed form. The model parameterizes a Gaussian approximation pθ (zt−1 | zt , y) = N µθ (zt , t, y), σt2 I , (13) where y is an optional conditioning input (e.g., a text prompt) and σt2 is fixed by the schedule. Using the standard z0 -prediction reparameterization, the predicted mean is rewritten in terms of a learned noise predictor ϵθ : 1 1 − αt µθ (zt , t, y) = √ zt − √ ϵθ (zt , t, y) . (14) αt 1 − ᾱt Throughout the main text, PFθ (zt−1 | zt , y) denotes pθ (zt−1 | zt , y) in GFlowNet notation; the subscript F refers to the (forward) sampling direction of the GFlowNet, not to the noising direction. Training objective. The denoiser is trained by minimizing the simplified denoising score-matching loss h 2i √ √ LDDPM = Ez0 ,ϵ,t ϵ − ϵθ ᾱt z0 + 1 − ᾱt ϵ, t, y , (15) which is, up to per-timestep weighting, equivalent to maximizing a variational lower bound on log pθ (z0 ) and to a weighted denoising score-matching objective. Our method does not re-train this objective; we keep the pretrained ϵpre frozen and learn only a low-rank correction ∆θ on top.
16
Score-based view. The crucial connection used by our method is between the noise predictor ϵθ and the score function of the marginal pt (zt | y): ϵθ (zt , t, y) ∇zt log pt (zt | y) = − √ . 1 − ᾱt
(16)
That is, the noise predictor and the marginal score differ only by a deterministic, time-dependent rescaling. As a consequence, a diffusion U-Net is, “for free,” a score model. Any modification we wish to apply to the sampler can be expressed as a modification to the per-step score, which is exactly the form on which ∇-GFlowNet operates. One-step clean-image estimate. At any timestep t, the noise predictor implies a clean-image estimate via Eq. (12): √ zt − 1 − ᾱt ϵθ (zt , t, y) √ x̂θ (zt , t, y) = D . (17) ᾱt Here D is the latent decoder; in pixel diffusion, D is the identity. This x̂θ is what allows us to evaluate eC from a noisy intermediate state, and it is the object differentiated in the the terminal forget energy E forward-looking flow score of Eq. (8). Latent diffusion models. Stable Diffusion-style latent diffusion models (LDMs) [32] run the diffusion process in the latent space of a pretrained encoder–decoder pair (E, D) rather than at pixel resolution. Latents zt are spatially much smaller than the corresponding image, and decoded images are obtained as x = D(z0 ). Conditioning on text prompts y enters through cross-attention layers in the U-Net. Our experiments use Stable Diffusion v1.5, and all derivations refer to latents zt unless otherwise stated. Why this matters for unlearning. Two properties of diffusion models make them a particularly clean substrate for our method: • Tractable per-step Gaussians. Both the pretrained transition PFpre and the LoRA-adapted transition PFθ are Gaussians with the same variance, so all log-density gradients used in the residual ∇-DB losses are available in closed form (Appendix D.2). • Score representation by construction. Any per-step modification to the sampler can be expressed entirely through modifications to the noise predictor, which is also a score model via Eq. (16). B.2
GFlowNets and the Detailed Balance Condition
What problem do GFlowNets solve? Generative Flow Networks (GFlowNets) [3] are amortized samplers for unnormalized reward distributions. Given a non-negative reward R(x) ≥ 0, the goal is to learn a policy whose terminal-state marginal satisfies PF⊤ (x) ∝ R(x),
(18)
where PF⊤ denotes the marginal at the end of the sampling chain. The defining feature is that this R proportionality is achieved without ever computing the partition function Z = R(x) dx. Once trained, sampling requires a single forward pass through the policy, in contrast to MCMC methods that require per-sample iteration. Trajectory and flow view. A GFlowNet builds a terminal sample through a sequence of intermediate states s0 → s1 → · · · → sT = x, governed by a forward (sampling) policy PF (st+1 | st ) and a backward policy PB (st | st+1 ). The flow function F (st ) ≥ 0 assigns a non-negative weight to each state and plays the role of an unnormalized marginal: intuitively, F (st ) measures the total reward reachable from st . In the diffusion setting, the trajectory is the denoising chain zT → · · · → z0 , PF is the denoising kernel, and PB is the (fixed) DDPM noising kernel. 17
Detailed Balance. The Detailed Balance (DB) objective is the most local of the standard GFlowNet objectives, in that it constrains a single transition rather than a full trajectory: PF (st+1 | st ) F (st ) = PB (st | st+1 ) F (st+1 ),
F (sT ) = R(sT ).
(19)
The terminal boundary condition F (sT ) = R(sT ) ties the flow to the reward at the data end of the chain. Whenever DB holds at every transition together with this boundary condition, a telescoping argument cancels the intermediate F -terms along any trajectory, leaving only the boundary contributions, and yields the desired proportional-sampling property PF⊤ (x) ∝ R(x). Why does the partition function cancel? DB only requires ratios of policy densities and flows. The unknown partition function Z enters F only as an overall multiplicative constant, which appears in both F (st ) and F (st+1 ) and therefore cancels in any pairwise ratio. This is why GFlowNets can sample from an unnormalized target without ever evaluating Z – exactly the property our unlearning eC (x)) requires, since the prompt-dependent partition function is intractable. target ppre (x | y) exp(−E How GFlowNets relate to MCMC and standard generative modeling. GFlowNets occupy a middle ground: like MCMC, they target an unnormalized density; like standard generative models, they amortize sampling into a single forward pass at inference time. This combination is precisely what concept unlearning at inference-time requires: a tilted diffusion sampler that runs at the same per-sample cost as the original. B.3
∇-GFlowNet: Score-Based Detailed Balance
Why standard DB is hard for diffusion. Direct enforcement of Eq. (19) requires evaluating log PF and log PB pointwise. For continuous, high-dimensional state spaces such as diffusion latents, log-densities of learned transitions can be expensive or unstable to evaluate, especially when PF is parameterized by a large neural network. This makes pure DB a poor match for diffusion-based samplers. The ∇ trick. Liu et al. [23] observe that one can sidestep log-densities by differentiating the DB condition with respect to the state, converting a log-probability balance into a score balance. Taking log of Eq. (19) and then ∇st+1 , and using that F (st ) does not depend on st+1 , gives the ∇-DB equation: ∇st+1 log PF (st+1 | st ) = ∇st+1 log PB (st | st+1 ) + ∇st+1 log F (st+1 ).
(20)
The same condition can be obtained by differentiating with respect to st , yielding a complementary reverse-score equation; see Appendix C.1. Crucially, all three terms are scores rather than logdensities, so the equation can be enforced as an ℓ2 -style score-matching loss using exactly the gradient information that diffusion models already expose through Eq. (16). Residual variant. The full DB condition asks for a flow F that is consistent with the absolute target reward. For unlearning, however, we are not changing the target distribution from scratch; we are tilting an already-trained diffusion model. It is therefore natural to factor out the pretrained sampler and learn only the residual ratio F̃ (zt , y) =
F (zt , y) , F pre (zt , y)
P̃Fθ (zt−1 | zt , y) =
PFθ (zt−1 | zt , y) , PFpre (zt−1 | zt , y)
(21)
which is exactly the residual ∇-DB condition stated as Eq. (5) in the main text. Subtracting the pretrained instance of Eq. (20) from the finetuned instance cancels the (fixed) noising kernel PB and yields a clean residual score equation; the formal derivation is in Appendix C.1. Why this is the right tool for our problem.
Three properties line up with concept unlearning:
• Native to diffusion. The optimization works in score space, which is the natural representation of a diffusion U-Net. 18
• Unnormalized targets. The flow function absorbs the unknown partition function, so the eC is sufficient. unnormalized energy E • Per-transition local objective. Each timestep contributes its own loss, so optimization decomposes over the chain and is compatible with mini-batch training. Together, these motivate our use of residual ∇-GFlowNet as the optimization backbone (Section 3); the unlearning-specific choices – the prompt-conditioned tilted target, the thresholded forget energy, and the forward-looking flow parameterization – are layered on top of this backbone.
19
C
Theoretical Analysis and Proofs
C.1
From Detailed Balance to the Trainable Residual ∇-DB Losses
For a fixed prompt y, let PB (zt |zt−1 ) denote the fixed DDPM noising transition, while PF denotes the denoising transition in the GFlowNet sampling direction. The DDPM denoising transition zt → zt−1 satisfies the detailed-balance relation PF (zt−1 |zt , y) F (zt , y) = PB (zt |zt−1 ) F (zt−1 , y).
(22)
Taking logarithms gives log PF (zt−1 |zt , y) + log F (zt , y) = log PB (zt |zt−1 ) + log F (zt−1 , y).
(23)
Differentiating with respect to the next denoised state zt−1 yields the forward ∇-DB equation ∇zt−1 log PF (zt−1 |zt , y) = ∇zt−1 log PB (zt |zt−1 ) + ∇zt−1 log F (zt−1 , y),
(24)
because F (zt , y) does not depend on zt−1 . Differentiating instead with respect to the conditioning latent zt yields the reverse equation ∇zt log PB (zt |zt−1 ) − ∇zt log PF (zt−1 |zt , y) = ∇zt log F (zt , y),
(25)
because F (zt−1 , y) does not depend on zt . Now let PFpre be the pretrained denoising policy and PFθ the finetuned one, both conditioned on the same prompt y. Subtracting the pretrained and finetuned forward equations cancels the fixed noising policy PB and gives ∇zt−1 log PFθ (zt−1 |zt , y) − ∇zt−1 log PFpre (zt−1 |zt , y) = ∇zt−1 log F̃ (zt−1 , y),
(26)
with F̃ = F/F pre . Subtracting the reverse equations gives ∇zt log PFpre (zt−1 |zt , y) − ∇zt log PFθ (zt−1 |zt , y) = ∇zt log F̃ (zt , y).
(27)
These two equations are exactly the targets enforced by Lfwd and Lrev . With the forward-looking parameterization from Eq. (8), the same residual flow network is evaluated at the state appearing on the right-hand side: • forward direction: evaluate ∇ log F̃ at (zt−1 , y); • reverse direction: evaluate ∇ log F̃ at (zt , y). Therefore the practical losses are 2 Lfwd = E ∇zt−1 log PFθ (zt−1 |zt , y) − ∇zt−1 log PFpre (zt−1 |zt , y) − ∇zt−1 log F̃ (zt−1 , y) , (28) 2 Lrev = E ∇zt log PFpre (zt−1 |zt , y) − ∇zt log PFθ (zt−1 |zt , y) − ∇zt log F̃ (zt , y) .
(29)
The terminal penalty Lterm = Ez0 ,y [∥gϕ (z0 , 0)∥2 ] enforces the data-end boundary condition. How the gradients are computed in practice. The forward residual score has a closed form for Gaussian transitions, which yields Eq. (28). The reverse score differentiates the transition log-density with respect to the conditioning variable, so it depends on the Jacobian of the denoiser mean. For PFθ (zt−1 |zt , y) = N (µθ (zt , t, y), σt2 I),
20
we have ∇zt log PFθ (zt−1 |zt , y) =
1 Jµ (zt , t, y)⊤ (zt−1 − µθ (zt , t, y)). σt2 θ
(30)
The pretrained term is obtained by replacing µθ with µpre . These Jacobian-vector products are computed with standard autodiff and do not require forming a dense Jacobian matrix. C.2
Distributional Correctness
The telescoping argument must be stated in integrated ratio form. A score identity of the form ∇ log P̃Fθ = ∇ log F̃ only determines log P̃Fθ up to an additive function of the conditioning state, so it is not by itself sufficient to justify telescoping across time. We therefore state the distributional theorem directly for the residual transition ratio. Eq. (32) is the multiplicative residual analogue of the detailed-balance factorization in Bengio et al. [3] and the integrated version of the residual result stated by Liu et al. [23, Proposition 4]. Theorem 1 (Distributional Correctness). For every fixed prompt y, let PFpre denote the pretrained denoising policy, let PFθ denote the finetuned denoising policy, and let E(x) ≥ 0 be the terminal forget energy used for unlearning. Define the residual transition ratio and residual flow by P̃Fθ (zt−1 | zt , y) :=
PFθ (zt−1 | zt , y) , PFpre (zt−1 | zt , y)
F̃ (zt , y) :=
F (zt , y) . F pre (zt , y)
(31)
Assume that for every denoising step t = 1, . . . , T , P̃Fθ (zt−1 | zt , y) =
F̃ (zt−1 , y) , F̃ (zt , y)
(32)
that the pretrained and finetuned models share the same initial noise prior zT ∼ N (0, I), and that at the data end z0 the residual terminal flow satisfies F̃ (z0 , y) ∝ exp − E(x) , x = D(z0 ), (33) while the initial residual flow is constant in zT on the support of the shared noise prior, i.e., there exists cinit (y) > 0 such that F̃ (zT , y) = cinit (y).
(34)
pθ (x | y) ∝ ppre (x | y) · exp − E(x) .
(35)
Then:
Proof. Because the pretrained and finetuned models share the same initial noise prior, the ratio of path probabilities along a denoising trajectory zT → zT −1 → · · · → z0 is T T Y Y pθ (z0:T | y) PFθ (zt−1 | zt , y) = = P̃ θ (zt−1 | zt , y). pre ppre (z0:T | y) t=1 PF (zt−1 | zt , y) t=1 F
(36)
Applying Eq. (32) gives T Y pθ (z0:T | y) F̃ (zt−1 , y) F̃ (z0 , y) = = ∝ exp − E(x) , ppre (z0:T | y) t=1 F̃ (zt , y) F̃ (zT , y)
(37)
where the last proportionality uses the assumptions that F̃ (z0 , y) ∝ exp(−E(x)) and F̃ (zT , y) = cinit (y), with cinit (y) independent of zT . The right-hand side depends on the trajectory only through
21
the terminal clean sample x, so marginalizing over intermediate latents preserves the same proportionality for the image distribution: pθ (x | y) ∝ exp − E(x) . ppre (x | y)
(38)
Rearranging gives the claim. Remark 1 (What score matching alone guarantees). Zero score loss enforces equality of gradients of log transition ratios. This only determines log P̃Fθ (zt−1 | zt , y) up to an additive function of (zt , y), so the theorem above is intentionally stated in the stronger integrated ratio form rather than inferred directly from gradient equality. C.3
DB Score Matching Correspondence
Proposition 1. For Gaussian denoising transitions PFpre (zt−1 | zt , y) = N (µpre (zt , t, y), σt2 I),
PFθ (zt−1 | zt , y) = N (µθ (zt , t, y), σt2 I), the residual forward score is
∇zt−1 log P̃Fθ (zt−1 | zt , y) = ∇zt−1 log PFθ (zt−1 | zt , y) − ∇zt−1 log PFpre (zt−1 | zt , y) pre
=
µθ (zt , t, y) − µ σt2
(zt , t, y)
.
(39) (40)
Proof. The Gaussian scores with respect to the next denoised state zt−1 are ∇zt−1 log PFθ = −
zt−1 − µθ (zt , t, y) , σt2
∇zt−1 log PFpre = −
zt−1 − µpre (zt , t, y) . σt2
(41)
Subtracting gives ∇zt−1 log PFθ − ∇zt−1 log PFpre =
µθ (zt , t, y) − µpre (zt , t, y) , σt2
(42)
which is exactly ∇zt−1 log P̃Fθ (zt−1 | zt , y) by definition of the residual transition ratio. Interpretation. Eq. (28) uses exactly this forward-vs-pretrained residual score. This is a local score-matching identity for the transition ratio; it does not by itself prove the global telescoping statement of Theorem 1. C.4
Reverse Gaussian Score
Proposition 2 (Gaussian reverse transition score). For a Gaussian denoising transition PFθ (zt−1 | zt , y) = N (µθ (zt , t, y), σt2 I), with variance independent of zt , the score with respect to the conditioning latent is ∇zt log PFθ (zt−1 | zt , y) =
1 Jµ (zt , t, y)⊤ (zt−1 − µθ (zt , t, y)). σt2 θ
Proof. Writing log PFθ (zt−1 |zt , y) = −
1 ∥zt−1 − µθ (zt , t, y)∥2 + Ct , 2σt2
22
(43)
where Ct does not depend on zt , the chain rule gives 1 ∇zt (zt−1 − µθ (zt , t, y))⊤ (zt−1 − µθ (zt , t, y)) 2 2σt 1 = 2 Jµθ (zt , t, y)⊤ (zt−1 − µθ (zt , t, y)). σt
∇zt log PFθ (zt−1 |zt , y) = −
(44) (45)
This is the expression used in the implementation of the reverse residual score. C.5
Adapting ∇-DB for Forget Energies
eC (x)) is always positive, so the For every prompt y, the terminal conditional density ratio exp(−E standard GFlowNet requirement that terminal weights be nonnegative is automatically satisfied. Proposition 3 (Derivative of the thresholded forget energy). Let eC (x) = λscale · exp α · max(sC (x) − τ, 0) − 1 . E
(46)
Then for sC (x) ̸= τ , eC ∂E = ∂sC
(
λscale · α · exp(α(sC − τ )), 0,
if sC > τ, if sC < τ.
(47)
eC is not differentiable; the implementation uses the zero subgradient. At the kink sC (x) = τ , E eC (x) = 0 and the derivative is zero. If Proof. If sC (x) < τ , then the maximum term is zero, so E sC (x) > τ , then eC (x) = λscale · exp α(sC (x) − τ ) − 1 , E (48) whose derivative is the stated exponential term. At sC (x) = τ , the left derivative is 0 while the right derivative is λscale α, so only a subgradient is defined. For the thresholded energy in Eq. (3), the energy gradient vanishes below the threshold and grows exponentially above it: • The threshold τ ensures benign images receive zero local energy gradient. • The exponential penalty concentrates updates on clearly concept-resembling images. • The positive terminal weight remains compatible with proportional sampling through eC (x)). exp(−E C.6
Optimization Caveat
The training objective is a nonconvex score-matching surrogate over the LoRA parameters and the auxiliary residual parameterization. We therefore do not claim a general “if and only if” theorem from L = 0 to distributional correctness, nor a global gradient-descent convergence rate for the LoRA-parameterized model. The rigorous statement is Theorem 1: if the learned residual transitions satisfy the integrated ratio-form DB relation together with the data-end boundary condition, then the terminal conditional marginal obeys pθ (x | y) ∝ ppre (x | y) exp(−E(x)) for the chosen terminal energy E. In practice, Lfwd , Lrev , and Lterm are the empirical surrogates used to encourage this condition.
23
D
Extended Derivations
D.1
Closed-Form Conditional Gibbs Projection
The main text uses a prompt-conditioned target, with fixed prompt population π(y). The optimization is therefore over conditional distributions p(· | y), while π(y) is held fixed. Proposition 4 (Conditional Gibbs projection for concept unlearning). Consider the constrained problem p∗C = arg min Ey∼π [KL(p(· | y) ∥ ppre (· | y))]
(49)
p
s.t.
Ey∼π, x∼p(·|y) [EC (x)] ≤ δC ,
(50)
over conditional distributions p(· | y) that are absolutely continuous with respect to ppre (· | y) for π-almost every y and have finite prompt-averaged energy. Assume the feasible set is nonempty and contains a strictly feasible conditional distribution. Then the minimizer is unique up to π-null prompt sets and has the form Z ppre (x | y) exp(−βEC (x)) ∗ pC (x | y) = , Zβ,C (y) := ppre (x | y) exp(−βEC (x)) dx, (51) Zβ,C (y) for some shared β ≥ 0 with Zβ,C (y) < ∞ for π-almost every y. Moreover, β satisfies the KKT conditions β ≥ 0, Ey∼π, x∼p∗C (·|y) [EC (x)] ≤ δC , β Ey∼π, x∼p∗C (·|y) [EC (x)] − δC = 0. (52) Proof. The feasible set is convex, and the prompt-averaged conditional KL is strictly convex in p(· | y) for π-almost every y, so any optimizer is unique up to π-null prompt sets. Because a strictly feasible point exists, Slater’s condition holds and the KKT conditions are sufficient. Form the Lagrangian with shared multiplier β ≥ 0 for the averaged forgetting constraint and a prompt-dependent normalization multiplier λ(y): Z p(x | y) L(p, β, λ(·)) = Ey∼π p(x | y) log dx ppre (x | y) Z + β Ey∼π p(x | y)EC (x) dx − δC Z + Ey∼π λ(y) p(x | y) dx − 1 . (53) Setting the first variation with respect to p(x | y) to zero for each prompt gives log
p(x | y) + 1 + βEC (x) + λ(y) = 0, ppre (x | y)
(54)
hence p(x | y) = ppre (x | y) exp(−1 − λ(y) − βEC (x)) . Normalizing for each prompt yields Z exp(1 + λ(y)) = ppre (x | y) exp(−βEC (x)) dx =: Zβ,C (y),
(55)
(56)
so p∗C (x | y) =
ppre (x | y) exp(−βEC (x)) . Zβ,C (y)
24
(57)
The remaining KKT conditions are precisely the shared-multiplier conditions for the prompt-averaged constraint: β ≥ 0, Ey∼π, x∼p∗C (·|y) [EC (x)] ≤ δC , β Ey∼π, x∼p∗C (·|y) [EC (x)] − δC = 0. (58)
Thus β is tied to the active constraint level in the stated constrained problem. When β is instead chosen directly as a hyperparameter, one is solving the relaxed penalized objective rather than fixing δ in advance. In the thresholded practical surrogate used in the main method, this scalar strength is eC through λscale , while the target remains prompt-conditioned through ppre (x | y) absorbed into E and Zβ,C (y). D.2
Gaussian Instantiation of ∇-DB
For Gaussian denoising transitions PFθ (zt−1 | zt , y) = N (µθ (zt , t, y), σt2 I) and PFpre (zt−1 | zt , y) = N (µpre (zt , t, y), σt2 I): zt−1 − µθ (zt , t, y) , σt2 zt−1 − µpre (zt , t, y) ∇zt−1 log PFpre = − . σt2 ∇zt−1 log PFθ = −
(59) (60)
In the DDPM framework with matched variances, the residual forward score simplifies to ∇zt−1 log P̃Fθ (zt−1 | zt , y) =
µθ (zt , t, y) − µpre (zt , t, y) . σt2
(61)
This is the finetuned-vs-pretrained model deviation divided by the noise variance. In our implementation, this is computed as (m_theta - m_ref) / sigma_sq_t. D.3
Non-Ideal Model Correction
The pretrained model PFpre was trained with a denoising objective rather than a GFlowNet objective, so it only approximately satisfies detailed balance. The residual term gϕ parameterizes this discrepancy within the continuation-score approximation. In our implementation, gϕ is prompt-independent; prompt dependence enters through the denoiser-based clean estimate x̂θ (zt , t, y). Let ∆t be the pretrained model’s approximation error at timestep t. The forward-looking parameterization becomes eC (x̂θ (zt , t, y)) · ᾱt + ∇ log F̃ (zt , y) = −∇zt E
gϕ (zt , t) | {z }
.
(62)
≈∆t +higher-order terms
The terminal loss ∥gϕ (z0 )∥2 ensures this correction vanishes at the data end, so the boundary flow is set by the chosen terminal energy rather than by an unconstrained correction term.
25
E
Additional Experimental Details
This section reports the additional implementation details required to reproduce the experiments in Section 5: the encoders and base diffusion model, the prompt sets used for forgetting and benign retention, the compute budget, and the practical guidance for tuning the small set of method-specific hyperparameters. E.1
CLIP Model
We use openai/clip-vit-large-patch14 (ViT-L/14, 224 × 224 input, output dimension 768) for both the forget-energy computation and the evaluation metrics. A single CLIP model is shared across (i) the descriptor set QC used to define the forget energy, and (ii) the evaluation similarity scores reported in the main text. Sharing the encoder ensures that the forget signal during training and the evaluation metric at test time are measured in the same embedding space; a different CLIP variant could be substituted, with the threshold τ recalibrated, without changing the rest of the pipeline. E.2
Base Diffusion Model and Adapters
All experiments adapt Stable Diffusion v1.5 (runwayml/stable-diffusion-v1-5). LoRA adapters are inserted into the U-Net cross- and self-attention projections (WQ , WK , WV , WO ) at rank r = 8, with the standard α = r scaling. Only the LoRA weights ∆θ and the auxiliary residual parameterization gϕ (channels {64, 128, 256, 256}, time-embedded) receive gradient updates; the U-Net backbone, latent autoencoder, text encoder, and CLIP model are all frozen. We use 50-step DDPM sampling at training and inference time. E.3
Prompt Sets and Generation Strategy
Why LLM-based prompt generation. Prompt construction for concept unlearning is a wellestablished design choice in the literature, and existing works fall broadly into two camps: templatebased prompts (a small set of fixed scaffolds with concept-name slots) and LLM-based prompts (generated by a language model under task-specific instructions). Both are used widely; we adopt the LLM-based route because it produces a more varied, natural, and semantically diverse prompt set than templates. An LLM can compose grammatically natural phrasings, vary scene context, framing, lighting, and activity, and cover a much broader region of plausible prompt space without manual enumeration. This matters for unlearning evaluation because narrow prompt templates can mask retention failures that only surface under naturalistic prompt distributions. We use GPT-4o-mini, which provides this diversity at a low API cost while still respecting the formatting constraints required for clean diffusion prompts. Prompt generation strategy. Prompts are generated under carefully designed system prompts that enforce three properties: (i) visual grounding—the concept must be a clearly visible, concrete subject that the diffusion model can depict in-frame, with the physical sense chosen for polysemous concepts (e.g., “apple” = fruit, not the company); (ii) scene and category diversity—prompts vary across indoor/outdoor settings, day/night, weather, activities, solitary vs. small-group scenes, and near vs. far framings, with no near-duplicate phrasings; and (iii) lexical simplicity—4–10 words, present tense, ≤ 1 adjective, no special tags, parameters, or stylistic suffixes. The same pipeline is reused for forget and retain prompts, with minor wording changes in the system prompt to switch the target subject and to exclude the unlearning concept token from the retain prompts. Forget Prompt Generation – System Prompt You are a dataset prompt writer for diffusion models (e.g., Stable Diffusion). Your job is to produce short, simple, natural-language prompts that depict a clearly visible instance of the specified concept in-frame. GENERAL STYLE: 4–10 words per prompt; one simple main clause; present tense preferred; everyday vocabulary only; 0–1 adjectives; no tags/parameters (no colons, aspect ratios, seeds, CFG, hashtags); minimal punctuation.
26
VISUAL GROUNDING (MANDATORY): The concept MUST be a visible, concrete subject in the image. If the concept is polysemous, use the physical object sense. The scene must make the concept clearly depictable and in-frame. Match number and determiners consistently (a/an/one/two/many). DIVERSITY REQUIREMENTS: Across prompts, balance indoor vs. outdoor, day vs. night, weather conditions, everyday activities, solitary vs. small-group scenes, and near vs. far framing. Avoid near-duplicates and repeated sentence structure.
Retain prompt generation. For the optional retain-prompt augmentation reported in Section 5.3 (Q4), we use a two-stage pipeline. We first ask the LLM to enumerate ∼50 candidate related concepts for each unlearning target, covering close siblings in the same category, broader category/hypernym terms, contextually co-occurring concepts, and analogous concepts from other domains, while explicitly excluding synonyms, subtypes, and parts of the target. From this pool we narrow to a smaller neighbor set using CLIP text–text similarity (or denoiser-based score-prediction distance as an alternative). For each selected neighbor, the LLM generates 10 diverse scene prompts under a replay-style system prompt that constrains the output to exclude the unlearning concept token and any obvious synonym. Retain Prompt Generation – System Prompt You are a prompt generator for Stable Diffusion models. INPUT: UNLEARN_CONCEPT (the concept being unlearned; must NOT appear in any prompt) and RELATED_CONCEPT (the target subject for retention). CONSTRAINTS: 5–10 words per prompt; present tense; simple, everyday vocabulary; ≤ 1 comma and ≤ 1 adjective per prompt; no special tags or parameters; do not include UNLEARN_CONCEPT or any obvious synonym/nickname. CONTENT: Describe simple, plausible scenes involving RELATED_CONCEPT. Ensure variety: mix indoor/outdoor settings, day/night, weather, activities, solitary vs. group, near vs. far framing. Do not repeat sentence structure; avoid near-duplicates. Keep all prompts safe and neutral in tone. OUTPUT: Exactly 10 prompts, one per line. No numbering, bullet points, section titles, or concept labels.
Prompt sets used in experiments. We use four prompt sets per concept: a forget set used during training, a held-out forget set used at evaluation, a benign retention set covering nearby but distinct concepts, and a general benign set covering unrelated content. Sizes and representative phrasings are summarized below. • Van Gogh forget (training): 50 prompts of the form “a painting in the style of Van Gogh”, “Van Gogh starry night”, “Van Gogh self-portrait”, etc. These are the prompts on which residual ∇-DB losses are computed. • Van Gogh forget (held-out): a disjoint set of 50 forget prompts used only for evaluation, ensuring that the reported forget rate is not driven by training-prompt memorization. • Van Gogh benign: 200 prompts spanning landscapes, portraits, animals, and objects in nonVan Gogh styles, used to measure benign-concept retention. • Pikachu forget (training/held-out): 50 + 50 prompts of the form “pikachu”, “a picture of pikachu”, “pikachu in a forest”, etc. • Pikachu benign: 200 prompts for non-Pikachu characters and general objects. The same retention prompts are used across methods, so retention numbers are directly comparable. E.4
Optimization and Schedule
LoRA parameters and ϕ are jointly optimized with AdamW at learning rate 1×10−4 , (β1 , β2 ) = (0.9, 0.999), weight decay 10−2 , and gradient clipping at norm 1.0. We use a batch size of 4 forget 27
prompts, with one denoising trajectory per prompt and |T | = 0.1T = 5 score-matched timesteps per trajectory. Training runs for 1,000 batches per concept, which is sufficient for convergence in all experiments. The loss weights in Eq. (9) are λrev = 1.0 and λterm = 0.1. E.5
Compute
All experiments are run on a single NVIDIA A100 (80 GB). End-to-end training takes 3–5 hours per concept, dominated by the trajectory rollout (Phase 1 of Algorithm 1) and the Jacobian-vector product in the reverse residual score (Phase 3). Inference uses standard 50-step DDPM sampling and incurs no additional cost beyond the base diffusion model – the LoRA correction is folded into the U-Net forward pass. E.6
Hyperparameter Sensitivity
The method exposes three concept-level hyperparameters: the energy scale λscale , the slope α inside the exponential, and the CLIP-similarity threshold τ . Their roles, recommended ranges, and tuning strategy are as follows. • λscale (tilt magnitude). Most sensitive overall: it controls the magnitude of the terminal energy and hence the strength of the unlearning tilt. Too small leaves the concept partially intact; too large destabilizes training. We sweep λscale ∈ {0.5, 1, 2, 5} and pick the smallest value that achieves a stable forget rate. • τ (decision boundary). Defines the no-tilt benign region. Set from the empirical CLIP-similarity distribution of the pretrained model on benign prompts: typically the 90–95 percentile of benign similarities, so that benign images receive zero terminal-energy gradient. In our experiments, we keep τ fixed across concepts within each category, using τ = 0.21 for Objects, τ = 0.22 for Characters, and τ = 0.20 for Style. • α (slope). Controls how sharply the energy grows above τ . Less sensitive than λscale and τ ; we use α = 10 throughout. Recommended tuning order. (1) Estimate τ from the benign-prompt CLIP-similarity histogram. (2) Tune λscale for stable training and adequate forget rate. (3) If forget images are slipping just above τ without being pushed away, increase α before increasing λscale further. E.7
Evaluation Protocol
This section expands the evaluation summary in Section 5.2 with the full automated evaluation methodology, dataset construction, and metric definitions used to produce the main quantitative results. VLM-based automated evaluation. To automate and standardize the assessment of unlearning and retention, we use Qwen2.5-VL-7B-Instruct [2] as an impartial evaluator. Modern VLMs are well suited to this role: they are explicitly trained for open-vocabulary visual question answering, and for binary, closed-ended questions of the form “Does this image contain ⟨concept⟩?” they produce robust judgments across a much wider semantic range than dedicated classifiers. For each accuracy metric, we generate a set of images per prompt and aggregate the VLM’s per-image yes/no answers into the reported score. Why not classifiers. We deliberately avoid fixed-vocabulary classifiers (e.g., ImageNet-1k pretrained models) and the category-specific heads provided by benchmarks such as UnlearnCanvas [39] as primary evaluators, for three reasons: • Restricted concept space. An ImageNet-trained classifier can only score concepts inside its 1000-class vocabulary, so it cannot evaluate styles (e.g., Van Gogh, Monet, cartoon), specific identities (Brad Pitt, Lionel Messi), fictional characters (Pikachu, Mickey Mouse), or many fine-grained variations that real unlearning evaluations need. Restricting the evaluator restricts
28
the experimental setup to whatever the classifier happens to know, which is exactly the wrong constraint for studying broad concept unlearning. • UnlearnCanvas-specific limitations. The classifiers shipped with UnlearnCanvas similarly cover mostly objects and styles from its own dataset, and the artistic styles in UnlearnCanvas are visually quite different from the canonical referent of those style names (e.g., the dataset’s “Van Gogh”-tagged images differ in color palette, brushwork, and composition from the historical paintings); we observed that the provided style classifier does not generalize well to images that follow the canonical style rather than the UnlearnCanvas style. This makes the classifier’s verdict unreliable as a stand-alone concept-unlearning metric. • Poor performance on AI-generated images. Standard classifiers are trained on natural-image distributions; they degrade noticeably on diffusion-generated images, where backgrounds, lighting, and scene composition can be unnaturally smooth, stylized, or artifact-prone. These shifts hurt classifier accuracy even when the depicted concept is clearly recognizable to a human or VLM, biasing the resulting unlearning/retention numbers in opaque ways. We empirically verified these failure modes on out-of-distribution and AI-generated images from the original Stable Diffusion model: conventional classifiers produced inconsistent and often incorrect labels, whereas Qwen2.5-VL produced substantially more accurate yes/no judgments aligned with manual inspection. Using a VLM also lifts the cap on which concepts we can evaluate—styles, characters, identities, and arbitrary objects can all be queried with the same evaluator—so the experimental setup is no longer bounded by a particular classifier’s vocabulary. CLIP score as a complementary metric. Alongside VLM-based accuracy, we report CLIP score (Uclip , RRclip , GRclip ) for every prompt set. CLIP score is the standard text–image alignment metric in the diffusion-model unlearning literature and is reported by essentially all existing baselines, so including it ensures our evaluation is directly comparable to prior work and not solely dependent on the VLM judgment. In our setup, the VLM-based accuracy captures whether the concept is present or absent (the discrete “did it forget?” question), while CLIP score captures graded text–image alignment on the benign part of the prompt. The two metrics are therefore complementary: a high VLM accuracy with a high CLIP score indicates that the target concept is removed without damaging the surrounding scene description. Evaluation datasets and scale. We construct three disjoint evaluation prompt sets per checkpoint: the forget evaluation set Deval_forget , the related-retain set Deval_related , and the general-retain set Deval_general . All evaluation prompts are held out from the training prompts to test generalization rather than memorization. We track approximately 60 distinct concepts (spanning unlearning targets, related concepts, and general themes), with 20 unique evaluation prompts per concept and 8 images per prompt, yielding approximately 10,000 images per model checkpoint. The general-retain set is fixed and unrelated to any unlearning target, providing a control group for measuring catastrophic forgetting on the model’s base capabilities. Related concept identification. To identify semantically entangled concepts for the related-retain set, we follow the methodology of EraseBench [1]. We use GPT-4o-mini with the system prompt: “Your main task is to help identify concepts for evaluating text-to-image models. The key idea is to identify 3–4 concepts that are semantically entangled with the Given Concept. . . ”. This yields adjacent concepts (e.g., Pikachu → Squirtle, Charmander; Van Gogh → Impressionism) that are used to measure the ripple effect [1, 39]. Metric definitions.
For each concept, we report:
• Unlearning accuracy (Uacc ) and CLIP score (Uclip ). Uacc measures erasure efficacy; high Uacc indicates successful unlearning. Uclip serves as a proxy for “in-prompt retainability” [31], ensuring that while the target concept is removed, the model still respects the benign context of the prompt. The ideal outcome is a high Uacc (concept removed) paired with a high Uclip (context preserved).
29
• Related retention (RRacc ) and CLIP score (RRclip ). To quantify the ripple effect, we evaluate performance on the semantically entangled concepts identified above. A high RRacc indicates surgical unlearning with minimal collateral damage. RRclip ensures that visual quality and text alignment for these neighboring concepts remain intact. • General retention (GRacc ) and CLIP score (GRclip ). GRacc assesses stability on a broad set of unrelated concepts, detecting catastrophic forgetting; GRclip verifies that general instructionfollowing and text-to-image alignment remain intact after unlearning. Generative quality. Beyond accuracy, we evaluate the fundamental generative quality of the unlearned model following standard practice. We adopt the Fréchet Inception Distance (FID) [16], computed using the evaluation script and methodology used in the UnlearnCanvas benchmark [39]. To measure preservation of distributional quality, we compute FID between images generated by the unlearned model and images generated by the original Stable Diffusion model on the same evaluation prompts, treating the pretrained-model distribution as the reference for benign generation. For all CLIP-score calculations (Uclip , RRclip , GRclip ), we use openai/clip-vit-large-patch14, ensuring alignment with standard evaluation protocols. FADE computation and retain-only reference. FADE [6] is computed as defined in Eq. (10); for diffusion models it is tractably estimated via weighted differences of denoising MSE losses between the unlearned model and a retain-only reference. Constructing a meaningful retain-only reference is the main practical challenge of FADE evaluation, since it requires a model that has never seen the target concept yet otherwise matches the pretrained-and-finetuned distribution. We follow the UnlearnCanvas [39] protocol to obtain such a reference. The pipeline has two stages: 1. Pretrained-and-finetuned model (used as the unlearning starting point). Starting from Stable Diffusion v1.5, we finetune on the full UnlearnCanvas dataset, which covers all styles. Concept unlearning is performed on top of this checkpoint, so all forgetting and retention metrics are measured relative to a model that has been explicitly trained on the target style. 2. Retain-only reference for FADE. For each target style Cstyle , we separately finetune a fresh Stable Diffusion v1.5 on UnlearnCanvas with Cstyle removed (i.e., on UnlearnCanvas\Cstyle ). This produces a model whose training distribution matches the pretrained-and-finetuned model in everything except the omitted style, exactly the gold-standard retain-only reference required by FADE. We carry out this construction for three artistic-style concepts that are well-supported by the UnlearnCanvas dataset—Van Gogh style, Picasso style, and Monet style—and compute the per-style FADE between the unlearned model and the corresponding retain-only reference. The Avg. FADE value reported in Table 1 is the mean of these three per-style FADE scores. This averaging matches the granularity at which the retain-only references are practical to train, while still providing a principled distributional comparison against a genuine concept-free baseline rather than only the pretrained model.
30
F
Performance Across Multiple Diffusion Architectures
We additionally evaluate Pikachu concept unlearning across multiple base diffusion architectures using the same evaluation protocol. The forget-prompt classifier accuracy column is reported directly, so lower values indicate stronger erasure; retain accuracies and CLIP scores remain higher-is-better. Table 4: Pikachu unlearning performance across base diffusion architectures.
Base model FAcc ↓ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑ SD v1.4 SD v1.5 SD v2 SD3
0.01 0.00 0.01 0.01
28.0 29.9 29.9 29.8
0.75 0.76 0.73 0.73
31
30.9 33.0 32.9 32.9
0.84 0.88 0.83 0.86
31.4 32.1 32.0 32.2
G
Per-Concept Quantitative Results
This section provides the full per-concept breakdown of the quantitative results summarised in Table 1. Each table reports results for a single unlearning target across all methods, including unlearning accuracy (UAcc ↑), related-retain accuracy (RRAcc ↑), general-retain accuracy (GRAcc ↑), and the corresponding CLIP scores (↑). For methods where no model targeting the concept directly was available, the entry is omitted (–). Table 5: Concept unlearning results for Apple.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.88 0.82 0.75 0.69 0.31 0.98 0.76 0.98 0.76 0.51
28.7 29.4 29.4 29.6 31.4 29.4 30.2 25.0 25.0 29.9
0.66 0.69 0.91 0.97 0.84 0.85 0.72 0.42 0.56 0.82
29.4 29.5 29.9 30.7 29.6 30.7 29.1 26.6 26.1 30.9
0.78 0.80 0.81 0.90 0.80 0.89 0.82 0.72 0.53 0.88
31.7 31.9 31.3 32.8 31.9 32.7 32.3 30.6 27.8 32.5
TILDE
0.97
26.8
0.75
28.4
0.85
31.9
Table 6: Concept unlearning results for Banana.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.99 0.97 0.93 0.91 0.69 1.00 0.85 0.92 0.82 0.66
27.3 27.4 26.7 27.3 26.8 29.4 29.1 26.1 22.9 29.3
0.41 0.55 0.88 0.94 0.72 0.85 0.72 0.33 0.48 0.89
29.5 30.6 31.0 30.9 29.7 32.2 30.3 27.8 25.9 32.1
0.76 0.80 0.85 0.90 0.81 0.88 0.83 0.73 0.49 0.88
31.5 31.9 31.8 32.6 31.8 32.7 32.2 30.6 27.4 32.5
TILDE
0.98
22.6
0.78
29.4
0.86
31.9
32
Table 7: Concept unlearning results for Brad Pitt.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.96 0.94 0.82 0.99 1.00 1.00 0.98 0.98 0.73 0.39
28.3 28.3 32.2 26.9 17.2 26.2 25.5 25.5 26.1 32.4
0.41 0.47 0.88 0.70 0.00 0.28 0.06 0.12 0.50 0.88
30.2 31.3 34.6 33.7 16.6 29.2 25.1 26.9 27.6 34.0
0.79 0.77 0.87 0.88 0.02 0.86 0.58 0.75 0.43 0.89
31.4 31.1 32.5 32.4 19.5 32.4 28.9 31.0 26.3 32.4
TILDE
0.98
27.2
0.55
30.3
0.88
32.2
Table 8: Concept unlearning results for Cartoon Style.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.70 0.89 0.24 0.97 1.00 0.23 0.89 0.93 0.97 0.49
28.5 28.4 32.1 28.4 21.4 29.4 26.1 25.6 23.9 29.6
0.80 0.80 0.94 0.79 0.00 0.78 0.35 0.13 0.26 0.88
30.1 30.2 31.8 30.5 20.9 29.8 25.8 22.8 25.8 31.2
0.83 0.84 0.89 0.87 0.02 0.88 0.71 0.48 0.48 0.87
32.1 32.1 32.8 32.5 19.3 32.5 30.1 28.2 27.3 32.5
TILDE
0.82
28.9
0.80
30.4
0.82
31.8
33
Table 9: Concept unlearning results for Cat.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.49 0.61 0.27 0.33 0.12 0.96 0.94 0.95 0.88 0.18
28.7 27.8 31.2 31.3 31.2 27.9 25.4 24.4 18.8 30.9
0.77 0.77 0.99 0.99 0.95 0.85 0.27 0.46 0.11 0.98
31.0 30.6 32.6 32.6 32.2 32.1 27.0 27.5 21.1 32.4
0.80 0.77 0.88 0.89 0.85 0.87 0.53 0.72 0.16 0.87
31.8 31.3 32.6 32.9 32.4 32.6 28.7 30.3 22.8 32.5
TILDE
0.96
26.3
0.82
30.9
0.87
32.1
Table 10: Concept unlearning results for Dog.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.39 0.46 0.84 0.23 0.21 0.81 0.61 0.90 0.94 0.12
28.9 27.8 25.9 30.8 30.0 27.5 27.6 25.3 21.5 29.6
0.81 0.77 0.96 1.00 0.94 0.76 0.60 0.52 0.26 0.95
31.7 31.0 30.2 33.2 32.8 31.6 29.8 28.0 22.1 32.6
0.82 0.79 0.73 0.91 0.86 0.87 0.68 0.75 0.21 0.88
31.8 31.3 29.5 32.8 32.4 32.5 30.4 30.4 23.7 32.4
TILDE
0.94
21.8
0.59
27.4
0.80
31.2
34
Table 11: Concept unlearning results for Golf Ball.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.97 1.00 0.30 0.92 0.85 0.77 0.96 1.00 0.83 0.48
27.5 24.4 32.9 31.7 29.4 31.4 29.1 24.3 23.5 30.1
0.26 0.13 0.65 0.66 0.48 0.48 0.33 0.09 0.09 0.53
28.3 26.5 30.5 30.4 29.7 30.4 28.9 25.4 23.3 29.8
0.79 0.75 0.88 0.89 0.83 0.89 0.75 0.70 0.35 0.89
31.5 30.9 32.6 32.8 32.2 32.7 31.1 30.2 25.6 32.6
TILDE
0.99
23.4
0.67
28.4
0.88
31.9
Table 12: Concept unlearning results for Lionel Messi.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
1.00 0.99 1.00 0.95 1.00 1.00 0.99 1.00 0.94 0.43
25.9 24.9 22.5 26.8 16.5 27.0 21.3 23.9 19.4 30.0
0.31 0.26 0.25 0.61 0.00 0.50 0.08 0.19 0.12 0.72
30.4 28.7 26.0 31.7 17.2 31.7 24.3 28.3 21.5 32.2
0.77 0.70 0.49 0.87 0.01 0.87 0.45 0.74 0.26 0.89
31.7 30.6 26.7 32.2 19.4 32.6 26.9 31.1 23.5 32.5
TILDE
0.99
25.8
0.69
32.3
0.88
32.5
35
Table 13: Concept unlearning results for Mickey Mouse.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.97 0.99 0.87 0.96 1.00 0.93 1.00 0.99 0.88 0.53
28.0 26.7 30.3 29.6 20.8 29.8 25.0 26.3 22.1 30.7
0.21 0.17 0.46 0.32 0.00 0.43 0.09 0.07 0.35 0.75
30.7 29.8 32.0 32.4 17.0 32.4 27.5 27.9 25.5 34.1
0.75 0.72 0.85 0.87 0.02 0.85 0.60 0.67 0.33 0.87
31.5 30.9 31.6 32.6 19.5 32.4 29.3 30.4 24.7 32.3
TILDE
0.97
28.7
0.56
31.8
0.85
31.9
Table 14: Concept unlearning results for Pikachu.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.96 1.00 1.00 0.97 0.96 0.96 0.97 0.97 0.83 0.72
27.9 0.0 21.8 27.9 27.9 30.9 26.6 26.2 23.6 30.8
0.50 0.00 0.68 0.74 0.66 0.63 0.42 0.45 0.38 0.78
30.8 0.0 27.0 33.8 33.6 33.2 29.7 29.2 25.1 34.2
0.75 0.00 0.47 0.87 0.85 0.85 0.64 0.69 0.27 0.87
31.5 0.0 26.1 32.4 32.3 32.3 30.2 30.0 24.3 32.2
TILDE
1.00
29.9
0.76
33.0
0.88
32.1
36
Table 15: Concept unlearning results for Van Gogh Style.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.76 0.81 0.49 0.95 1.00 0.68 0.84 0.43 0.68 0.29
30.1 29.4 32.5 27.6 17.9 21.8 29.2 20.2 20.9 31.7
0.78 0.75 0.91 0.81 0.00 0.57 0.61 0.43 0.00 0.81
32.9 32.2 33.6 32.5 18.1 29.5 30.9 24.2 21.4 31.0
0.84 0.81 0.89 0.85 0.00 0.85 0.71 0.49 0.04 0.88
32.1 31.7 32.6 32.4 20.7 32.1 30.4 25.8 19.7 31.6
TILDE
0.95
27.1
0.75
31.5
0.86
31.4
Table 16: Concept unlearning results for Monet Style.
Method
UAcc ↑ UCLIP ↑ RRAcc ↑ RRCLIP ↑ GRAcc ↑ GRCLIP ↑
ESD-u ESD-x UCE CA MACE DUO EraseFlow Meta SHS EDiff
0.17 0.41 0.12 0.66 1.00 0.77 0.84 0.71 0.76 0.18
30.3 29.5 32.0 28.3 20.4 25.5 28.3 24.5 22.9 32.5
0.87 0.57 0.93 0.56 0.07 0.12 0.60 0.26 0.26 0.85
31.1 31.0 33.5 31.2 20.9 26.7 29.4 26.0 24.8 33.1
0.84 0.81 0.82 0.88 0.01 0.88 0.73 0.69 0.36 0.87
31.1 31.6 32.0 32.4 19.4 32.5 30.2 29.9 25.6 32.4
TILDE
0.87
27.3
0.74
30.9
0.83
31.5
37