Concept Acupuncturist: An Efficient and Precise Concept Erasure Method for Diffusion Models via Sparse Autoencoders

Concept Acupuncturist: An Efficient and Precise Concept Erasure Method for Diffusion Models via Sparse Autoencoders Yiran Xiahou∗ , Siliang Qin∗ , Kai Chen∗ , Shengzhi Zhang† , Renyang Liu‡ ∗ Institute of Information Engineering, Chinese Academy of Sciences, China † Department of Computer Science, Metropolitan College, Boston University, USA

arXiv:2610.32059v1 [cs.CV] 23 Oct 2026

‡ Institute of Data Science, NationalUniversity of SingaporeSingapore, Singapore

However, retraining on filtered datasets is often computationally expensive [12], [13], and post-processing remains vulnerable to prompt manipulation [14], [15]. Approaches for direct model modification, particularly machine unlearning (MU), offer a more direct and controllable way to remove specific concepts without full retraining [16], [17]. Existing MU methods can be broadly divided into three categories: inference-time intervention, model editing and fine-tuning-based unlearning. Inference-time approaches [18], [19], [20], [21], [22], [23], [24] typically suppress undesirable concepts by attaching external, removable modules during inference, which modify model outputs without altering model parameters and thus leave concept-related representations encoded within the model. Such interventions may introduce artifacts and degrade generation quality [25], while their inference overhead becomes non-negligible under large-scale deployment. Model editing methods [26], [27], [28] directly modify model parameters through closed-form or localized updates to remove target concepts. However, the highly distributed and entangled representations in diffusion models [1], [2], [3], [5] make precise parameter localization difficult. More fundamentally, existing methods focus on finding low-interference update directions rather than identifying the parameters or internal features that encode the target concept [28]. Fine-tuning-based MU methods, in contrast, directly modify model parameters without additional inference overhead, making them widely adopted for safer image generation. However, Warning: This paper includes images that may contain non-explicit nudity and may be sensitive to some readers. they still face two fundamental challenges: (i) limited edit precision, since global or layer-wise updates often interfere with unrelated concepts due to the polysemantic nature of I. I NTRODUCTION neurons [29], [30], [31], making it difficult to precisely Text-to-image diffusion models [1], [2], [3], [4], [5] have remove a target concept without affecting others; and (ii) high demonstrated strong capabilities in image synthesis [6], [2] computational cost, as they often update a large portion of and editing [7], [8]. Training these models on large-scale web model parameters (e.g., all cross-attention layers or global datasets improves performance, but also exposes them to diverse weights [9], [32]) and require executing the denoising process and weakly curated content. Consequently, such models may up to an intermediate timestep to obtain supervision [9] due to produce harmful or policy-violating content, e.g., Not-Safe- the iterative nature of diffusion models, thereby incurring fullFor-Work (NSFW) imagery [9], [10] or infringing outputs trajectory cost even for local updates and leading to substantial that replicate protected identities or artistic styles without training overhead, especially when scaling to multiple concepts authorization [11]. These concerns motivate the development or iterative updates. of various strategies for mitigating unsafe content, including To overcome these limitations, we propose Concept Acupuncdataset curation, post-processing and direct model modification. turist (C ONCEPTACP), a structured fine-tuning framework that

Abstract—Diffusion-based image generation models produce high-quality images but may generate harmful or policy-violating content, motivating machine unlearning (MU) to selectively remove target concepts while preserving general capabilities. However, existing MU approaches often struggle with the distributed and entangled nature of concept representations, leading to inefficient optimization and imprecise edits when applied through large-scale fine-tuning over full diffusion trajectories. To bridge this gap, we propose Concept Acupuncturist (C ONCEPTACP), a novel MU framework that integrates interpretability-guided analysis with inference-free fine-tuning for efficient and precise concept removal. By establishing mappings between semantic concepts and crossattention blocks in the U-Net denoiser and localizing the most concept-relevant block for targeted optimization, C ONCEPTACP avoids full diffusion trajectory optimization, thereby improving fine-tuning efficiency. To mitigate concept entanglement, we propose a feature-level correction mechanism based on Sparse Autoencoders (SAEs), where SAE-derived sparse, semantically meaningful feature activations are manipulated to construct targeted correction signals, which are then projected back to the output space of the corresponding block and used as supervision for precise concept removal. Consequently, C ONCEPTACP eliminates the need for external inference-time modules while achieving efficient and fine-grained unlearning. Instantiated on SDXL-Turbo, C ONCEPTACP improves the forgetting success rate by 36.34 percentage points in the standard one-step generation setting on average over the best-performing baseline, while preserving image quality and prompt fidelity, and generalizes effectively across other Stable Diffusion backbones. Our code is available at https://github.com/Eran-Xia/ConceptACP.

enables inference-free and localized updates for parameter vation of unrelated concepts, improving forgetting success rate optimization. Inspired by recent interpretability studies showing by 36.34 percentage points on average under the standard that concept-related information can be decomposed into one-step setting over the best-performing baseline. We further structured and semantically meaningful components [33], observe consistent gains on SDXL and SD v1.4, demonstrating we introduce Sparse Autoencoders (SAEs), an interpretable transferability across backbones. In terms of efficiency, our tool that decomposes neuron activations into sparse feature block-level, inference-free fine-tuning reduces computational representations within the U-Net denoiser, to enable fine- cost by 46.4% compared to full diffusion-based method without grained concept removal. We further reformulate diffusion fine- this design. In addition, we introduce two metrics, Feature tuning as a localized block-level learning problem, avoiding Overlap Score and Feature Activation Correlation Score, full denoising trajectories during optimization. Nevertheless, to characterize internal feature dynamics before and after realizing this design presents two technical challenges: unlearning. Results show that our method induces featureC1: difficulty in internalizing feature-level interventions level changes consistent with the intended concept suppression into model parameters. Existing SAE-based methods typically and non-target preservation, providing internal evidence for its treat SAEs as inference-time intervention modules [22], [23]. precise unlearning. Additional results demonstrate preserved A natural question is therefore whether the benefits of such image quality and robustness against adversarial prompts. feature-level editing can be directly embedded into model Our main contributions are outlined below: parameters. However, SAEs operate on the residual updates of • We propose C ONCEPTACP, a novel unlearning framework corresponding blocks and require explicit feature manipulation that integrates interpretability-guided feature analysis into to achieve forgetting, making their direct translation into parameter-level fine-tuning for diffusion models. To the parameter updates non-trivial. best of our knowledge, this is the first work to use SAEC2: difficulty in identifying the target block and conderived feature corrections as supervision for localized and structing reliable supervision signals. Concept representations internalized fine-tuning in diffusion model unlearning. in diffusion models are distributed across multiple cross- • We introduce a structured fine-tuning paradigm that jointly attention blocks and evolve over the denoising trajectory, improves edit precision and efficiency. For precision, we use making it challenging to identify an appropriate block and SAE-derived feature corrections as fine-grained supervision, derive consistent supervision for localized fine-tuning. while for efficiency, we construct block-level input–output To address C1, we move beyond treating SAEs as external supervision to enable inference-free fine-tuning of a single modules but as a mechanism for constructing supervision cross-attention block. signals. Given prompts containing the target concept, we • Extensive experiments show that C ONCEPTACP effectively identify concept-relevant features and suppress them according removes target concepts while preserving model utility and to their relevance scores via a feature selection procedure. The substantially reducing fine-tuning computational cost. modified features are then projected back to the output space of the corresponding cross-attention block. This process yields II. BACKGROUND AND R ELATED W ORK target outputs that are equivalent to performing feature-level editing, but expressed in the model’s native representation space. A. Text-to-Image Models Diffusion models [1], [2], [3], [4] synthesize realistic images By pairing these corrected outputs with the original inputs, we construct a supervision dataset that enables internalization of by gradually denoising random noise through a learned reverse diffusion process. These models typically consist of three key feature-level interventions into model parameters. To address C2, we reformulate diffusion fine-tuning as components: a text encoder that converts input prompts into a localized block-level learning problem. Specifically, we semantic embeddings, a U-Net denoiser that conditions on the establish mappings between semantic concepts and cross- text embedding and iteratively refines a latent representation, attention blocks using a pool of pretrained SAEs, building upon and a decoder that maps the latent representation back to the prior analysis [33]. Based on the mappings, we identify one image space. Among these components, the U-Net serves as block most strongly associated with the target concept together the core of the denoising process, comprising downsampling with its relevant features and then construct a block-level input– and upsampling pathways connected through residual blocks, output supervision dataset from a small set of text prompts often interleaved with attention or transformer modules. at selected timesteps. The target outputs are obtained via the Traditional Stable Diffusion models, such as the 1.x and feature-level correction process described above. Importantly, 2.x versions, typically require tens of denoising steps (e.g., we collect the block’s activations across timesteps in a single 50) to generate high-quality images. To improve image fidelity forward pass, avoiding repeated denoising during fine-tuning and prompt adherence, Stable Diffusion XL (SDXL) [5] was while retaining further flexible feature manipulation through introduced. SDXL-Turbo [35], on the other hand, is a distilled the cached activations. variant of SDXL, trained with techniques such as adversarial We implement C ONCEPTACP on SDXL-Turbo and evaluate and progressive distillation to accelerate generation. It typically it on the EvalIGMU [34] benchmark across four representative requires only 1–4 steps to generate an image while maintaining concepts. Experiments show that C ONCEPTACP achieves competitive visual quality. Architecturally, the U-Net in SDXLstrong balance between forgetting effectiveness and preser- Turbo comprises a downsampling path, a bottleneck section and

2

an upsampling path with transformer blocks to capture global context. With growing demand for real-time and interactive text-to-image generation, SDXL-Turbo has gained significant attention for its fast inference.

To formalize this, Surkov et al. [33] take the residual updates in each cross-attention block of a diffusion model as input to SAEs to analyze the role of each block in image generation. For the 𝑙-th transformer block of the U-Net in diffusion model, the input-output relationship can be expressed as:

B. Machine Unlearning

𝑖𝑛 𝑖𝑛 𝐷 [𝑙] 𝑖𝑜𝑢𝑡 (1) 𝑗 = 𝐷 [𝑙] 𝑖 𝑗 + 𝑇 [𝑙] (𝐷 [𝑙] , 𝑒)𝑖 𝑗 , MU aims to remove the influence of specific subsets of data 𝑖𝑛 𝑜𝑢𝑡 ∈ R ℎ×𝑤×𝑑 denote the residual activaor learned knowledge from a trained model [36], [37]. Typically, where 𝐷 [𝑙] , 𝐷 [𝑙] existing methods for T2I models either erase individual samples tions before and after the block, with ℎ and 𝑤 representing the or suppress undesired concepts. The latter, which has received spatial dimensions (height and width) of the residual update. substantial attention in recent studies, aims to significantly 𝑇 [𝑙] denotes the function computed by the block given the suppress the T2I model’s ability to generate particular undesired text embedding e. Let the encoder and decoder functions of the SAE be: concepts while preserving its performance on unrelated content. The unlearning targets explored in prior research include: 𝑓 (𝑥) = 𝐸 𝑁𝐶 (𝑥) = 𝜎(𝑊 𝐸 𝑁 𝐶 (𝑥 − 𝑏 𝑝𝑟 𝑒 ) + 𝑏 𝑎𝑐𝑡 ), (2) (1) semantics (e.g., nudity, violence, and gore), (2) stylistic 𝑥ˆ = 𝐷𝐸𝐶 ( 𝑓 (𝑥)) = 𝑊 𝐷𝐸𝐶 𝑓 (𝑥) + 𝑏 𝑝𝑟 𝑒 , (3) attributes (e.g., Van Gogh), and (3) objects (e.g., church and parachute). Existing MU approaches can be broadly categorized where 𝑏 𝑝𝑟 𝑒 and 𝑏 𝑎𝑐𝑡 are learnable bias terms. During the into three strategies: training of SAE on this specific cross-attention block, the input Model Fine-tuning. Fine-tuning-based methods achieve un- is set to the residual update: learning by updating model parameters to erase target con𝑖𝑛 Δ𝐷 [𝑙] 𝑖 𝑗 = 𝐷 [𝑙] 𝑖𝑜𝑢𝑡 (4) cepts. For example, ESD [9] shifts predictions from concept𝑗 − 𝐷 [𝑙] 𝑖 𝑗 . conditioned to unconditional outputs, AC [38] align distribuThis training strategy is applicable to both models with few tions for concept removal, SA [39] replaces forgotten concepts inference steps, such as SDXL-Turbo, and models that require with alternative distributions, attention redirection [11] finemulti-step denoising, such as SD v1.4, by collecting residual tunes cross-attention layers, MACE [40] combines attention updates across multiple timesteps [22], [41]. editing with LoRA for multi-concept erasure, and TRACE [32] Encoding this residual via the SAE yields constrains edits to later denoising stages. Model Editing. Model editing-based approaches aim to remove 𝑆𝑖 𝑗 = ENC(Δ𝐷 [𝑙] 𝑖 𝑗 ). (5) concepts without additional training by directly modifying Applying this encoding across all spatial locations produces model weights. RECE [26] iteratively eliminates residual a sparse feature map 𝑆 ∈ Rℎ×𝑤×𝑛 𝑓 . The spatial activation of a embeddings, and ConceptPrune [27] prunes FFN neurons 𝜌 ℎ×𝑤 , capturing correlated with the target concept. SPEED [28] further improves single learned feature 𝜌 is denoted as 𝑆 ∈ R prior preservation by introducing null-space constrained param- the distribution of that feature across the image. In this way, eter editing, where model updates are restricted to directions the SAE can be used to analyze the functional role of the corresponding cross-attention block during image generation. that minimize interference with non-target concepts. The analysis identifies that down.2.1 controls global Inference-time Intervention. Inference intervention-based composition, mid.0 captures low-level semantics, up.0.0 methods manipulate prompts or feature activations during adds structural details, and up.0.1 refines color and style. inference. SLD [18] appends concept descriptors to prompts, Here, these indices denote specific cross-attention blocks in SAFREE [19] projects toxic token embeddings orthogonally to the U-Net, located at its down-sampling, bottleneck and upharmful concepts, TraSCE [20] provides trajectory-level guidsampling stages, respectively. ance, and SafeRedir [21] redirects the prompt embeddings of unsafe concepts. Based on Sparse Autoencoders, SAEUron [22] D. Threat Model trains them on U-Net feature activations, ConceptSteerers [23] With the rapid progress of generative artificial intelligence, applies SAEs to the text encoder, and SAEmnesia [24] uses diffusion-based image generation models have been widely supervised SAE training for clearer concept-neuron mapping. deployed in online platforms and local applications. However, this flexibility also raises safety and copyright concerns, as C. Sparse Autoencoder in Stable Diffusion users may intentionally generate inappropriate content (e.g., Prior work [33] represents the first application of SAEs to nudity, violence) or unintentionally produce images that mimic Stable Diffusion models, facilitating a systematic examination copyrighted artistic styles (e.g., Van Gogh). We consider two of the internal representations and functional organization of realistic deployment scenarios for T2I diffusion models in U-Net during the image synthesis process. In this work, SAEs practical deployment settings. are trained on SDXL-Turbo, whose efficient and fast sampling S1: Online Deployment with Controlled Access. In this pipeline makes it particularly suitable for interpretability studies, setting, the service provider deploys the diffusion model online focusing on four cross-attention blocks that exhibit clearly and exposes its interface for user interaction under controlled interpretable behaviors. conditions. The adversaries seek to induce the model to

3

generate unsafe or policy-violating images. Their access to the model is strictly black-box: they can only interact with the system through text prompts and observe the generated outputs. As a result, adversaries may attempt prompt-based attacks, such as carefully crafted or obfuscated prompts, to bypass safety filters in online deployments. S2: Public Release of Model Checkpoints. We assume that the model providers who host or distribute text-to-image diffusion models are trusted. Their goal is to maintain a balance between safety and utility, preventing the generation of harmful or undesired content while preserving the model’s ability to produce benign and creative outputs. In practice, such providers may share model checkpoints with the research community. Adversaries can download the released model weights and deploy the model locally with full control. They can attempt low-cost strategies to bypass safety mechanisms, e.g., by removing external modules or directly querying the base model without the added controls. Under this setting, the model provider does not bear responsibility for any consequences resulting from downstream modification to the model weights.

Fig. 1. Visualization of the top-6 activated SAE features across different cross-attention blocks. The black rectangle visually masks the region containing nudity.

III. E MPIRICAL S TUDY MU methods in diffusion models have evolved rapidly in recent years. However, many design choices, including localization strategy, editing paradigm, optimization scope and evaluation protocol, remain largely empirical. In this section, we systematically investigate these design choices through a series of empirical studies. Our findings not only reveal the limitations of existing approaches but also motivate the design principles adopted in C ONCEPTACP.

SAEs provide a potential solution by decomposing polysemantic neurons into more sparse and interpretable features, motivating feature-level localization for more precise concept unlearning. In Figure 1, we visualize the six most highly activated SAE features across four representative cross-attention blocks for the same input. Notably, visual features associated with the naked concept show strong activation in the down.2.1 cross-attention block. This observation indicates that concepts are localized to specific internal representations rather than uniformly distributed across the network.

A. The Limitations of Neuron-level Localization Unlearning methods must first determine where to edit, as precise localization enables effective concept removal with minimal interference to unrelated concepts. Existing approaches perform localization at different levels of the network. For example, ESD [9] modifies model parameters globally, UCE [42] edits concept representations through cross-attention layers, and ConceptPrune [27] identifies and suppresses concept-related neurons in the FFN modules of the U-Net. However, these approaches still rely on neuron-level or architecture-level localization, which faces fundamental challenges due to the polysemantic nature of neurons [29], [30], [31], where a single neuron may encode information about multiple concepts, hindering selective concept removal. As shown in Table III, although existing methods achieve considerable forgetting success rates, they often degrade image quality, suggesting that coarse localization can affect representations beyond the target concept. While recent methods such as SPEED [28] improve the preservation of unrelated concepts by constraining parameter updates to less interfering subspaces, they mainly focus on how to edit after selecting the editing space. However, achieving precise unlearning also requires addressing where to edit, i.e., accurately identifying the internal representations associated with the target concept.

Observation 1: Neuron-level localization is insufficient for precise concept removal due to neuron polysemanticity, often affecting unrelated representations and degrading generation quality. B. The Fragility of Inference-time Steering Instead of modifying model parameters, inference-time unlearning methods perform concept suppression during inference by introducing external modules or modifying the generation pipeline. However, such approaches remain fragile due to their bypassability in open-source settings and the additional computation incurred for each generation. As shown in Table I, we evaluate three representative methods (TraSCE [20], SAFREE [19] and SAEUron [22]) on SDXL-Turbo and observe that they introduce 14.4%–125.8% additional inference latency, which translates into continuous computational expenses during deployment despite avoiding one-time editing costs. In contrast, parameter editing incurs a one-time cost with zero inference overhead, whereas inferencetime steering incurs continuous overhead that accumulates with the number of requests. Using a representative GPU serving cost of $4/hour based on on-demand NVIDIA A100 pricing and

4

TABLE I I NFERENCE OVERHEAD COMPARISON OF INFERENCE - TIME CONCEPT FORGETTING METHODS ON SDXL-T URBO .

Method

SAEUron

TraSCE

SAFREE

Inference Overhead

+73.3%

+125.8%

+14.4%

the original SDXL-Turbo latency of 0.6848 seconds (Table IX), the measured overhead translates to approximately $110–$960 additional cost per million generation requests, which is nonnegligible at scale. Moreover, SAEUron leverages SAEs for targeted inferencetime suppression but directly manipulates intermediate representations, potentially disturbing the original feature distribution and introducing visual artifacts [25]. Together with the inference overhead discussed above, these limitations make direct parameter editing a more robust alternative, as it permanently modifies the model without additional inference cost.

Fig. 2. Discrepancy between classifier-based evaluation and human perception after concept unlearning.

accurately reflect human perception, resulting in discrepancies between automated assessment and actual visual understanding. As illustrated in Figure 2, generated images that receive similar classifier predictions may exhibit different levels of concept presence from a human perspective. Therefore, output-level metrics alone provide incomplete evidence of genuine concept forgetting. Analyzing internal representations associated with visual concepts can provide complementary insights into the forgetting process, yet remains largely unexplored so far in existing works.

Observation 2: Inference-time interventions incur nonnegligible runtime overhead, remain vulnerable to bypass attacks, and may introduce visual artifacts by disturbing intermediate representations. C. The Inefficiency of Large-scale Fine-tuning Existing fine-tuning-based unlearning methods [9], [40], [32] suffer from substantial computational overhead due to both large-scale parameter updates and the iterative nature of diffusion models. These methods typically modify a large portion of model parameters (e.g., multiple cross-attention layers or global weights), while obtaining supervision at a specific denoising timestep requires executing all preceding steps. Consequently, updating even a local module remains coupled with the full denoising trajectory, leading to inefficient optimization. As shown in Table VIII, fine-tuning a single cross-attention block without executing the complete denoising trajectory reduces the per-iteration training time to 53.6% of the original approach. This demonstrates that a significant portion of the optimization cost comes from the dependency on the full diffusion trajectory rather than the actual parameters being updated. This highlights the substantial computational overhead of large-scale fine-tuning, which is not necessarily accompanied by improved concept forgetting.

Observation 4: Output-level evaluation provides incomplete evidence of concept forgetting as classifier predictions may not align with human perception, motivating complementary analysis of internal representations. E. The Generalization Gap Across Diffusion Architectures Most existing MU methods were developed for Stable Diffusion v1.4/1.5. However, modern diffusion models such as SDXL-Turbo introduce substantial differences in model architecture and inference, including deeper U-Net structures and significantly fewer sampling steps. When directly applied to SDXL-Turbo, existing methods exhibit notable performance degradation (e.g., ESD), which we hypothesize is partly due to its fast-sampling regime (1–4 vs. tens of steps in standard SD). Quantitative results and visualizations (see §VI-B) further support these limitations under fast-sampling conditions.

Observation 3: Existing fine-tuning-based unlearning methods incur substantial optimization costs due to largescale parameter updates and full-trajectory dependency, while local updates may achieve comparable forgetting with substantially lower cost.

Observation 5: Existing unlearning methods, particularly timestep-wise interventions, show limited crossarchitecture generalization, as SD1.x-based approaches degrade significantly on modern fast-sampling models.

D. The Limitations of Output-based Evaluation

Overall, these observations highlight the need for unlearning approaches that are both precise and effective across different diffusion paradigms, including traditional multi-step diffusion models as well as modern large-scale, fast-sampling ones, as addressed by the method proposed in this work.

Existing diffusion unlearning methods are typically evaluated through generated outputs, where classifiers are used to determine whether the target concept remains present after unlearning. However, classifier-based evaluation may not

5

Fig. 3. Overview of C ONCEPTACP. Given a target concept, we first automatically identify relevant cross-attention blocks and associated features, then such features are selectively modified and mapped to the corresponding cross-attention outputs to guide fine-tuning.

Subsequently, we construct fine-tuning datasets by modifying the localized features controlled by a scaling factor 𝑎 (defined in §IV-C) and projecting them back to the output space of the corresponding cross-attention block. These corrected representations serve as supervision signals for subsequent parameter optimization. Crucially, instead of performing end-toend diffusion inference during training, we only need a single forward pass to record input–output pairs of the identified cross-attention block along the denoising trajectory, enabling inference-free fine-tuning. Finally, we perform block fine-tuning using a joint objective (Eq. (11) in §IV-D) that balances target concept suppression and preservation of generation quality. The resulting parameter updates internalize the SAE-based feature corrections into the model, enabling precise concept erasure without additional inference-time components. The updated cross-attention block is then used in place of the original block during generation. Overall, this framework enables precise and interpretable unlearning across different concept categories.

IV. C ONCEPT ACUPUNCTURIST A. Overview

Based on the empirical analysis in §III, we identify the need for a more precise and efficient fine-tuning approach. Our key idea is to leverage SAEs for feature-level localization and intervention, while internalizing their corrections into model parameters through localized fine-tuning. Unlike prior methods that attach SAEs to the final model during inference [22], [23], [43], we instead use them to construct supervision signals for parameter optimization. This design introduces no additional inference overhead while embedding the unlearning effect directly into model parameters, thereby addressing S1 and supporting S2, while requiring attackers to perform weightlevel manipulation rather than merely bypass inference-time interventions. Meanwhile, we localize the update to a single cross-attention block most relevant to the target concept, thus avoiding end-to-end optimization along the full diffusion trajectory and reducing the computational cost of fine-tuning. An overview of our approach is shown in Figure 3. We begin B. Feature Localization with feature localization to identify the block most strongly As shown in the first panel of Figure 3, C ONCEPTACP begins associated with the target concept, along with its corresponding with feature localization. To remove a specific concept, we first features. We establish mappings between semantic concepts identify its corresponding features and the cross-attention block and cross-attention blocks, and apply the feature localization in which they are most strongly represented. Prior work [33] algorithm introduced in §IV-B to automatically identify both reports that four SAEs trained on cross-attention blocks provide the target block and the corresponding features to be modified. interpretable representations of distinct visual elements. Guided This joint localization substantially narrows the fine-tuning by this observation, we use publicly available general-purpose scope, enabling faster and more targeted updates. SAEs as the candidate pool to localize the features and target

6

Fig. 5. Effects of suppressing feature #4913 (nudity) and enhancing feature #4998 (cartoon style) on generated images. Fig. 4. Effect of suppressing key features on generated images. Suppressing #4913 removes nudity, while no single feature fully removes the Church concept.

respect to concept 𝑐 is defined as: 𝜇 𝑐 [𝑖, 𝑓 ] 𝜇¬𝑐 [𝑖, 𝑓 ] 𝑆(𝑐, D, 𝑖, 𝑓 ) = ∑︁ − ∑︁ . ′ 𝜇 𝑐 [𝑖, 𝑓 ] 𝜇¬𝑐 [𝑖, 𝑓 ′ ]

(6)

block associated with the concept, without requiring additional 𝑓′ 𝑓′ SAE training. Notably, C ONCEPTACP can also utilize taskspecific SAEs, although the publicly available SAEs already Here, 𝜇 𝑐 [𝑖, 𝑓 ] and 𝜇¬𝑐 [𝑖, 𝑓 ] denote the mean activations of provide sufficiently informative features for precise localization feature 𝑓 in block 𝑖 averaged over feature activations produced by the SAE from D𝑐 and D¬𝑐 , respectively. Normalizing by in our experiments. the sum of all feature means captures each feature’s relative Although it is possible to identify features related to a importance within the block. The block containing the feature given concept by examining the feature heatmaps or manipwith the highest relevance score is selected as the target block. ulating feature activations to observe the resulting changes We therefore select this block for fine-tuning and consider its in generated images, this process is time-consuming and high-scoring features as the concept-related features. unreliable, particularly when concept-related activations are Because a single concept may be distributed across multiple weak or ambiguous. Moreover, while SAEs promote sparse features, we introduce a threshold 𝜏 to filter out weakly related representations, a given concept may still correspond to multiple features. Let 𝑠 𝑚𝑎𝑥 denote the maximum relevance score among features, particularly when training data for that concept is features in the selected block. We set 𝜏 = 𝜌𝑠 𝑚𝑎𝑥 , where 𝜌 = 0.2 limited or when the concept involves diverse visual elements. is the relative threshold; features with scores below 𝜏 are treated As shown in Figure 4, our preliminary analysis indicates that as unrelated to the concept. The pseudocode for the featurefeature #4913 on down.2.1 can effectively remove Nudity, selection process is provided in Algorithm 1 in Appendix A. whereas suppressing any one of the top-5 most activated features individually fails to fully eliminate Church. C. Fine-tuning Dataset Generation To address these challenges, inspired by the feature selection As shown in the second panel of Figure 3, after identifying strategy in SAEUron [22], we further develop an automated feature localization algorithm to locate relevant features and the features to suppress, we attenuate their activations via the the target block. Specifically, we compute the statistical SAE by multiplying them with a negative factor and decode the activations of features over subsets of relevant and irrelevant modified output, which represents the residual update between data, assigning each feature a score based on the differences in the output and input of the corresponding cross-attention block. their activation distributions, where a higher score indicates a Combining this residual with the original block input yields stronger correlation with the target concept. Using this scoring the concept-erased output for 𝑝 𝑐 , which forms the input-output mechanism, we identify both the cross-attention block and the pairs as part of the fine-tuning data to update the parameters. specific features representing the concept to be erased, forming However, before this process, a key question arises: by how the basis for the subsequent fine-tuning stage. Unlike SAEUron, much should feature activations be suppressed? As shown which predefines the cross-attention block for each concept, in Figure 5, preliminary experiments indicate that different we automatically search across a broader pool of SAEs and suppression factors produce distinct visual effects. Smaller use concept-specific thresholds for more flexible selection. factors weaken concept removal but reduce interference with We denote the dataset containing the target concept as D𝑐 unrelated content, whereas larger factors enhance removal but and the dataset excluding it as D¬𝑐 . To quantify the association may distort the image or impair unrelated concepts due to overbetween intermediate features and the target concept, we suppression or inadvertent activation. Furthermore, concepts evaluate the SAEs trained on the four interpretable cross- linked to multiple features may require different suppression attention blocks over both D𝑐 and D¬𝑐 . Specifically, we magnitudes for each feature. compute relevance score for each feature by comparing its To tackle this issue, we introduce a scaling factor 𝑎 to activation patterns across the two datasets, thereby capturing control the suppression strength. For each concept, the most how strongly it is correlated with the presence of the target relevant feature (with the highest score) is suppressed by 𝑎, concept. The relevance score of feature 𝑓 in block 𝑖 with while other features are suppressed proportionally according to

7

their relative scores. The choice of 𝑎 is determined empirically we extract the target block and construct training pairs and its optimal value may vary across concept. ⟨𝐷 in [𝑙, 𝑡], 𝐷 out [𝑙, 𝑡]⟩ for optimization. This latent I/O formulaFormally, after localizing the relevant features, we curate a tion bypasses repeated diffusion inference during fine-tuning fine-tuning dataset from a small collection of textual prompts. and avoids storing the full denoising trajectory, substantially Based on the computed scores, we modify the sparse map 𝑆 𝑝 = reducing GPU memory consumption and enabling larger batch [𝑧 1 , 𝑧2 , 𝑧3 , . . . , 𝑧 𝑚 ] of intermediate features obtained from the sizes under the same hardware budget. SAE for a prompt 𝑝 sampled from the concept-specific dataset D. Loss Function D𝑐 . The suppressed sparse map is 𝑆 ′𝑝 = [ 𝑧˜1 , 𝑧˜2 , 𝑧˜3 , . . . , 𝑧˜𝑚 ], After constructing the fine-tuning dataset, we fine-tune the where each 𝑧˜ 𝑗 is given by: parameters of the cross-attention block. To erase the target  concept while minimizing the impact on unrelated concepts, 𝑎 · 𝑧 𝑗, if 𝑗 = arg max 𝑠 𝑘 ,    𝑘   𝑠   we optimize the loss function Ledit for the 𝑙-th cross-attention 𝑗 𝑧˜ 𝑗 = 𝑎 · 𝑠 · 𝑧 𝑗 , if 𝑠 𝑗 ≥ 𝜌𝑠max and 𝑗 ≠ arg max 𝑠 𝑘 , (7) block with parameters 𝜃 . The loss consists of two terms: a max  𝑙 𝑘   𝑧 , concept-erasure term and a preservation term. Formally, otherwise, 𝑗  min Ledit = L 𝑐 + 𝜆L¬𝑐 ,

(11)

where 𝑧 𝑗 is the original activation of feature 𝑗, 𝑠 𝑗 represents the relevance score, and 𝑠max = max 𝑗 𝑠 𝑗 denotes the highest relevance score among the features in the selected block. The most relevant feature (with score 𝑠max ) is scaled by 𝑎, where 𝑎 < 0 controls the strength and direction of the feature correction. Other features scores above the threshold  with  𝑠𝑗 𝜌𝑠max are scaled by 𝑎 · 𝑠max . Features with scores below the threshold remain unchanged. Given the modified sparse map 𝑆 ′𝑝 , we compute the residual update of the corresponding cross-attention block according to Eq. (3) in §II-C:

where (𝑥, 𝑦) denotes the input-output pairs ⟨𝐷 in [𝑙], 𝐷 out [𝑙]⟩ for the 𝑙-th cross-attention block. The weight 𝜆 balances concept erasure and preservation, and its value is chosen empirically depending on concept types such as nudity, style and object.

Δ𝐷 ′𝑐, 𝑝 [𝑙] = OUTPUT(𝑆 𝐴𝐸 [𝑙]) = 𝑊 𝐷𝐸𝐶 [𝑙] · 𝑆 ′𝑝 + 𝑏[𝑙], (8)

V. N EW M ETRICS

𝜃𝑙

L𝑐 =

L¬𝑐 =

where 𝑏[𝑙] denotes the bias term of the 𝑙-th block. The concepterased output of the 𝑙-th cross-attention transformer block which is used as the supervision signals for fine-tuning the ′ corresponding cross-attention block, denoted as 𝐷 out 𝑐, 𝑝 [𝑙], is then computed following Eq. (4) in §II-C: ′

in ′ 𝐷 out 𝑐, 𝑝 [𝑙] = 𝐷 𝑐, 𝑝 [𝑙] + Δ𝐷 𝑐, 𝑝 [𝑙].

1 |D𝑐 [𝑙] | 1 |D¬𝑐 [𝑙] |

∑︁

2

𝑓 𝜃𝑙 (𝑥) − 𝑦 2 ,

(12)

( 𝑥,𝑦) ∈ D𝑐 [𝑙]

∑︁

2

𝑓 𝜃𝑙 (𝑥) − 𝑦 2 ,

(13)

( 𝑥,𝑦) ∈ D¬𝑐 [𝑙]

As discussed in §III-D, existing MU methods primarily rely on output-level metrics, such as the Forgetting Success Rate, which may not fully reflect changes in internal representations due to non-standardized evaluation datasets [34] and discrepancies between classifier predictions and human perception [44]. Consequently, output-level metrics provide limited insight into whether the target concept has been erased from internal representations, making it difficult to understand the internal mechanisms behind unlearning. To complement output-level evaluation, we consider an internal perspective that examines how model representations change during unlearning. Rather than introducing a standalone benchmark, our goal is to provide a post-hoc analysis tool that can be applied to different unlearning methods, helping interpret their effectiveness from the perspective of internal representation changes. To this end, we use SAEs as a unified analysis tool for probing intermediate representations. SAEs model the residual between input and output latents of a given block at a diffusion timestep, learning a sparse and interpretable feature basis. Prior work [33] shows that these learned features generalize across closely related models; for example, SAEs trained on SDXL can be directly applied to SDXL-Turbo at the same block, suggesting that they capture stable and semantically meaningful patterns rather than model-specific artifacts. Based on this property, we treat SAEs as a fixed feature space for analyzing model behavior before and after unlearning, independent of the specific MU method. Since the semantic

(9)

To simultaneously preserve the model’s performance on other tasks, we employ an additional portion of fine-tuning data, D¬𝑐 [𝑙], during training. Let the prompts without the target concept 𝑐 be passed through the 𝑙-th cross-attention block without modifying any activations, and denote the output as 𝐷 out ¬𝑐, 𝑝 [𝑙]. We use this as the output corresponding to D¬𝑐 [𝑙] in the input-output pairs of the fine-tuning dataset. Specifically, D𝑐 [𝑙] enforces concept erasure, while D¬𝑐 [𝑙] mitigates collateral effects on unrelated concepts. For efficiency, we directly use the input activation vector to the block, denoted as 𝐷 in [𝑙], instead of the original prompt, so that fine-tuning can be performed without re-running the preceding pipeline for each prompt. Thus, we formally define two fine-tuning datasets based on the original prompts as: n  o out′ D𝑐 [𝑙] = 𝐷 in [𝑙, 𝑡], 𝐷 [𝑙, 𝑡] | 𝑝 ∈ D , 𝑡 ∈ T , 𝑐 𝑐, 𝑝 𝑐, 𝑝 n  o out D¬𝑐 [𝑙] = 𝐷 in ¬𝑐, 𝑝 [𝑙, 𝑡], 𝐷 ¬𝑐, 𝑝 [𝑙, 𝑡] | 𝑝 ∈ D¬𝑐 , 𝑡 ∈ T , (10) in where 𝐷 in 𝑐, 𝑝 [𝑙] and 𝐷 ¬𝑐, 𝑝 [𝑙] denote the input vectors to block 𝑙 for the corresponding prompts. During fine-tuning,

8

meaning of SAE features remains largely consistent, changes in their activations can directly reflect how unlearning affects internal representations. From this perspective, we define two complementary internal metrics that quantify changes in feature activations induced by unlearning. These metrics serve as diagnostic tools rather than replacements for output-level evaluation, as they are measured in an SAE-derived feature space and may thus inherently favor SAE-based approaches. FO Score. We introduce the Feature Overlap (FO) Score, which measures how the most activated SAE features change before and after unlearning. Specifically, for the cross-attention module associated with the target concept, we use its corresponding SAE to extract the top-𝐾 activated features under two settings: prompts containing the target concept 𝑐 and prompts excluding it. We denote the pre post resulting feature sets before and after unlearning as F𝑐 , F𝑐 , pre post F¬𝑐 and F¬𝑐 , respectively. We then compute the feature overlap ratios: pre

post

pre

VI. E VALUATION In this section, we present the experimental setup, including the datasets, baselines and evaluation metrics, followed by the detailed results. Through extensive experiments, we address the following research questions (RQs): • RQ1: How effective is C ONCEPTACP in unlearning compared with other methods? (§VI-B) • RQ2: How well does C ONCEPTACP preserve its generation capability for concepts unrelated to the forgotten target compared with other methods? (§VI-C) • RQ3: How precisely does C ONCEPTACP remove target concepts from the model’s internal representations? (§VI-D) • RQ4: How robust is C ONCEPTACP when forgetting given concepts? (§VI-E) • RQ5: How does C ONCEPTACP perform in multi-concept unlearning? (§VI-F) • RQ6: How do the hyperparameters affect the performance of C ONCEPTACP? (§ VI-G)

post

∩ F𝑐 | |F ∩ F¬𝑐 | , 𝑟 ¬𝑐 = ¬𝑐 . (14) 𝐾 𝐾 Intuitively, successful unlearning should disrupt the dominant feature activations associated with the target concept, resulting in a smaller 𝑟 𝑐 . This indicates that the most strongly activated features before and after unlearning differ substantially, suggesting that the original concept-related representations have been altered. Meanwhile, it should preserve the feature activations for unrelated concepts, leading to a larger 𝑟 ¬𝑐 , which implies minimal interference with non-target visual semantics. To provide a unified and interpretable measure, we combine these two criteria by normalizing them into a single score: 𝑟𝑐 =

|F𝑐

complementary view of unlearning behavior at the distribution level. We combine these two criteria into a unified score: (1 − sim𝑐 ) + sim¬𝑐 FAC Score(𝑐) = , (17) 2 which lies in [0, 1], where higher values indicate more precise unlearning, i.e., stronger suppression of target-concept activation patterns with minimal impact on unrelated features. The FO Score captures changes in dominant feature identities, while the FAC Score measures shifts in feature activation distributions. Together, they provide complementary evidence of target-concept suppression and non-target preservation.

A. Setup

(1 − 𝑟 𝑐 ) + 𝑟 ¬𝑐 FO Score(c) = , (15) 2 which lies in [0, 1], with higher values indicating more precise unlearning, i.e., stronger suppression of target-concept features while preserving unrelated ones. FAC Score. Complementary to FO, the Feature Activation Correlation (FAC) Score measures changes in feature activation distributions rather than discrete feature identities. For the cross-attention module associated with the target concept, we compute the mean SAE feature activation maps over D𝑐 and D¬𝑐 before and after unlearning, denoted as pre post pre post M 𝑐 , M 𝑐 , M¬𝑐 , M¬𝑐 , respectively, and measure their cosine similarity: pre

post

sim𝑐 = cos(M 𝑐 , M 𝑐 ),

pre

Unlearning Tasks. Following prior works [45], [46], [34], [9], [42], we consider three representative unlearning scenarios: global-abstract-style, local-abstract-style and local-concreteobjects, followed by four specific tasks, i.e., Nudity, VanGogh, Church and Parachute. These four concepts are commonly adopted as representative unlearning targets, with Church and Parachute in particular being widely used due to their known difficulty to forget effectively. Other unlearning tasks that can be classified into the above three categories are treated as the same type and handled with the same pipeline. Models. We conduct our main experiments on SDXL-Turbo, which supports high-quality image generation with 1–4 inference steps. We evaluate both 1-step and 4-step generation, denoted as T1 and T4, respectively, and further evaluate our method on multi-step models, including SDXL with 30 steps and SD v1.4 with 50 steps. Datasets. For feature analysis and fine-tuning data construction, we use a synthetic prompt dataset generated with ChatGPT, containing 50 prompts per concept. For benchmarking concept unlearning, we adopt EvalIGMU benchmark [34], which covers four representative concepts: Nudity, VanGogh, Church and Parachute. For robustness evaluation, we further consider three safety-related benchmarks: Inappropriate Image Prompts

post

sim¬𝑐 = cos(M¬𝑐 , M¬𝑐 ). (16)

These similarities capture how much the overall activation distribution of concept-related features is preserved and reflect the stability of feature activations for unrelated concepts. Effective unlearning should decrease sim𝑐 while preserving sim¬𝑐 , indicating stronger changes in target-related activations and minimal changes to unrelated ones. Unlike FO which focuses on whether the same features remain dominant, FAC captures how strongly features are activated, providing a

9

Nudity T1

T4

VanGogh T1 T4

Church T1 T4

Parachute T1 T4

SDXLTurbo

ESD

UCE

Concept Prune SAE Uron Concept ACP (Ours)

Fig. 6. Qualitative comparison of forgetting performance across four concepts on SDXL-Turbo. T1 denotes generation in one timestep, and T4 denotes generation in four timesteps.

(I2P) [47], Model Manipulation Attack (MMA) dataset [48], and the UnlearnDiffAtk method [49]. Detailed dataset statistics and generation configurations are provided in Appendix B-A. Baselines. We compare our approach against representative unlearning methods, including ESD [9], UCE [42], ConceptPrune [27] and SAEUron [22]. These methods respectively represent fine-tuning-based, model editing, neuron pruning and SAE-based inference approaches. For a fair comparison, we reproduce all baselines on the SDXL-Turbo model following their official implementations. Brief descriptions and implementation details are provided in Appendix B-B. Metrics. We evaluate unlearning performance from four aspects. Unlearning effectiveness is measured using Forgetting Success Rate (FSR), computed by EvalIGMU concept classifiers [34], and CLIP Accuracy (CLIP Acc.), which assesses alignment with positive and negative concept anchors in CLIP embedding space [50], [51], [38]. Image quality is evaluated using Fréchet Inception Distance (FID) [52]. Robustness is assessed by Attack Success Rate (ASR) under adversarial prompting. Finally, we measure unlearning precision using two internal metrics: FO Score and FAC Score proposed in §V, which provide a post-hoc diagnostic of changes in the model’s internal representations during unlearning. Details can be found in Appendix B-C. B. Unlearning Effectiveness Table II reports the FSR and CLIP Acc. across four representative unlearning tasks, and Figure 6 illustrates generated images. Additional visual comparisons with the baselines can be found in Figure 14 in Appendix C-A. As shown in Table II, C ONCEPTACP achieves the highest average FSR and the lowest average CLIP Acc. under both T1 and T4, demonstrating strong unlearning effectiveness. At T1, it improves FSR by 36.34 and reduces CLIP Acc. by 32.10 compared to the best baseline. At T4, it maintains its advantage, improving FSR by 24.10 and reducing CLIP Acc. by 12.48. These results demonstrate that C ONCEPTACP effectively erases target

10

concepts while preserving unrelated content, even under fastsampling conditions. Observations. We summarize four key observations from our experimental results: First, ESD underperforms mainly due to its dependence on interventions across denoising timesteps. While it typically finetunes 50-100 denoising steps on models such as SD v1.4, the 1–4 sampling steps in SDXL-Turbo limit its ability to suppress target concepts, resulting in suboptimal forgetting. In contrast, although trained with single-step generation, C ONCEPTACP generalizes well to multi-step inference, achieving the highest average FSR under T4. Moreover, as shown in Figure 10 in Appendix C-A, the forgetting effect persists across extended generation schedules of up to 30 steps, demonstrating effectiveness across different generation lengths. This is consistent with prior findings [41] that early timesteps establish coarse layouts and scene composition, making interventions at this stage effective for controlling generation. Second, forgetting for Parachute is less consistent. As illustrated in Figure 6, C ONCEPTACP removes most instances in many cases, although residual fragments may remain; a detailed analysis is provided in §VII. Other methods often retain substantial portions of the parachute, indicating weaker erasure. Notably, ESD achieves an unusually high FSR of 82.80 under T4, but manual inspection reveals dense, abstract, and cartoonish outputs that are less realistic and semantically coherent than those of C ONCEPTACP. This suggests that a high FSR does not necessarily correspond to better visual quality or more faithful concept removal. Third, for VanGogh in the one-step generation case, we observe that the main subject of the generated images can be affected in Figure 14 in Appendix C-A. This phenomenon can be attributed to two factors: (i) the visualized examples are selected under a trade-off between FSR and FID, and the VanGogh classifier does not always perfectly align with human perception; and (ii) VanGogh represents a global style that inherently influences the entire image. Therefore, for global style concepts such as VanGogh, we do not recommend using an excessively large suppression coefficient 𝑎 in practice. We provide a more detailed analysis of this issue in §VI-G. For other concepts and under T4, C ONCEPTACP consistently achieves strong forgetting with comparable image quality and no noticeable degradation. Finally, compared with SAEUron which employs SAEs as external modules, our method achieves notably higher FSRs for Nudity, VanGogh, Church and Parachute, demonstrating the effectiveness of our feature-level fine-tuning strategy. Effectiveness Across Diffusion Backbones. We further evaluate C ONCEPTACP on SDXL and SD v1.4, with results reported in Table III. C ONCEPTACP achieves effective concept forgetting across diverse target concepts on both backbones, demonstrating its generalization beyond the SDXL-Turbo. These results suggest that our representation-guided editing strategy is robust to different diffusion architectures and sampling configurations without requiring architecture-specific modifications. Details are provided in Appendix C-A.

TABLE II C OMPARISON OF UNLEARNING PERFORMANCE ACROSS FOUR CONCEPTS UNDER T1 AND T4. FSR (%, ↑), CLIP ACC . (%, ↓) AND FID (↓) ARE REPORTED . T HE BEST FSR AND CLIP ACC . ARE IN BOLD , AND THE SECOND - BEST ONES ARE UNDERLINED .

Method

T1 (one-step generation)

Metric

Nudity VanGogh Church Parachute

Avg.

Nudity VanGogh Church Parachute

Avg.

7.44 96.40 7.44 96.40 0.69 28.72 82.48 31.25 38.32 80.40 34.98 13.82 88.80 31.92 94.16 37.44 36.04

17.80 93.72 17.20 93.58 0.61 41.48 79.92 21.16 38.78 70.36 27.34 43.68 79.00 27.85 80.02 38.26 29.84

11.60 95.36 25.12 96.96 59.79 30.56 83.12 36.95 71.20 60.88 45.82 24.80 91.92 34.61 97.60 36.64 39.09

5.52 95.08 28.16 87.18 37.53 31.70 83.92 25.01 45.02 62.00 37.41 22.86 70.68 27.79 69.12 49.52 30.52

FSR CLIP Acc. FID FSR ESD CLIP Acc. FID FSR UCE CLIP Acc. FID FSR ConceptPrune CLIP Acc. FID FSR SAEUron CLIP Acc. FID FSR ConceptACP CLIP Acc. FID SDXL-Turbo

57.68 89.28 58.96 88.72 1.13 90.00 60.80 4.95 75.60 43.44 32.88 76.90 71.20 30.23 95.92 33.20 41.49

1.60 95.20 0.80 95.20 0.22 12.80 90.80 21.31 26.40 70.80 20.84 50.00 70.40 19.46 84.40 16.00 20.01

1.60 94.00 1.60 94.00 0.38 34.40 85.60 27.11 14.80 86.80 20.64 34.00 85.60 29.77 45.60 66.40 21.80

TABLE III U NLEARNING PERFORMANCE ON DIFFERENT DIFFUSION BACKBONES . FSR (%, ↑) AND FID (↓) ARE REPORTED .

Method

Metric Nudity VanGogh Church Parachute 10.00 49.80 34.00 94.70 76.00 121.96 48.00 113.79 100.00 45.29

41.60 81.04 45.00 96.73 77.90 197.32 42.10 47.38 89.20 43.35

Method

Metric Nudity VanGogh Church Parachute

Avg

FSR FID FSR FID FSR FID FSR FID FSR FID

73.30 86.97 66.20 65.09 51.30 81.09 97.40 266.78 95.40 82.29

Concept Prune SAEUron

92.80 164.06 92.40 63.07 93.60 183.26 36.80 28.54 93.60 41.95

2.00 46.80 2.00 90.26 42.00 225.27 20.00 20.72 64.00 40.51

Concept ACP

UCE

61.60 63.48 51.60 138.88 100.00 258.78 63.60 26.48 99.20 45.66

0.80 95.20 3.60 87.20 18.79 7.20 92.40 22.88 21.60 56.80 27.80 20.00 49.60 21.47 70.40 18.80 21.79

2.00 93.20 82.80 67.60 21.79 36.40 87.20 28.95 21.20 79.20 29.57 17.60 74.80 21.84 46.00 74.40 22.79

Concept

Score

Avg

FSR FID FSR FID FSR FID FSR FID FSR FID

ESD

7.68 96.56 1.12 96.96 49.75 52.64 72.96 11.25 66.08 51.12 46.43 29.04 66.40 33.23 62.48 68.24 38.40

TABLE IV C OMPARISON OF UNLEARNING PRECISION . FO S CORE (↑) AND FAC S CORE (↑) ARE REPORTED , WITH 𝐾 = 20 FOR FO EVALUATION .

(a) SDXL Method

T4 (four-step generation)

Nudity

VanGogh

Church

Parachute

Avg.

ESD

FO FAC

0.68 0.60

0.68 0.73

0.75 0.60

0.93 0.97

0.76 0.73

UCE

FO FAC

0.60 0.51

0.60 0.56

0.53 0.51

0.55 0.55

0.57 0.53

ConceptPrune

FO FAC

0.58 0.55

0.60 0.73

0.70 0.58

0.43 0.52

0.58 0.60

ConceptACP

FO FAC

0.95 0.93

0.83 0.97

0.95 0.98

0.85 0.95

0.90 0.96

(b) SD v1.4

ESD UCE Concept Prune SAEUron Concept ACP

77.20 102.07 80.80 96.18 69.60 91.38 100.00 310.46 96.00 93.57

98.00 86.71 60.00 35.36 81.60 86.76 99.60 218.35 91.60 70.64

60.00 78.38 62.00 62.95 28.00 70.42 98.00 267.95 96.00 72.65

58.00 80.72 62.00 65.85 26.00 75.78 92.00 270.34 98.00 92.31

its limited parameter changes and lower FSR. Compared to neuron-level pruning in ConceptPrune, our method delivers substantially higher FSR while keeping comparable FID in T1 (higher by only 2.50 on average) and lower FID in T4 (lower by 6.89 on average). Compared with SAEUron, C ONCEPTACP likewise achieves higher average FSR with only a slight FID increase (1.99 in T1 and 2.73 in T4), demonstrating a favorable trade-off between forgetting and generation quality. UCE achieves the lowest average FID among baselines in T4 but has markedly lower FSR, further highlighting this trade-off.

C. Unrelated Concept Preservation

D. Feature-Level Unlearning Precision

To assess the impact on unrelated content, we compare FID across tasks in Table II. Overall, C ONCEPTACP maintains competitive FID while achieving stronger forgetting. ESD attains the lowest average FID in T1, consistent with

Table IV reports the changes in internal feature activations before and after unlearning. Note that this evaluation is not applicable to methods that perform inference-time unlearning

11

SDXL-

TABLE V C OMPARISON OF ROBUSTNESS ON N UDITY EVALUATED BY ASR (%, ↓) WITH 4 DENOISING STEPS .

Method

I2P

MMA

UnlearnDiffAtk

Avg.

SDXL-Turbo ESD UCE ConceptPrune SAEUron ConceptACP

13.32 12.89 9.45 0.75 4.62 3.33

42.00 38.20 36.70 6.30 11.50 16.60

36.61 30.83 30.44 1.35 18.11 5.97

30.64 27.31 25.53 2.80 11.41 8.63

Turbo

ESD

UCE

Concept Prune

SAE Uron

Concept ACP

TABLE VI ROBUSTNESS ON FOUR UNLEARNING TASKS UNDER U NLEARN D IFF ATK EVALUATED BY ASR ↓ (%) WITH 4 DENOISING STEPS . B OLD = BEST ( MINIMUM ); UNDERLINE = SECOND BEST.

Method

Nudity

VanGogh

Church

Parachute

Avg.

SDXL-Turbo ESD UCE ConceptPrune SAEUron ConceptACP

36.61 30.83 30.44 1.35 18.11 5.97

14.14 26.34 7.87 3.53 12.54 7.98

44.32 45.45 27.27 15.91 35.23 15.91

29.17 22.50 5.83 12.50 34.17 21.67

31.06 31.28 17.85 8.32 25.01 12.88 Fig. 7. Generated images from different unlearning methods on I2P, MMA and UnlearnDiffAtk prompts.

without modifying model parameters. For ESD, we evaluate the checkpoint obtained after four-step fine-tuning. Overall, ConceptPrune, SAEUron and C ONCEPTACP show On average, C ONCEPTACP improves FO and FAC over stronger robustness, achieving notably lower ASRs than ESD the second-best method by 0.14 and 0.23, respectively. It and UCE. Among them, ConceptPrune attains the best robustachieves the highest FO and FAC scores on Nudity, VanGogh ness, consistent with its direct zeroing of neurons associated and Church, and performs comparably to ESD on Parachute, with the target concept. Although the robustness of our method indicating effective forgetting with limited interference on is slightly lower than that of ConceptPrune, it consistently unrelated concepts. outperforms the other baselines across all three tests. Interestingly, ESD attains a notably high score on the Table VI further reports robustness against UnlearnDiffAtkParachute concept, consistent with its strong FSR. However, generated adversarial prompts across three additional forgetting the reason for this unusually effective unlearning, as well as tasks. Although such prompts show limited transferability the distorted and abstract appearance of its generated images, across diffusion backbones, C ONCEPTACP achieves the secondremains unclear and warrants further investigation. best average ASR, ranking only behind ConceptPrune. ComNotably, ConceptPrune attains relatively low scores. Analysis bined with the feature-level analysis, these results demonstrate of intermediate results shows that the top-𝐾 feature overlap that C ONCEPTACP provides strong robustness while better on D¬𝑐 between pre- and post-unlearning models is small, preserving unrelated concepts. suggesting disturbance to irrelevant features. Thus, despite a Figure 7 presents qualitative comparisons among different reasonable forgetting success rate, ConceptPrune exhibits lower unlearning methods. The last three methods show more stable feature-level precision. In contrast, C ONCEPTACP induces suppression of Nudity, while ESD and UCE are less robust. larger changes on D𝑐 while preserving features on D¬𝑐 , Interestingly, we observe a different visual behavior of our demonstrating finer-grained control than neuron-level pruning. method: it sometimes removes the entire human subject on the UCE records scores close to 0.5, indicating limited unlearn- benchmark prompts, whereas under adversarial prompts it more ing effectiveness with little impact on unrelated concepts. often induces clothed appearances. Under clean prompts, the human subject remains intact, suggesting that C ONCEPTACP E. Robustness does not erase the broader concept of the human body. A Prior attacks show that concepts erased by MU methods detailed discussion is provided in § VII. can be recovered through adversarial text prompts [49] or image-guided multi-modal inputs [53]. We further evaluate F. Multi-Concept Unlearning the robustness of unlearning methods against adversarial We evaluate the ability of our method to support multiprompts. Table V reports the ASR under the I2P, MMA and concept unlearning under two representative scenarios, with UnlearnDiffAtk settings for Nudity task. performance assessed by the IGMU MULTI MULTI C classifier.

12

20 0

90 0.0

0.4

0.8

1.2

1.6

84

40

82

20

80 0.0

0.4

0.8

λ

(a) Nudity

1.2

1.6

54

60

52

40

50

20 0

2.0

100

56

FSR FID

0.0

0.4

0.8

1.2

1.6

56 80 54 60 52 40 50 20

48

0

2.0

FSR FID

0.0

0.4

0.8

1.2

1.6

λ

λ

λ

(b) VanGogh

(c) Church

(d) Parachute

FID Score

60

0

2.0

80

FSR (%)

40

100

86

FID Score

95

88

FSR (%)

60

FSR FID

80

FSR (%)

FSR (%)

100

100

FID Score

FSR FID

80

FID Score

100

48

2.0

Fig. 8. Effect of 𝜆 on FSR and FID for four unlearning tasks.

89

0

78

20 0

−10 −9 −8 −7 −6 −5 −4 −3

a

FSR FID

77

50

40 20 0

−10 −9 −8 −7 −6 −5 −4 −3

a

(a) Nudity

60

FSR FID

48

−10 −9 −8 −7 −6 −5 −4 −3

a

(b) VanGogh

(c) Church

50

60 40

49

20 0

FSR FID

FID Score

40

51 80

FSR (%)

79

FSR (%)

60

100 52

80

FID Score

90

40 20

100 80

80

FSR (%)

FSR (%)

60

100

91

FID Score

FSR FID

80

FID Score

100

48

−10 −9 −8 −7 −6 −5 −4 −3

a

(d) Parachute

Fig. 9. Effect of the scale parameter 𝑎 on FSR and FID for four unlearning tasks. TABLE VII M ULTI - CONCEPT UNLEARNING RESULTS . Scenario

Concept

FSR (%) ↑

FID ↓

Different Blocks

Nudity (single) Van Gogh (single) Nudity (multi) Van Gogh (multi)

98.00 92.40 98.40(+0.40) 92.80(+0.40)

89.80 77.34 106.08(+16.28) 86.19(+8.85)

Church (single) Parachute (single) Church (multi) Parachute (multi)

80.00 40.00 64.00(-16.00) 42.00(+2.00)

47.12 50.56 66.61(+19.49) 66.61(+16.05)

TABLE VIII R ESOURCE USAGE OF FINE - TUNING . T IMES EXCLUDE DATA LOADING AND I/O. L ATENT I/O DENOTES ⟨𝐷IN [𝑙 ], 𝐷OUT [𝑙 ] ⟩ PAIRS , WHILE P ROMPT→L ATENT DENOTES PROMPT- TO - LATENT SUPERVISION . Setting

Total Time

Avg. / Epoch

Dataset

Latent I/O Prompt→Latent

670.3s 1,249.7s

6.7s 12.5s

∼72MB ∼29MB

performance and image quality, with the suppression factor fixed at 𝑎 = −5. As illustrated in Figure 8, smaller 𝜆 values tend to cause under-forgetting, while excessively large values may Concepts on Different Blocks. When target concepts are lead to over-editing and degraded fidelity. Moderate values localized in different blocks (e.g., Nudity and VanGogh), multi- generally provide the best trade-off. Specifically, 𝜆 = 0.2 concept unlearning requires no additional training and can performs best for Nudity and VanGogh, while 𝜆 = 1.8 and be achieved by directly combining the checkpoints of the 𝜆 = 2.0 are optimal for Church and Parachute, respectively. individually unlearned blocks. As shown in Table VII, the Visual differences across 𝜆 settings are subtle (Figure 11), FSRs of Nudity and VanGogh remain comparable to their indicating that quantitative metrics are more effective for single-concept counterparts after joint unlearning, indicating capturing their effects on concept erasure. that the unlearning effects are well preserved. Although FID Effect of the Scaling Factor 𝑎. We examine how the increases moderately due to the accumulation of modifications suppression factor 𝑎 affects forgetting performance. As shown across blocks, the overall generation quality remains acceptable. in Figure 9, increasing the magnitude of 𝑎 generally improves Concepts on the Same Block. When multiple concepts are FSR but may increase FID. Moderate suppression provides entangled within the same block (e.g., Church and Parachute), the best balance, while excessively strong suppression can we perform sequential unlearning by applying an additional over-edit images and reduce visual quality, as illustrated by the unlearning step to a model where one concept has already VanGogh examples in Figure 12. Accordingly, we set 𝑎 = −8, been erased. Results show that both concepts remain effectively −7, −9 and −5 for Nudity, VanGogh, Church and Parachute, suppressed after multi-concept unlearning. The FID, evaluated respectively. Consistent with the third observation in §VI-B, a on five object categories excluding the unlearned concepts, milder 𝑎 = −6 or −5 is also recommended for VanGogh. increases to 66.61, indicating a controlled impact on generation Efficiency and Resource Overhead. We evaluate the efquality. Qualitative results in Figure 13 in Appendix C-C further ficiency of our fine-tuning pipeline in Table VIII. Overall, show that unlearning effectiveness is largely preserved without C ONCEPTACP introduces minimal computational overhead. Constructing the fine-tuning dataset is fast, and our inferenceevident degradation on unrelated categories. free fine-tuning based on latent I/O is substantially more G. Ablation Study efficient than prompt-based fine-tuning in both runtime and Effect of the Regularization Parameter 𝜆. We analyze the memory usage. In particular, bypassing the diffusion inference effect of the merge weight 𝜆 on the trade-off between forgetting stage enables faster convergence and lower GPU memory Same Block

13

consumption, while AMP further improves training efficiency (e.g., 1.34 s/epoch on SDXL and 0.63 s/epoch on SD v1.4). Detailed settings and comparisons with baselines are provided in Table IX in Appendix C-D. VII. D ISCUSSION

Future Work. Future work could explore targeted SAE training approaches that make features related to the target concept more concentrated, as well as feature-selection strategies that adapt across layers and concept types, and strengthen the robustness against stronger adversarial attacks. E THICAL C ONSIDERATIONS

SAE Generality and Concept-Specific Features. Our experiments directly use publicly available general-purpose SAEs, which already provide sufficiently informative features for effective concept localization and unlearning. As shown in Table X in Appendix D, Church and Parachute involve more numerous and dispersed features than Nudity and VanGogh, with Parachute showing relatively weaker forgetting, suggesting that its representation is less cleanly decomposed into sparse features. This suggests an opportunity for further improving feature localization through concept-specific SAE training [24], which may yield more concentrated concept-relevant features. Concept Entanglement in Nudity Unlearning. We observe that on the EvalIGMU benchmark, C ONCEPTACP sometimes removes the entire human subject when suppressing Nudity. This is partly influenced by the suppression strength, but more fundamentally, unlike methods [9], [42] that steer the target toward a predefined alternative direction (e.g., Clothed), C ONCEPTACP directly locates and suppresses the features associated with the target concept without specifying a replacement representation, which may lead the model to avoid generating human subjects in some cases to prevent nudity. Importantly, this does not indicate systematic overforgetting: human subjects remain intact for other prompts, and unrelated-concept evaluations show that human figures can still be generated normally. The observed removal is therefore input-dependent and can be alleviated with milder suppression, as discussed in §VI-G and as shown in Figure 12 in Appendix C-D. Moreover, as noted by SafeGen [44], nudity and sexually explicit content are not necessarily equivalent; our unlearning target is specifically nudity rather than sexually explicit content as a whole. This is visually evident in the fourth row of Figure 7, where generated images still contain sexually suggestive elements, demonstrating the fine-grained and targeted nature of our unlearning. VIII. C ONCLUSION AND F UTURE W ORK Conclusion. In this work, we present C ONCEPTACP, a novel unlearning method that transforms interpretability-guided feature interventions into supervision signals for fine-tuning and internalizes their effects into model parameters. By localizing the update to a single cross-attention block, C ONCEPTACP enables efficient and precise concept forgetting without repeatedly running diffusion trajectories or introducing inferencetime overhead. Experiments show that C ONCEPTACP achieves superior forgetting performance over SOTA baselines while maintaining image quality, and substantially improves finetuning efficiency. Overall, this work bridges interpretability and fine-tuning-based MU, advancing safer and more controllable generative models.

14

This paper studies concept erasure for text-to-image diffusion models to reduce the generation of undesired content (e.g., NSFW concepts, copyrighted styles, or specific objects) while limiting collateral effects on unrelated content. As with other model-editing techniques, the proposed approach is dualuse: feature localization and targeted modification could be misapplied to disable safety mechanisms or to suppress benign or protected attributes. We therefore frame C ONCEPTACP as a compliance- and safety-oriented unlearning technique and encourage practitioners to apply appropriate governance measures when deploying edited models. Our experiments may involve sensitive prompts and generated outputs; we minimize manual inspection, rely primarily on automated filtering and quantitative evaluation, and avoid displaying explicit content in qualitative examples. This work does not involve new data collection from human subjects or user studies; we use established benchmarks and follow their intended research use and licensing constraints. R EFERENCES [1] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in ICML, vol. 37, 2015, pp. 2256–2265. [2] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” 2022. [Online]. Available: https://arxiv.org/abs/2204.06125 [3] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “Highresolution image synthesis with latent diffusion models,” in CVPR. IEEE, 2022, pp. 10 674–10 685. [4] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, “Photorealistic text-to-image diffusion models with deep language understanding,” in NeurIPS, 2022. [5] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “SDXL: improving latent diffusion models for high-resolution image synthesis,” in ICLR. OpenReview.net, 2024. [6] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “GLIDE: towards photorealistic image generation and editing with text-guided diffusion models,” in ICML, vol. 162. PMLR, 2022, pp. 16 784–16 804. [7] Y. Huang, J. Huang, Y. Liu, M. Yan, J. Lv, J. Liu, W. Xiong, H. Zhang, L. Cao, and S. Chen, “Diffusion model-based image editing: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, p. 4409–4437, 2025. [8] J. Wang, J. Hu, X. Ma, H. Ma, X. Wei, and E. Wu, “Image editing with diffusion models: A survey,” 2025. [Online]. Available: https://arxiv.org/abs/2504.13226 [9] R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and D. Bau, “Erasing concepts from diffusion models,” in ICCV, 2023, pp. 2426–2436. [10] J. Rando, D. Paleka, D. Lindner, L. Heim, and F. Tramèr, “Redteaming the stable diffusion safety filter,” 2022. [Online]. Available: https://arxiv.org/abs/2210.04610 [11] G. Zhang, K. Wang, X. Xu, Z. Wang, and H. Shi, “Forget-menot: Learning to forget in text-to-image diffusion models,” in CVPR Workshops, 2024, pp. 1755–1764. [12] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman et al., “Laion5b: An open large-scale dataset for training next generation image-text models,” in NeurIPS, vol. 35, 2022, pp. 25 278–25 294.

[13] Z. Zhao, J. Duan, X. Hu, K. Xu, C. Wang, R. Zhang, Z. Du, Q. Guo, and Y. Chen, “Unlearnable examples for diffusion models: Protect data from unauthorized exploitation,” 2024. [Online]. Available: https://arxiv.org/abs/2306.01902 [14] P. Schramowski, C. Tauchmann, and K. Kersting, “Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content?” in ACM FAccT, 2022, pp. 1350–1361. [15] C. Zhang, M. Hu, W. Li, and L. Wang, “Adversarial attacks and defenses on text-to-image diffusion models: A survey,” Information Fusion, vol. 114, p. 102701, 2025. [16] M. Liu, S. Zhang, and C. Long, “Wukong framework for not safe for work detection in text-to-image systems,” 2025. [Online]. Available: https://arxiv.org/abs/2508.00591 [17] Q. Shi, C. Jin, J. Zhang, and Y. Gu, “Retrack: Data unlearning in diffusion models through redirecting the denoising trajectory,” 2025. [Online]. Available: https://arxiv.org/abs/2509.13007 [18] P. Schramowski, M. Brack, B. Deiseroth, and K. Kersting, “Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models,” in CVPR, 2023. [19] J. Yoon, S. Yu, V. Patil, H. Yao, and M. Bansal, “Safree: Training-free and adaptive guard for safe text-to-image and video generation,” in ICLR, 2025. [20] A. Jain, Y. Kobayashi, T. Shibuya, Y. Takida, N. Memon, J. Togelius, and Y. Mitsufuji, “Trasce: Trajectory steering for concept erasure,” 2025. [Online]. Available: https://arxiv.org/abs/2412.07658 [21] R. Liu, K. Chen, H. Qiu, J. Zhang, K.-Y. Lam, T. Zhang, and S.-K. Ng, “Saferedir: Prompt embedding redirection for robust unlearning in image generation models,” 2026. [Online]. Available: https://arxiv.org/abs/2601.08623 [22] B. Cywiński and K. Deja, “Saeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders,” in ICML, 2025. [23] D. Kim and D. Ghadiyaram, “Concept steerers: Leveraging k-sparse autoencoders for controllable generations,” 2025. [Online]. Available: https://arxiv.org/abs/2501.19066 [24] E. Cassano, R. Renzulli, M. Nurisso, M. Zaffaroni, A. Perotti, and M. Grangetto, “Saemnesia: Erasing concepts in diffusion models with supervised sparse autoencoders,” 2025. [Online]. Available: https://arxiv.org/abs/2509.21379 [25] E. Cassano, R. Renzulli, R. Ahmed, M. Grangetto, and S. Alaniz, “Look but don’t touch with sparse autoencoders for unlearning in diffusion models,” 2026. [Online]. Available: https://arxiv.org/abs/2606.31699 [26] C. Gong, K. Chen, Z. Wei, J. Chen, and Y.-G. Jiang, “Reliable and efficient concept erasure of text-to-image diffusion models,” in ECCV, 2024. [27] R. Chavhan, D. Li, and T. Hospedales, “Conceptprune: Concept editing in diffusion models via skilled neuron pruning,” 2024. [Online]. Available: https://arxiv.org/abs/2405.19237 [28] O. Li, Y. Wang, X. Hu, H. Jiang, T. Liang, Y. Hao, G. Ma, and F. Feng, “SPEED: Scalable, precise, and efficient concept erasure for diffusion models,” in The Fourteenth International Conference on Learning Representations, 2026. [29] C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter, “Zoom in: An introduction to circuits,” Distill, 2020. [30] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah, “Toy models of superposition,” 2022. [Online]. Available: https://arxiv.org/abs/2209.10652 [31] A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, H. Cunningham, N. L. Turner, C. McDougall, M. MacDiarmid, C. D. Freeman, T. R. Sumers, E. Rees, J. Batson, A. Jermyn, S. Carter, C. Olah, and T. Henighan, “Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet,” Transformer Circuits Thread, 2024. [32] F. Carter, “Trace: Trajectory-constrained concept erasure in diffusion models,” 2025. [Online]. Available: https://arxiv.org/abs/2505.23312 [33] V. Surkov, C. Wendler, A. Mari, M. Terekhov, J. Deschenaux, R. West, C. Gulcehre, and D. Bau, “One-step is enough: Sparse autoencoders for test-time text-to-image diffusion models,” in NeurIPS, 2025. [34] R. Liu, W. Feng, T. Zhang, W. Zhou, X. Cheng, and S.-K. Ng, “Rethinking machine unlearning in image generation models,” in CCS, 2025. [35] A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach, “Adversarial diffusion distillation,” in ECCV, 2024, pp. 87–103.

15

[36] L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, “Machine unlearning,” in IEEE S&P, 2021, pp. 141–159. [37] A. F. Cooper, C. A. Choquette-Choo, M. Bogen, M. Jagielski, K. Filippova, K. Z. Liu, A. Chouldechova, J. Hayes, Y. Huang, N. Mireshghallah, I. Shumailov, E. Triantafillou, P. Kairouz, N. Mitchell, P. Liang, D. E. Ho, Y. Choi, S. Koyejo, F. Delgado, J. Grimmelmann, V. Shmatikov, C. D. Sa, S. Barocas, A. Cyphert, M. Lemley, danah boyd, J. W. Vaughan, M. Brundage, D. Bau, S. Neel, A. Z. Jacobs, A. Terzis, H. Wallach, N. Papernot, and K. Lee, “Machine unlearning doesn’t do what you think: Lessons for generative ai policy, research, and practice,” 2024. [Online]. Available: https://arxiv.org/abs/2412.06966 [38] N. Kumari, B. Zhang, S. Wang, E. Shechtman, R. Zhang, and J. Zhu, “Ablating concepts in text-to-image diffusion models,” in ICCV, 2023, pp. 22 634–22 645. [39] A. Heng and H. Soh, “Selective amnesia: A continual learning approach to forgetting in deep generative models,” in NeurIPS, vol. 36, 2023, pp. 17 170–17 194. [40] S. Lu, Z. Wang, L. Li, Y. Liu, and A. W.-K. Kong, “Mace: Mass concept erasure in diffusion models,” in CVPR, 2024, pp. 6430–6440. [41] B. Tinaz, Z. Fabian, and M. Soltanolkotabi, “Emergence and evolution of interpretable concepts in diffusion models,” 2025. [Online]. Available: https://arxiv.org/abs/2504.15473 [42] R. Gandikota, H. Orgad, Y. Belinkov, J. Materzynska, and D. Bau, “Unified concept editing in diffusion models,” in WACV, 2024, pp. 5099– 5108. [43] A. Muhamed, J. Bonato, M. Diab, and V. Smith, “Saes Can improve unlearning: Dynamic sparse autoencoder guardrails for precision unlearning in llms,” 2025. [Online]. Available: https: //arxiv.org/abs/2504.08192 [44] X. Li, Y. Yang, J. Deng, C. Yan, Y. Chen, X. Ji, and W. Xu, “Safegen: Mitigating sexually explicit content generation in text-to-image models,” in CCS, 2024, pp. 4807–4821. [45] Y. Zhang, X. Chen, J. Jia, Y. Zhang, C. Fan, J. Liu, M. Hong, K. Ding, and S. Liu, “Defensive unlearning with adversarial training for robust concept erasure in diffusion models,” in NeurIPS, 2024, pp. 36 748–36 776. [46] B. H. Lee, S. Lim, S. Lee, D. U. Kang, and S. Y. Chun, “Concept pinpoint eraser for text-to-image diffusion models via residual attention gate,” in ICLR, 2025. [47] Artificial Intelligence & Machine Learning Lab at TU Darmstadt, “Inappropriate image prompts (i2p),” https://huggingface.co/datasets/ AIML-TUDA/i2p, 2023, accessed: 2025-10-09. [48] Y. Yang, R. Gao, X. Wang, T.-Y. Ho, N. Xu, and Q. Xu, “MMA-Diffusion: MultiModal Attack on Diffusion Models,” in CVPR, 2024. [49] Y. Zhang, J. Jia, X. Chen, A. Chen, Y. Zhang, J. Liu, K. Ding, and S. Liu, “To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now,” in ECCV, 2024. [50] Y. Wu, S. Zhou, M. Yang, L. Wang, W. Zhu, H. Chang, X. Zhou, and X. Yang, “Unlearning concepts in diffusion model via concept domain correction and concept preserving gradient,” in AAAI, 2025. [51] S. Li, J. van de Weijer, T. Hu, F. S. Khan, Q. Hou, Y. Wang, and J. Yang, “Get what you want, not what you don’t: Image content suppression for text-to-image diffusion models,” in ICLR, 2024. [52] G. Parmar, R. Zhang, and J.-Y. Zhu, “On aliased resizing and surprising subtleties in gan evaluation,” in CVPR, 2022, pp. 11 400–11 410. [53] R. Liu, G. Li, T. Zhang, and S.-K. Ng, “Image can bring your memory back: A novel multi-modal guided attack against image generation model unlearning,” in ICLR, 2026.

A PPENDIX OVERVIEW This appendix provides supplementary details and experimental results that complement the main paper. Specifically, the appendix includes: • Appendix A: Feature Localization Algorithm. A formal

description of the feature localization procedure used to identify the block and concept-relevant features for targeted unlearning. • Appendix B: Evaluation Details. Detailed descriptions of the evaluation datasets, baseline methods, and metrics used

to assess unlearning effectiveness, image quality, robustness, Algorithm 1 Feature Localization for Concept 𝑐 Require: Concept 𝑐, datasets D𝑐 , D¬𝑐 , interpretable blocks and unlearning precision. {1, . . . , 𝑁 } • Appendix C: More Evaluation Results. Additional experimental results covering forgetting effectiveness, cross- Ensure: Selected block 𝑏 ∗ , relevant features F ⊆ features of block 𝑏 ∗ backbone generalization, robustness against adversarial 1: for each block 𝑖 ∈ {1, . . . , 𝑁 } do prompts, multi-concept unlearning, ablation studies, and 2: for each feature 𝑓 in block 𝑖 do computational efficiency. 3: Compute relevance score: • Appendix D: Relevant Features. The concept-relevant SAE features identified by C ONCEPTACP across different 𝑆(𝑐, D, 𝑖, 𝑓 ) concepts and diffusion backbones, together with an analysis of their consistency and diversity. 4: end for 5: Store top relevance score for block 𝑖: A PPENDIX A F EATURE L OCALIZATION A LGORITHM 𝑠𝑖 = max 𝑆(𝑐, D, 𝑖, 𝑓 ) 𝑓 In this section, we present a formal, symbolic description 6: end for of our feature localization algorithm. 7: Select block with highest top score: Specifically, given a target concept 𝑐 to be erased, we categorize it into two types: attribute-like concepts 𝑐 𝑎 (e.g., 𝑏 ∗ = arg max 𝑠𝑖 𝑖 Nudity, VanGogh style) and object-like concepts 𝑐 𝑜 (e.g., ∗ 8: Let 𝑠 𝑓 = max 𝑓 𝑆(𝑐, D, 𝑏 , 𝑓 ) Church, Parachute). For attribute-like concepts 𝑐 𝑎 , let D𝑐𝑎 9: Define threshold 𝜏 = 𝜌𝑠 𝑓 be a dataset of prompts that contain 𝑐 𝑎 and D¬𝑐𝑎 a paired dataset of prompts that do not contain 𝑐 𝑎 . For object-like 10: Select relevant features: concepts 𝑐 𝑜 , let D𝑐𝑜 be a dataset of prompts that containing F = { 𝑓 | 𝑆(𝑐, D, 𝑏 ∗ , 𝑓 ) ≥ 𝜏} 𝑐 𝑜 and D¬𝑐𝑜 a dataset of prompts that exclude 𝑐 𝑜 but may include other objects; in this case, the two sets are not pairwise 11: return 𝑏 ∗ , F aligned. For convenience, we denote the dataset containing the target concept as D𝑐 and the dataset excluding it as D¬𝑐 , VanGogh, and 1,500 images for Church and Parachute. For unifying the notation across concept types. experiments on SDXL and SD v1.4, multi-concept unlearning For each feature 𝑓 in an interpretable block 𝑖, we compute and ablation studies, we generate one image per prompt to a relevance score 𝑆(𝑐, D, 𝑖, 𝑓 ) that quantifies how strongly reduce computational overhead while maintaining consistent the feature encodes the target concept across the dataset, evaluation across settings. with higher scores indicating stronger association. For each For robustness, we use two datasets and one attack method block 𝑖, we record the maximum relevance score 𝑠𝑖 = to evaluate our model: max 𝑓 𝑆(𝑐, D, 𝑖, 𝑓 ) and select the block with the highest 𝑠𝑖 , denote this block by 𝑏 ∗ . We then define a threshold 𝜏 = 𝜆0 × 𝑠 𝑓 , • I2P. Inappropriate Image Prompts (I2P) [47] contains manually curated NSFW text prompts. We select 931 prompts where 𝑠 𝑓 = max 𝑓 𝑆(𝑐, D, 𝑏 ∗ , 𝑓 ), and select all features in related to sexual content to evaluate robustness in nudity 𝑏 ∗ exceeding 𝜏. This procedure retains only highly conceptunlearning. specific features for downstream fine-tuning while preserving • MMA. Model Manipulation Attack dataset (MMA) [48] unrelated features. contains 1,000 adversarial nudity-targeted prompts and is The complete workflow is summarized in Algorithm 1. The designed to evaluate robustness in nudity unlearning and the output, namely the selected block 𝑏 ∗ and feature set F , provides resilience of safety mechanisms under adversarial prompting the basis for feature-level interventions during fine-tuning, conditions. enabling precise suppression of the target concept with minimal • UnlearnDiffAtk. UnlearnDiffAtk [49] is a white-box attack impact on unrelated content. that perturbs text prompts to bypass unlearning and induce A PPENDIX B generation of the undesired concept. Since UnlearnDiffAtk E VALUATION D ETAILS was originally developed for SD v1.5, we use the adversarial prompts generated from SD v1.5 to mount a transferA. Datasets based attack against SDXL-Turbo backbones to assess For computing feature scores and constructing fine-tuning the robustness of C ONCEPTACP and the baselines under datasets, we use a synthetic dataset generated with ChatGPT, adversarial prompt manipulations. where each concept class contains 50 prompts. For comparison with prior unlearning methods on SDXLTurbo, we adopt the EvalIGMU benchmark [34], which provides 250 prompts for Nudity and VanGogh, and 300 prompts for Church and Parachute. Five images are generated per prompt, resulting in 1,250 images for Nudity and

16

B. Evaluation Baselines We compare our approach against representative unlearning methods spanning fine-tuning, model editing, and inferencetime forgetting, and re-implement these methods on SDXL-

Turbo for evaluation. The descriptions and implementation details are as follows. • ESD [9]. A fine-tuning-based unlearning method that use the frozen original model to produce predictions, which are then linearly combined to form a negative guidance signal for updating the new model parameters. For style concepts, ESD fine-tunes the cross-attention parameters, whereas for nudity and object concepts, it fine-tunes the unconditional parameters. We reproduce ESD on SDXL-Turbo under both one-step and four-step fine-tuning settings, and evaluate it correspondingly in one-step and four-step generation. • UCE [42]. A model-editing based unlearning method that requires no additional training. It modifies the parameters of cross-attention via a closed-form update. The work evaluates concept erasure on attributes such as artistic styles, nudity and gender bias. We reproduce UCE on SDXL-Turbo for evaluation. • ConceptPrune [27]. A neuron pruning unlearning method that alters model parameters by removing relevant neurons based on an importance score computed for each weight during generating of a specific concept. This work evaluates concept erasure on attributes such as artistic styles, nudity and gender bias. We reproduce ConceptPrune on SDXLTurbo with four-step generation and use it for both one-step and four-step evaluations. • SAEUron [22]. An inference-time unlearning method that mounts SAEs on the cross-attention blocks of the SD v1.4 U-Net to suppress target concepts. It focuses on forgetting objects and artistic styles rather than safety-related concepts. We reproduce SAEUron on SDXL-Turbo for evaluation.

• [Image Quality] FID Score. Fréchet Inception Distance

(FID) [52] measures the distributional distance between generated and reference images. Lower FID indicates better image quality and fidelity after unlearning. Specifically, the generated images are obtained from the fine-tuned model on D¬𝑐 , while the reference images are from the original model on D¬𝑐 . • [Robustness] ASR. The Attack Success Rate (ASR) is the proportion of adversarial prompts that successfully reactivate the forgotten concept. Lower ASR implies stronger robustness against adversarial reactivation. To further evaluate unlearning precision beyond outputlevel metrics, we report two complementary internal metrics introduced in Section V: • [Unlearning Precision] FO Score. Measures the overlap between the top-𝐾 activated SAE features before and after unlearning for both the target concept and unrelated concepts, capturing whether target-related features are effectively removed while non-target features are preserved. • [Unlearning Precision] FAC Score. Measures the cosine similarity between SAE feature activation distributions before and after unlearning on target and non-target datasets, reflecting distribution-level changes in internal representations.

A PPENDIX C M ORE E VALUATION R ESULTS A. Forgetting Effectiveness In Figure 14, we visualize the forgetting results for four concept-specific tasks under different unlearning methods, reporting both single-step and four-step generation for each task. For each concept, we randomly select four images to C. Metrics illustrate the outcomes. These results provide a more direct We adopt four evaluation metrics to assess the performance visual demonstration of the effectiveness of our method. As of unlearning methods: shown, C ONCEPTACP achieves strong forgetting across all • [Unlearning Effectiveness] FSR. The Forgetting Suc- four concepts. In the parachute task, the model successfully cess Rate (FSR) is the proportion of generated images removes the parachute-shaped object, although some parachute that no longer contain the target concept, as determined fabric remnants remain. by a concept detector. The content detectors used here In Figure 10, we evaluate the one-step fine-tuned model are IGMU BI MULTI C [34] for Nudity and Style, and under up to 30-step image generation. The results show that IGMU MULTI MULTI C [34] for Object. A higher FSR even for multi-step inference, performing fine-tuning on only a indicates stronger unlearning effectiveness. small number of early timesteps is sufficient to achieve effective • [Unlearning Effectiveness] CLIP Acc. Following prior forgetting of the target concept. This observation is consistent work [50], [51], [38], we also use CLIP Accuracy (CLIP with the findings of Tinaz et al. [41], which suggest that early Acc.) to quantify unlearning effectiveness. For each concept timesteps primarily establish coarse object layouts and scene we define two text anchors: a positive anchor 𝐴 that describes composition, and interventions at this stage can effectively the concept and a negative anchor 𝐵 that is unrelated. Given control the generation process. a generated image 𝐼, let Effectiveness Across Diffusion Backbones. Beyond the default SDXL-Turbo setting, we further evaluate the effectiveness 𝑠 𝐴 = CLIP(𝐼, 𝐴), 𝑠 𝐵 = CLIP(𝐼, 𝐵). of C ONCEPTACP under different diffusion backbones and We classify 𝐼 as containing the concept if 𝑠 𝐴 > 𝑠 𝐵 ; otherwise generation step settings. For SDXL, we perform 30-step image it is classified as not containing the concept. CLIP Accuracy generation and collect intermediate representations every five is the fraction of images correctly classified under this rule. diffusion steps during the denoising process for fine-tuning. In our experiments, the positive anchors are “naked”, “van The hyperparameters are set to 𝜆 = 0.5 and 𝑎 = −6 for nudity, gogh style”, “church” and “parachute” for the four tasks 𝜆 = 0.2 and 𝑎 = −6 for Van Gogh, 𝜆 = 1.8 and 𝑎 = −9 for respectively, and the negative anchor is the empty string church, and 𝜆 = 2.0 and 𝑎 = −5 for parachute. The results on “𝑁𝑈 𝐿𝐿”. SDXL are reported in Table III.

17

Fig. 10. The unlearning effects of our one-step fine-tuned model when applied to single-step to 30-step generation.

For SD v1.4, the SAEs are adopted from SAEUron [22]. During fine-tuning, training data are collected every ten diffusion steps, resulting in five sets in total. Both conditional and unconditional SAE reconstructions are included in the fine-tuning process. We use concept-specific hyperparameters, with (𝜆, 𝑎) = (30, −17.5) for Nudity, (8, −15) for VanGogh, (8, −20) for Church, and (15, −120) for Parachute. The unlearning performance is evaluated under 50-step generation, and the results are reported in Table III. Across both diffusion backbones, C ONCEPTACP achieves effective concept forgetting across diverse target concepts, demonstrating generalization beyond SDXL-Turbo. These results suggest that our representation-guided editing strategy is robust to different diffusion architectures and sampling configurations without requiring architecture-specific modifications.

TABLE IX E FFICIENCY COMPARISON FOR SINGLE - CONCEPT UNLEARNING ON SDXL-T URBO . M OD . PARAM . REPORTS THE NUMBER OF MODIFIED U-N ET PARAMETERS , WITH THE PERCENTAGE INDICATING THE PROPORTION OF TOTAL U-N ET PARAMETERS . I NFER . T IME MEASURES RUNTIME FOR GENERATING ONE IMAGE USING 4 DENOISING STEPS , WITH RELATIVE CHANGE WITH RESPECT TO THE ORIGINAL MODEL .

B. Robustness In Table VI, we report the attack success rates (ASR) of different unlearning methods against adversarial prompts generated by UnlearnDiffAtk. Although ConceptPrune achieves the lowest average ASR, our method ranks second overall with one best and one second-best result across the four concept tasks. Together with the results on I2P and MMA, these findings indicate that C ONCEPTACP maintains strong overall robustness under both intrinsic and adversarial evaluations. Compared with Table II, we observe that UCE, whose unlearning performance is relatively weak under benign conditions, achieves one best and two second-best ASRs under UnlearnDiffAtk, whereas SAEUron, which previously demonstrates strong forgetting, performs worse in this setting. The inconsistent performance of UCE and SAEUron across evaluations underscores the limited cross-model transferability of such attacks. Developing SDXL-Turbo-specific adversarial forgetting benchmarks would provide a more comprehensive assessment of unlearning robustness.

forgetting with the scaling factor fixed at 𝑎 = −5. Visual differences across different values of 𝜆 are subtle. Therefore, the optimal 𝜆 for the four forgetting tasks is selected based on quantitative metrics. Effect of the Scaling Factor 𝑎. For the scaling factor 𝑎, we vary it from −3 to −10 while fixing 𝜆 at its optimum, as shown in Figure 12. Small magnitudes, such as, 𝑎 = −3, yield weaker removal of the target concept; for instance, Van Gogh style retains characteristic stars patterns. Increasing |𝑎| strengthens suppression, and for nudity the person may disappear. However, larger magnitudes can degrade overall image quality. In our experiments, we observe that suppression factors approaching −10 noticeably reduce aesthetic quality, which we hypothesize arises from imbalance due to overly strong feature activation or suppression. We therefore do not recommend using 𝑎 < −10. Efficiency and Resource Overhead. For the efficiency and resource overhead, we evaluate our fine-tuning pipeline on a single NVIDIA A800 GPU (80GB PCIe) with CUDA 12.5 and NVIDIA driver version 555.42.06. Constructing the finetuning dataset for 50 prompts takes only 14.07 seconds, which shows that preparing the dataset is highly efficient and does not introduce noticeable overhead. Using precomputed latent input-output pairs, fine-tuning 50 samples for 100 epochs takes 670.3 seconds (11.17 minutes), averaging 6.7 seconds per epoch, excluding data loading and I/O overhead. For comparison, using prompt-based input requires 1,249.7 seconds (20.83 minutes) for the same dataset and epochs, averaging

C. Multi-concept Unlearning In multi-concept unlearning, we present qualitative results in Figure 13, which visualize the effects of sequentially erasing Church and Parachute. D. Ablation Study Effect of the Regularization Parameter 𝜆. In Figure 11, we examine the effect of the regularization coefficient 𝜆 on

18

Method

Mod. Param.

Infer. Time(Δ)

SDXL-Turbo NA 0.6848 s ESD 545.67 MB (21.25%) 0.6840 s (-0.12%) UCE 340.79 MB (13.27%) 0.6864 s (+0.23%) ConceptPrune 6.41 MB (0.25%) 0.6802 s (-0.68%) SAEUron 13.12 MB (2.04%) 1.1872 s (+73.3%) ConceptACP 350.15 MB (13.64%) 0.6768 s (-1.17%)

Fig. 11. Forgetting effect across four unlearning tasks under different 𝜆 values with a=-5. Generation step: 4.

Ori

𝑎 = −3

𝑎 = −4

𝑎 = −5

𝑎 = −6

𝑎 = −7

𝑎 = −8

𝑎 = −9

𝑎 = −10

Nudity

VanGogh

Church

Parachute Fig. 12. Forgetting effect across four unlearning tasks under different 𝑎 values. Rows correspond to different categories, while columns correspond to 𝑎 values. Generation steps: 4.

12.5 seconds per epoch. Latent I/O fine-tuning bypasses the inference stage of the diffusion model and therefore avoids storing intermediate activations, resulting in significantly lower GPU memory consumption and enabling a larger batch size under the same hardware budget. We further observe that enabling automatic mixed-precision (AMP) training reduces the average per-epoch fine-tuning time from 6.7 seconds to 1.34 seconds on SDXL-Turbo and 0.63 seconds on SD v1.4, demonstrating additional efficiency gains. Although the latentbased dataset consumes more storage (71–72 MB) than the prompt-based dataset (28–29 MB), the reduction in fine-tuning time demonstrates a significant efficiency gain. In Table IX, we compare our method with other baselines in terms of modification time, parameter changes and inference time.

19

Notably, SAEUron, which performs unlearning via an external SAE module, incurs a 73.3% increase in inference overhead on SDXL-Turbo. In contrast, although our fine-tuning-based approach requires slightly more time upfront to adapt the model, it introduces no additional inference cost. Overall, while maintaining strong forgetting performance, the time overhead of our method remains practical, requiring less than three minutes to forget a single concept. A PPENDIX D R ELEVANT F EATURES In Table X, we present the cross-attention blocks and feature indices associated with four representative concepts in both the SDXL-Turbo and SDXL models. The SAEs used for these

Fig. 13. Results of single- and multi-concept unlearning on SDXL-Turbo. Generation steps: 4.

two models are shared [33], and the same dataset is employed when we identify the features corresponding to each concept. For Nudity, Church and Parachute, the highly activated features in the same cross-attention blocks are largely consistent across the two models, while for VanGogh, despite one overlapping feature, noticeable differences are observed. In addition, Nudity and VanGogh involve relatively few related features, whereas Church and Parachute are associated with a larger number. We speculate that, unlike abstract concepts such as Nudity or VanGogh, concepts like Church consist of multiple distinct visual components (e.g., spires, walls), which naturally lead to a larger set of associated features. The complexity and diversity of these visual elements make such objects harder to forget without inadvertently affecting unrelated concepts, which also explains why Church and Parachute usually serve as representative concepts for evaluating object-level forgetting. In addition, the lack of sufficiently diverse concept-relevant samples in the dataset used to train the SAEs may further prevent these features from being well concentrated. This observation provides an interesting direction for future exploration.

20

TABLE X T HE RELATED FEATURES FOR THE FOUR MAIN CONCEPTS IDENTIFIED BY C ONCEPTACP ON SDXL-T URBO AND SDXL.

Model

Concept

Location

Feature IDs

SDXL -Turbo

Nudity VanGogh Church

down.2.1 up.0.1 down.2.1

Parachute

down.2.1

#4913, #4001 #3166, #3950 #3799, #2495, #2005, #2831, #3551, #4256, #4008, #1292 #642, #3260, #4525, #3453, #4241, #3851, #4966, #4882, #4764

Nudity VanGogh

down.2.1 up.0.1

Church

down.2.1

Parachute

down.2.1

SDXL

#4913, #4001 #4950, #3166, #2008, #4197, #1566 #3799, #2005, #3551, #2495, #4008 , #2831, #3921 #642, #4525, #3260, #3453, #4882, #4764, #3851 , #1712, #4966

T1 SDXL-

Turbo

ESD

UCE

Concept Prune

T4 SAE Uron

Concept SDXLACP Turbo

ESD

UCE

Concept Prune

Nudity

VanGogh

Church

Parachute

Fig. 14. Comparison of forgetting performance across four unlearning tasks under two generation steps.

21

SAE Uron

Concept ACP

Related documents

• Concept Acupuncturist: An Efficient and Precise Concept Erasure Method for Diffusion Models via Sparse Autoencoders#1447426
• TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models#615311
• TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models#614344
• Enhanced Masked Diffusion Unlearning#1400275
• To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion#927398