DP-SAPF: Saliency-Aware Parameter Fine-tuning of Public Models for Differentially Private Image Synthesis
arXiv:2605.30312v1 [cs.CR] 28 May 2026
Chen Gong University of Virginia
Kecen Li NUS
Zinan Lin Microsoft Research
Abstract
One direction is pretraining synthesizers on public datasets and then finetuning models on sensitive images with Differentially Private Stochastic Gradient Descent (DP-SGD) [3, 7, 8]. However, prior work [1] shows that pretraining synthesizers on public datasets is time-consuming and often fails to synthesize high-resolution images with high quality. For example, DPImageBench [1] reports that PrivImage (a predominant DP image synthesis method using public datasets) [8] achieves only 61.2% downstream classification accuracy on CelebA [9] datasets with 64 × 64 resolution, and the training of the synthesizer is time-consuming. To tackle the challenges, the other direction is to directly use public models for DP fine-tuning [2]. The public models like Stable Diffusion [10] are pretrained on massive datasets using vast computational resources, allowing them to capture robust generative priors, represent complex visual distributions, and generate high-resolution images. These public models are readily accessible via open-source platforms like HuggingFace. However, the parameter-heavy nature of public models makes fine-tuning on sensitive data computationally expensive. For example, Stable Diffusion-v1-51 (a public model) has about 1 billion parameters [10]. To address this issue, existing DP image synthesis methods using public models [1–5,11] contribute two paradigms based on their reliance on fine-tuning. The first category comprises fine-tuning-free methods [4, 5, 12–14], like PE [5]. This approach iteratively guides public models to generate synthetic images that align with sensitive data by selecting and refining the most similar synthetic images. However, these fine-tuningfree methods suffer from quality degradation when the target synthetic distribution significantly diverges from the sensitive images [1, 4]. A second category involves fine-tuning-based methods [2, 3, 11, 15], like DP-LoRA [2]. These methods mitigate high computational costs by using LoRA [16] under DP to reduce the number of trainable parameters in public models. Existing DP synthesis methods neither perform LoRA on all parameters of attention layers fine-tuning [2, 10] nor follow-
Differentially private (DP) image synthesis generates images that preserve the statistical characteristics of a sensitive dataset, enabling sensitive data analysis and usage while providing rigorous guarantees of privacy leakage. Existing methods fine-tune public models using DP Stochastic Gradient Descent (DP-SGD) on sensitive images to generate synthetic images. But full fine-tuning public models on sensitive images is computationally expensive, because current public models typically contain a large number of parameters. Recent work proposes heuristically using Low-Rank Adaptation (LoRA) on all attention-layer parameters of public models to reduce the number of trainable parameters. However, we argue that exhaustive LoRA coverage across all attention-layer parameters is suboptimal in a DP setting, as it leads to noise accumulation and collapse during private training. To address this issue, we propose DP-SAPF, which uses a saliency-aware strategy to identify specific target parameters for LoRA training under DP. DP-SAPF is inspired by the fact that larger gradients signify higher saliency, indicating that these parameters are most critical for the DP learning. Specifically, we feed the sensitive images into public models, compute gradients, and add noise to the gradients to satisfy DP. Then, DP-SAPF identifies the most salient parameters, those exhibiting high gradient magnitudes on sensitive images, for DP fine-tuning. Experiments on four sensitive image datasets show that DP-SAPF improves the utility and fidelity of synthetic images while requiring fewer computational resources than fine-tuning methods without parameter selection.
1
Tianhao Wang University of Virginia
Introduction
Differentially private (DP) image synthesis generates artificial images that maintain the statistical characteristics of sensitive datasets under DP [1]. This allows for dataset sharing while providing rigorous guarantees to reduce privacy leakage of sensitive images. The assistance of public resources has catalyzed the rapid development of DP image synthesis [1–6].
1 https://huggingface.co/stable-diffusion-v1-5
1
ing established practices in non-DP image synthesis [17, 18]. They heuristically optimize only the attention layers [2, 3]. However, this paper claims that exhaustive fine-tuning coverage across all attention layers is suboptimal in a DP setting, as it leads to excessive noise accumulation and training collapses. For example, as shown in Section 3.1, the attention layers within the middle block of the Stable Diffusion exhibit high empirical sensitivity. These layers act as refining encoderextracted features and ensuring semantic alignment with text, where even parameter updates under DP may significantly steer the synthetic performance [10]. Thus, we investigate: how to relieve the training collapse when using public models in DP fine-tuning to optimize the synthetic performance? This challenge has not been explicitly addressed in prior DP image synthesis methods [1–4, 8]. To answer this question, this paper proposes SaliencyAware Parameter Fine-tuning of public models for DP image synthesis (DP-SAPF), which addresses two fundamental challenges in private fine-tuning.
that the parameter matrices closer to the model’s input and output layers exhibit larger gradients, suggesting that these matrices play more critical roles in private training. • This paper proposes DP-SAPF, the first framework to introduce fine-grained dynamically parameter selection into DP fine-tuning. We present the observation that empirical parameter selection from prior works can render DP fine-tuning unstable. DP-SAPF paves the way for future research in DP image synthesis. • DP-SAPF introduces a saliency-aware parameter selection method for DP fine-tuning. By dynamically identifying the most critical parameters based on the gradients derived from different sensitive image datasets under DP, DP-SAPF reduces the number of parameters required for fine-tuning, facilitating efficient and stable DP image synthesis. • Extensive experiments across four sensitive datasets present that DP-SAPF improves utility and fidelity of synthetic images, compared to fine-tuning methods without parameter selection, while consuming fewer computational resources.
• Regarding the selection mechanism. When finetuning public models on sensitive images, some parameters already equipped with general features exhibit very small gradients, indicating that little adjustment is needed, whereas parameters suffering from domain mismatch show increased gradients [19]. In such small-gradient regions, the injected DP noise can easily dominate the true gradient and derail training. DP-SAPF uses these gradients as an analytical proxy. By selecting parameters with large gradients (with DP), it can bypass fragile layers and maintain training stability.
2
Background
This section introduces the background, including the DP notion, diffusion models, current DP image synthesis methods that take advantage of public models, and the LoRA method.
2.1
Differential Privacy
The Differential Privacy (DP) [21] provides a robust mathematical framework for quantifying the privacy risk associated with data leakage. The concept of DP is defined as follows.
• Concerning the structural granularity of selection. Moreover, DP-SAPF transitions from coarse layer-wise tuning to a matrix-wise selection strategy. This design choice exploits the functional heterogeneity within attention modules, where different weight matrices (even within a layer, there are different parts, modeled by matrices) exert disproportionate influence on fine-tuning performance [20]. Thus, matrix-level selection enables a more fine-grained balancing between effective fine-tuning and the perturbations introduced by DP noise.
Definition 1 (Differential Privacy [21]). A randomized algorithm Q achieves (ε, δ)-DP if, for any two neighboring datasets D and D′ (one can be derived from the other by adding or removing a single record), and for all possible output sets O ⊆ Range(Q), the following inequality is satisfied, Pr[Q(D) ∈ O] ≤ eε Pr[Q(D′ ) ∈ O] + δ.
(1)
Here, ε > 0 denotes the privacy budget, where lower values correspond to more stringent privacy protections. The parameter δ ≥ 0 means the probability of a privacy breach beyond the ε guarantee. A fundamental property of DP is its immunity to post-processing [21]: for any data-independent function F, the composition F ◦ Q maintains the same (ε, δ)-DP guarantee. This ensures that downstream analysis performed on synthetic images satisfying DP mechanisms, incurs no additional privacy degradation. DP-SGD. We adopt DP-SGD [22] to satisfy privacy constraints while fine-tuning our DP image synthesizer. This mechanism sanitizes the learning process by first bounding the ℓ2 norm of individual gradients to C and then perturbing
Section 3 presents more motivation and technical details of DP-SAPF. Figure 1 presents the framework of DP-SAPF. We conduct extensive experiments across four sensitive datasets and four public models, presenting that DP-SAPF improves downstream classification accuracy and reduces the FID score, relative to baselines. For instance, under ε = 10.0 and using ‘Stable-Diffusion-v1-51 ’ as the public model, the FID score is reduced by 83.1%, and the accuracy increases by 31.5% compared to DP-LoRA averaged across four sensitive image datasets. We conduct analysis for various privacy budget shows that the synthetic performance is insensitive to the selection ratio within the range of [20%, 50%]. Thus, tuning the selection ratio in practice is straightforward. We observe 2
Table 1: Comparison of existing DP image synthesis methods leveraging public models. The first two methods listed above the horizontal line are fine-tuning-free, whereas the remaining methods use fine-tuning.
Saliency-Aware Parameter Selection
Sensitive Images Calculate gradient for each parameter matrices
Public Model
Private Fine-tuning
Method PE [5] Aug-PE [12] DP-Finetune [7] DP-LDM [3] DP-LoRA [2] DP-SAPF (Ours)
Selected
Parameter Matrix Gradient Select parameter matrices with high gradient
Training Methods Fine-Tuning Free Fine-Tuning Free DP-SGD DP-SGD DP-SGD + LoRA DP-SGD + LoRA
Parameter Selection Full Parameter Attention Attention Saliency-Aware Selection
Year 2024 2024 2023 2024 2025 2026
This work focuses on DP image synthesis methods using public models. Table 1 presents current state-of-the-art popular methods. Previous work neither finetunes all parameters of public models [7] nor follows the fine-tuning practices commonly used in non-DP image synthesis [17, 18]; instead, it fine-tunes only the attention layers [2, 3]. Unlike these fixedlayer strategies, our DP-SAPF introduces a saliency-aware selection mechanism that dynamically identifies the critical parameters based on sensitive images under DP.
Sensitive Images Finetuning parameter matrices Parameter matrices Selected parameter matrices
Figure 1: The framework of DP-SAPF. In the first stage, DPSAPF computes the gradient of sensitive images with respect to each parameter matrix. Parameter matrices exhibiting large gradients are identified as salient and selected for fine-tuning. In the second stage, these selected parameter matrices are updated on the sensitive dataset using DP-SGD with LoRA, while all remaining parameter matrices are kept frozen.
2.2
Diffusion Model
Recent advancements in diffusion models have significantly enhanced image generation capabilities, leading current DP image synthesis methods to primarily adopt diffusion models as the synthesizers [4, 5, 7, 8, 24]. Traditional Diffusion Models. Diffusion models [25] consist of two complementary stages:
the aggregated result with Gaussian noise N(0, σ2d I). At each step, we utilize a Poisson sampling strategy (sampling rate B q) to extract a sub-batch of images Dsub s = {xi }i=1 from the sensitive dataset Ds . The realized batch size B is a random variable following a Binomial distribution, whereas B∗ = qN denotes the expected batch size. Following the standard practice in DP-SGD, B∗ is the fixed normalization constant for the gradient sum to ensure the unbiasedness of the estimator and consistency in privacy accounting [22]. The parameters θ of the synthesizer are updated via the following noisy gradient, ! 1 B C 2 λ ∑ Clip (∇L(θ, xi ),C) + B∗ N(0, σd I) , (2) B∗ i=1
• Forward diffusion, which incrementally perturbs a clean image x0 by adding Gaussian noise over T steps, producing a sequence {xi }Ti=1 that transfers to pure noise. • Reverse diffusion, which iteratively removes noise to recover a clean image from random initialization. In the forward processes, the transition between consecutive noisy images, denoted as p(xt |xt−1 ), follows a multidimensional Gaussian, p p(xt |xt−1 ) = N xt ; 1 − βt xt−1 , βt I , (3) where βt is a hyperparameter that controls the noise variance at step t. Defining ᾱt = ∏ts=1 (1 − βs ), the √ marginal distribution of xt given x0 is, p(xt |x0 ) = N xt ; ᾱt x0 , (1√ − ᾱt )I . This √ allows direct sampling of xt from x0 via, xt = ᾱt x0 + et 1 − ᾱt , et ∼ N (0, I) . The objective of diffusion models learn a denoising network eθ (xt ,t) that predicts the noise et added at each step [25],
where L is the objective function of DP image synthesizer, and λ is the learning rate and σ2d is the variance of Gaussian noise. The Clip(·,C) denotes the norm-clipping operation that scales the gradient ∇L if its ℓ2 norm exceeds the threshold C. Following previous works [1–3, 7], we use Rényi DP (RDP) [23] to track the privacy loss, as detailed in Section A. DP Image Synthesis. By creating synthetic datasets that mimic real-world distributions while satisfying DP, DP image synthesis enables organizations to share high-quality synthetic data without compromising individual privacy.
L = Ex0 ∼D,t∼U{1,T }, et ∼N(0,I) ∥et − eθ (xt ,t)∥22 .
(4)
The D is the dataset of images, and U{1, T } means the uniform distribution over time steps. Once trained, eθ enables image synthesis by progressively denoising Gaussian noise. 3
Ratio
1.0 0.8 0.6 0.4 0.2 0.0
Self-Attention Proportions across UNet Blocks
In principle, one solution for DP image synthesis is to directly pre-train a model entirely on public datasets before applying DP fine-tuning. This has the advantage of eliminating any pretraining–finetuning overlap concerns (as discussed in [32]), but it is computationally expensive and slow [1]. On the other hand, there already exist many high-quality public models released on open platforms. Therefore, DP-SAPF follows the more practical path of directly leveraging these public models. Although pretraining data might partially overlap with sensitive data, we mitigate this concern by evaluating across multiple public models and conducting all comparisons within each individual model. This ensures a fair evaluation.
Gradient Noise
A0 A1 A0 A1 A0 A1 0 A0 A1 A2 A0 A1 A2 A0 A1 A2 D0- D0- D1- D1- D2- D2- M-A U1- U1- U1- U2- U2- U2- U3- U3- U3-
Layer Position Figure 2: Each cell shows the ratio of noise scale and gradient norm for self-attention on CIFAR-10. The public model is ‘Stable-Diffusion-v1-5’. ‘Di-A j’ = Down block i, attention layer j; ‘Ui-A j’ = Up block i, attention layer j. ‘M-A0’ = Middle block, attention layer 0.
3 Latent Diffusion Models. Latent diffusion models [10] are a variant of diffusion models that operate in a latent space instead of in pixel space. This approach uses a pretrained autoencoder consisting of an encoder E and a decoder R. Specifically, the encoder E maps a clean image x0 into a lowerdimensional latent representation z0 = E(x0 ). The forward diffusion process, consistent with the logic in Equation (3), is then applied to z0 to produce a sequence of latent variables {zt }Ti=1 √ variable at step t can be expressed as, √ . The latent zt = ᾱt z0 + et 1 − ᾱt , et ∼ N(0, I). The denoising network eθ (zt ,t, c) is then trained to predict the noise et added to the latent vector, often incorporating a conditioning vector c (e.g., text or class labels) via an attention mechanism [26]. The training objective is,
This section elaborates on our methods, DP-SAPF, including the motivation, technical details, and privacy analysis.
3.1
Motivation
Existing works [2, 3] restrict fine-tuning to attention layers, a heuristic inherited from non-DP image synthesis methods [17, 18]. In contrast, DP-SAPF posits that exhaustive fine-tuning of all attention layers is suboptimal under DP, as such extensive parameter coverage exacerbates noise accumulation, making training collapse. To the best of our knowledge, this issue has not been explicitly addressed in prior DP image synthesis methods [1–4, 8]. Specifically, as shown in Figure 2, the attention layers of the middle block [10] exhibit a high ratio of noise scale and gradient norm. These layers are essential in processing feature refinement and text-to-image alignment. Under DP, minor parameter updates can steer generative performance. As shown in Table 2, removing fine-tuning of the mid-block attention layer leads to a substantial improvement in synthetic performance. However, identifying sensitive layers is time-consuming, as public models typically contain a large number of parameters and complex architectures. Even worse, this strategy is not DP-compliant, as manual parameter selection relies on repeated inspection of sensitive data [33]. This motivates us to use gradient-based metrics to analytically select parameters that improve DP fine-tuning. The first question is how to select the parameters. The motivation for using gradient as a selection saliency metric is primarily rooted in the Signal-to-Noise Ratio (SNR) during DP training [34]. In DP-SGD, a fixed noise is injected into all trainable parameters to satisfy DP. Parameters with small gradients possess a low SNR, meaning their learning signal is easily drowned out by the noise. In contrast, parameters with large gradient magnitudes exhibit higher noise resilience, as their signal is strong enough to remain impactful even after perturbation. By selecting these high-gradient parameters, DP-SAPF ensures that the optimization is concentrated on a high-SNR subspace, where the learning gradient dominates the noise. Section 5.2 shows that DP-SAPF can adaptively
LLDM = E(x0 ,c)∼D,t∼U{1,T },et ∼N(0,I) ∥et − eθ (zt ,t, c)∥22 . (5) After training, a latent sample zT is drawn from a Gaussian distribution and iteratively denoised to recover z0 . Finally, the high-resolution image is reconstructed by passing the denoised latent through the decoder, x̃ = R(z0 ). By operating in this compressed space, latent diffusion models significantly reduce the training and sampling costs while preserving high perceptual quality. This efficiency makes them a preferred backbone for recent DP synthesis methods, such as DP-LDM [3] and DP-LoRA [2].
2.3
Methodology
Threat Model
We assume data providers hold sensitive images, e.g., medical images, and sharing them directly poses privacy risks. Several approaches propose generating datasets to replace real ones [27]. However, this method does not fully mitigate privacy concerns, as adversaries still infer sensitive information using synthetic images. DP-SAPF enforces imagelevel DP, a general protection against various inference attacks targeting individual images, such as membership inference [28]. This paradigm has been widely explored in prior works [1, 4, 12–14, 29–31], and we follow the same approach. 4
Table 2: Acc (%) and FID of synthetic images for CIFAR-10, using ‘Stable-Diffusion-v1-5’ as the public model, under ε = 10. ‘DP-LoRA w/o Middle’ means using DP-LoRA to finetune the public model but excluding the middle block. ‘LoRA w/o DP’ means a non-DP LoRA fine-tuning.
Algorithm 1: Saliency-Aware Parameter Selection. Input :Sensitive dataset Ds with size N and estimated size N ∗ ; the public model Mpub ; the noise scale σs ; the weight matrices candidate Θfull = {Wi }Ki=1 . // Gradient Calculation 1 Init clip gradient set G = ∅; 2 while xi ∈ Ds do 3 Calculate gradient for xi on M using Equation (5) and obtain gi = [vec(∇W1 Li ), . . . , vec(∇WK Li )]; 4 Using hEquation (6) to clip the gi and i obtain
Metrics DP-LoRA DP-LoRA w/o Middle LoRA w/o DP DP-SAPF Acc FID
13.3 384.3
69.7 31.3
82.4 12.9
74.6 24.6
avoid fine-tuning low SNR parameters in the public model under DP, preventing the model from being derailed by the accumulation of noise in low-signal components. The second question concerns the structural granularity of the selection. Whereas conventional approaches typically finetune entire layers [3, 10], DP-SAPF instead operates at a matrix-wise granularity. This design choice is grounded in the functional heterogeneity within attention modules. Recent work [20] shows that the query, key, and value matrices contribute unequally to fine-tuning performance, with certain matrices (e.g., value metrics in attention [10]) exerting substantially greater influence than others. Thus, matrix-wise selection enables a more favorable balance between model expressivity and the distortions introduced by DP. Experiments conducted in Section 5.2 shows that matrixwise selection achieves better synthetic performance than layer-wise selection, on studied sensitive image datasets.
3.2
ḡi = ΓCs (gi ) W , · · · , ΓCs (gi ) W 1
K
;
G = G ∪ ḡi ; end 7 Init average clipped gradient set S = ∅; 8 while Wi ∈ Θ do 9 Calculate average clipped gradient Sk (Ds ) = N1∗ ∑Ni=1 ḡi,k , where, ḡi,k = ΓCs (gi ) W ; k 10 Obtain G̃s,k by using Equation (7); 11 S = S ∪ G̃s,k ; 12 end // Parameter Selection 13 Based on the norms of the noisy gradient S = [G̃s,1 , · · · , G̃ ], we selectthe top-c matrices: s,K K ′ G̃s,k k=1 ; Θ = arg top- c 5
6
Output :The saliency-aware parameter matrix set Θ′ .
Saliency-Aware Parameter Selection
joint clipping operator ΓCs to the concatenated gradient vector gi = [vec(∇W1 Li ), . . . , vec(∇WK Li )], where vec(·) denotes the flattening operator. The joint clipping operator ΓCs (gi ) calculates the average gradient ḡi = ΓCs (gi ) as follows, s K ∥gi ∥2 ḡi = gi max 1, , ∥gi ∥2 = ∑ ∥vec(∇Wk Li )∥2 . Cs k=1 (6) The clipped gradient ḡi is then partitioned back into its per-matrix components ḡi,k = ΓCs (gi ) W . For each matrix k Wk ∈ Θfull , we compute the average clipped gradient Sk (Ds ) over the sensitive dataset D of size N,
As introduced in Section 3.1, DP-SAPF selects a subset of salient weight matrices from the public model by analyzing the gradients induced by sensitive images. Following prior works [2, 3, 17], DP-SAPF further restricts matrix-wise selection to the attention layers. Their parameters exhibit strong responsiveness to task-relevant information while maintaining reliable training behavior, making them an effective and robust subset for adaptation [35]. Fine-tuning feed-forward or normalization layers tends to either achieve marginal improvements or introduce training instability [17]. Each attention layer comprises a set of projection matrices that parameterize the query, key, and value transformations, denoted as Wq , Wk , Wv . These head-specific projections are usually followed by an output projection that linearly combines the concatenated head outputs [10]. Our selection targets these three weight matrices in attention layers. Let the public model consist of K weight matrices Θfull = {W1 , W2 , . . . , WK } that we aim to select. For each sample in the sensitive image dataset xi ∈ Ds , Ds = {xi }Ni=1 , we perform a forward pass to compute the task loss LLDM , as defined in Equation (5), followed by backpropagation to obtain the gradients with respect to every candidate matrix, gi = ∇W j Li j = 1, · · · , K . To bound sensitivity, we apply a
Sk (Ds ) =
1 N ∑ ḡi,k , N ∗ i=1
(7)
where N ∗ approximates sensitive image dataset size N as introduced in Appendix B. The private selection criterion is based on the noisy average gradient, G̃s,k = Sk (Ds ) + N 0, σ2s ∆2s I , (8) where σ2s is a hyperparameter controlling the scale of the injected Gaussian noise, and the global sensitivity is ∆s = Cs /N ∗ . In Theorem 1, we prove this procedure satisfies DP. 5
Theorem 1. The averaged clipped gradient query [S1 (Ds ), ..., SK (Ds )] has global ℓ2 sensitivity ∆s = Cs /N ∗ . For any Rényi order α > 1, adding Gaussian noise N 0, σ2s ∆2s I to Sk (Ds ) for the weight matrices {W1 , . . . , WK } ensures that the resulting mechanism satisfies (α, γs )-RDP for γs = 2σα2 .
Algorithm 2: The workflow of DP-SAPF. Input :Public model M with parameter matrices Θ′ ; the number of matrix in Θ′ , H; sensitive dataset Ds ; learning rate λ; finetuning batch size B and estimated size B∗ ; the noise scale σd ; the clip bound C; finetuning iteration td . // Private Fine-tuning 1 Init h = 0; ′ 2 Init Θtrain = {(Ak , Bk ) | Wk ∈ Θ , k = [1, H]}, where Ak ∼ N(0, I) and Bk is initialized to zero; 3 while h < td do B 4 Sample subset Dsub s = {xi }i=1 from Ds ; 5 Init gradient set G = ∅; 6 for xi ∈ Dsub s do 7 Calculate the gradient for xi , using Equation (5), (∇Ak Li , ∇Bk Li ), ∀ Wk ∈ Θ′ ; 8 Calculate the aggregate using L gi = Wk ∈Θ′ vec ∇Ak Li ⊕ vec ∇Bk Li ; 9 Using Equation (11) to clip gi and obtain ḡi ; 10 G = G ∪ ḡi ; 11 end 12 Calculate the privatized aggregated gradient, g p = λ B1∗ ∑ḡi ∈G ḡi + BC∗ N(0, σ2d I) ; 13 The g p is used to update (Ak , Bk ) for all Wk ∈ Θ′ ; 14 h = h + 1; 15 end ′ ′ 16 Obtain Wk for all Wk ∈ Θ using Equation (9); Output :The well-finetuned public model M ′ .
s
We provide the proof of Theorem 1 in Section B. Based on the norms of the noisy gradient [G̃s,1 , · · · , G̃s,K ], we select the top-c matrices for fine-tuning. Formally, the selected parameter subset is defined as, K Θ′ = arg top- c G̃s,k k=1 , where c denotes the ratio specifying how many matrices with the largest noisy gradient norms are selected for fine-tuning. Algorithm 1 describes the workflow of our saliency-aware parameter selection mechanism, which determines the parameter matrices to be fine-tuned on sensitive data.
3.3
DP Fine-Tuning
Building on prior work on public model fine-tuning (both DP and non-DP image synthesis) [2, 10, 17, 18], this work adopts LoRA as an efficient adaptation mechanism under DP, given the large parameter counts of public models. Fine-tuning all parameters is time-consuming. LoRA [16] trains a public model by injecting low-rank trainable updates into its weight matrices, instead of optimizing all model parameters. By limiting the number of trainable parameters, LoRA improves computational efficiency and facilitates privacy-preserving optimization, while maintaining the expressive power of the public model. The following theorem shows that given a fixed clipping threshold C and noise multiplier σd , DP-SGD applied to full-parameter fine-tuning and LoRA fine-tuning satisfy the same level of (ε, δ)-DP.
and Bk are initialized asymmetrically: Ak is typically sampled from a Gaussian distribution, while Bk is initialized to zero [16]. This formulation reduces the number of trainable parameters from mk ×hk to r ×(mk +hk ). All original weights in Θfull remain frozen. The set of trainable parameters is restricted to the adapters associated with the salient subset, Θtrain = {(Ak , Bk ) | Wk ∈ Θ′ }. In DP-SAPF, during each training iteration, for every samB ple xi in a sensitive sub-batch Dsub s = {xi }i=1 , we compute the gradients with respect to all trainable LoRA parameters associated with the selected matrices Θ′ . For each selected weight matrix Wk ∈ Θ′ , LoRA introduces a pair of low-rank parameters (Ak , Bk ) Given the latent diffusion objective in Equation (5), we compute the per-sample gradients, (∇Ak Li , ∇Bk Li ), ∀ Wk ∈ Θ′ . To satisfy DP, we construct an aggregate gradient vector by vectorizing and concatenating the gradients of all selected matrix adapters, which is defined as follows, M gi = vec ∇Ak Li ⊕ vec ∇Bk Li , (10)
Theorem 2. Given a fixed clipping threshold C, noise multiplier σd , sampling rate q, and number of iterations td , the (ε, δ)-DP guarantee of DP-SGD remains identical regardless of whether the training is performed on the full parameter set Θ or a low-rank subspace ΘLoRA . We provide the proof of Theorem 2 in Appendix Section B. Based on the saliency-aware selection procedure introduced in Section 3.2, we obtain a subset Θ′ = {W1 , . . . , WH } ⊆ Θfull , which contains the H = c × K matrices. LoRA updates are then applied only to matrices in Θ′ . Formally, for each selected matrix Wk ∈ Θ′ with shape mk × hk , where mk and hk are the dimensions of the weight matrix Wk , LoRA represents the adapted weight as, W′k = Wk + ∆Wk ,
∆Wk = Ak Bk ,
(9)
Wk ∈Θ′
where Ak ∈ Rmk ×r and Bk ∈ Rr×hk are low-rank matrices, and r ≪ min(mk , hk ) denotes the rank of the decomposition. Ak
where vec(·) denotes the vectorization operator and ⊕ signifies concatenation. To bound the global ℓ2 -sensitivity of the 6
Table 3: Summary of sensitive image datasets, including resolution, data split statistics, and the number of categories. Datasets CIFAR-10 OCTMNIST CelebA Camelyon
Training 45,000 97,477 162,770 302,436
Validation 5,000 10,832 19,867 34,904
Test 10,000 1,000 19,962 85,054
Resolution 32×32 128×128 256×256 96×96
Table 4: Summary of public models used in our experiments. Public Model Stable-Diffusion-v1-51 Stable-Diffusion-2-1-base2 Realistic-v63 Prompt2med4
Category 10 4 2 2
Resolution 512×512 512×512 896×896 512×512
Size 1B 1B 1B 1B
Year 2022 2022 2024 2024
σd , convert the composed RDP cost (α, γs + γd (σd )) to its corresponding (ε, δ)-DP guarantee [36], and select the smallest σd that satisfies the given privacy budget. The definitions and more details of RDP in DP-SAPF are provided in Section A. Section 5.3 shows that under our default configuration (σs = 5, ε = 10), the privacy overhead of the parameterselection stage accounts for only 0.17% of the total privacy budget. This indicates that the parameter selection is highly budget-efficient, preserving the vast majority of the privacy allowance for the subsequent fine-tuning phase.
model update, the combined gradient vector gi is constrained by a clipping threshold C, ∥gi ∥2 ḡi = gi max 1, , (11) C where ∥gi ∥22 = ∑Wk ∈Θ′ ∥vec(∇Ak Li )∥22 + ∥vec(∇Bk Li )∥22 . Following the clipping step, the gradients are averaged across the sub-batch and perturbed with Gaussian noise as defined in Equation (2) to satisfy the (ε, δ)-DP requirement. These privatized gradients are then used to update only the LoRA parameters (Ak , Bk ) for all Wk ∈ Θ′ . After private training, (Ak , Bk ) are incorporated into the effective weights W′k in Equation (9) for all Wk ∈ Θ′ . Detailed text-processing is provided in Section C.2, and the workflow of DP-SAPF is summarized in Algorithm 2.
3.4
Source Stability AI Stability AI Hugging Face Hugging Face
4
Experimental Setup
Baselines. We elaborate on current DP image synthesis methods in Section 7. This paper focuses on methods that leverage public models. Built on this constraint, we select five predominant DP image synthesis methods that use public models to aid DP image synthesis, including PE [5], Aug-PE [12], DP-LDM [3], DP-LoRA [2], and DP-Finetune [7]. PE and Aug-PE are fine-tuning free methods. Besides, DP-LDM, DPLoRA, and DP-Finetune are fine-tuning-based methods. Implementations. All experiments are implemented with Python 3.8 on a server with 4 NVIDIA GeForce A6000 Ada and 512GB of memory. We aim at conditional generation for these datasets (i.e., each generated image is associated with the class label). Following practical adoption in DPImageBench [1], we set DP parameter δ = 1/(Npriv × log Npriv ), where Npriv means the number of samples in training private datasets as presented in Table 3. Section C.1 provides further details on our hyperparameter settings. Investigated Datasets. We perform experiments on four image datasets CIFAR-10 [37], OCTMNIST [38], CelebA [9], and Camelyon [39]. The investigated datasets are prevalently used in previous DP image synthesis methods [7, 24]. CIFAR-10 comprises 10 classes of natural images with the resolution of 32 × 32. OCTMNIST consists of 109,309 retinal Optical Coherence Tomography (OCT) images categorized into 4 classes with the resolution of 128 × 128 [38]. CelebA contains over 202,599 facial images of 10,177 celebrities, each annotated with 40 attributes with the resolution of 256 × 256; following prior work [1, 5, 24], we use the “Gender” attribute to classify images as male or female. Camelyon includes 455,954 histopathological image patches of human tissue with the resolution of 96 × 96, labeled based on the presence of at least one tumor cell pixel. As shown in Table 3, all datasets are split into training, validation, and test sets. It
Privacy Analysis
In DP-SAPF, two components consume the privacy budget: (1) the parameter-selection stage that computes per-sample gradients, clips them to the ℓ2 bound Cs , and adds Gaussian noise; and (2) the fine-tuning stage that trains the selected parameters using DP-SGD. The parameter-selection stage is a single Gaussian mechanism that satisfies (α, γs )-RDP. During DP-SGD, each iteration samples a sub-batch with ratio q and injects Gaussian noise with scale σd , which corresponds to an SGM [23], as introduced in Section A. According to the RDP analysis of SGM [22, 23], the td fine-tuning iterations incur an RDP cost (α, γd (σd )), where γd depends on q, td , and σd . Because RDP composes linearly [36], the overall privacy guarantee of DP-SAPF is (α, γs + γd ). To better understand how the privacy budget is distributed across the two stages, we define the privacy budget ratios as, rs = γs /(γs + γd ), rd = γd /(γs + γd ), which measures the fraction of the total RDP cost consumed by the parameterselection stage and the DP-SGD fine-tuning stage, respectively. These ratios quantify the extent to which each stage contributes to overall privacy protection. To ensure that DP-SAPF satisfies a target (ε, δ)-DP, we choose the privacy parameters in three steps. (1) We first set the noise scale σs to determine the RDP cost (α, γs ) for the feature-query stage. (2) We fix the number of fine-tuning iterations td and the sampling ratio q, under which the RDP cost of DP-SGD becomes a function of the noise scale σd , denoted by (α, γd (σd )). (3) We search over different values of 7
Table 5: FID and Acc (%) of synthetic images generated by different public models with DP-SAPF and baselines under ε = {1, 10}, on CIFAR-10, OCTMNIST, CelebA, and Camelyon. The best values are highlighted in bold in each column. Public Model
SD-v1-5
SD-2-1-base
Realistic-v6
Prompt2med
Method
CIFAR-10 ε=1 ε = 10 FID Acc FID Acc
OCTMNIST ε=1 ε = 10 FID Acc FID Acc
CelebA ε=1 ε = 10 FID Acc FID Acc
Camelyon ε=1 ε = 10 FID Acc FID Acc
PE Aug-PE DP-LDM DP-LoRA DP-Finetune DP-SAPF PE Aug-PE DP-LDM DP-LoRA DP-Finetune DP-SAPF PE Aug-PE DP-LDM DP-LoRA DP-Finetune DP-SAPF PE Aug-PE DP-LDM DP-LoRA DP-Finetune DP-SAPF
16.4 24.0 240.4 366.0 191.9 32.1 17.4 25.6 119.4 301.1 155.5 32.1 28.5 25.6 143.2 283.2 200.7 35.6 24.0 21.3 265.0 374.5 203.5 30.2
132.5 143.2 398.2 320.3 397.8 85.4 135.6 132.4 304.5 377.3 394.2 89.3 164.9 177.3 234.1 300.8 326.4 161.0 84.4 81.5 277.3 282.2 255.5 108.1
44.6 41.1 312.8 244.7 277.9 25.0 42.8 40.2 299.3 231.5 268.3 27.2 37.9 32.1 294.3 233.2 251.4 23.5 44.5 47.3 214.7 299.2 224.9 28.4
62.3 63.1 405.8 513.7 242.8 139.0 68.4 69.4 388.8 440.4 290.3 70.4 84.8 79.4 384.3 487.2 321.9 125.2 41.2 39.0 376.3 388.3 343.9 96.3
67.5 44.8 13.0 12.8 15.0 73.1 66.3 61.3 15.3 11.7 14.4 69.9 63.5 61.9 32.3 12.2 11.9 68.2 64.8 65.5 13.2 11.4 13.7 70.8
10.9 12.9 183.8 384.3 145.2 26.6 12.1 11.1 142.1 289.4 124.0 27.2 23.3 21.1 122.2 199.3 184.7 30.8 61.8 35.0 231.8 265.9 146.3 25.4
66.9 49.8 14.6 13.3 14.2 74.6 68.5 66.9 16.1 14.9 16.2 72.3 65.6 63.2 36.3 13.9 12.3 68.0 37.5 67.7 12.1 13.2 12.5 72.8
27.1 28.4 25.0 25.0 31.9 44.6 25.3 26.0 25.0 25.0 25.0 41.4 25.0 25.0 26.3 25.0 25.0 25.0 36.3 36.0 25.0 25.0 25.4 42.4
104.2 92.1 347.8 347.0 345.2 77.9 112.1 110.3 317.9 333.5 298.1 80.0 133.5 132.0 302.8 267.4 307.2 81.7 39.5 48.2 197.4 243.7 203.9 99.1
68.0 70.2 57.3 61.4 60.8 84.2 70.2 73.2 60.1 59.3 61.2 85.0 71.4 74.5 62.2 64.3 59.3 88.5 70.1 72.2 61.4 50.0 57.4 83.9
40.4 42.5 281.6 447.7 274.6 23.6 36.8 38.3 220.5 200.4 231.6 24.0 38.9 40.3 238.4 241.8 195.8 19.2 39.7 40.0 200.6 235.5 195.3 26.0
64.1 65.6 61.0 66.3 61.7 90.2 73.0 74.7 63.1 64.1 66.2 90.5 73.8 70.3 56.7 61.5 60.0 92.1 68.4 72.8 65.8 59.9 60.0 88.5
69.7 70.1 60.4 56.3 57.6 72.5 66.6 65.2 52.3 50.2 60.0 74.1 65.5 69.6 50.6 52.4 58.2 79.8 65.3 70.1 58.9 54.7 51.9 78.9
55.9 51.9 248.8 437.1 187.6 145.0 60.4 63.4 300.0 377.9 188.7 55.3 79.5 85.5 343.2 399.3 285.7 118.6 40.7 40.4 326.4 321.7 288.5 89.4
50.4 64.9 69.8 62.1 65.3 79.9 60.9 62.3 62.7 60.2 59.3 78.2 64.3 66.2 51.7 56.5 57.7 71.7 70.2 71.3 59.0 50.5 53.4 80.1
• RQ1. Does DP-SAPF outperform baseline methods in generating high-quality images across the four diverse datasets?
is noticed that some prior works [8, 24] use downsampled versions of the CelebA and Camelyon datasets at a resolution of 32 × 32. In contrast, DP-SAPF uses the original-resolution images, making our experiments more challenging than those in these earlier studies [1]. Public Models. We evaluate our method using four widely-used public models, including ‘Stable-Diffusionv1-51 ’, ‘Stable-Diffusion-2-1-base2 ’, ‘Realistic-v63 ’, and ‘Prompt2med4 ’. Given the dominance of the stable diffusion library5 [17, 18], our selection includes two official stable diffusion base models and two specialized finetuned versions: ‘Realistic-v6’ for photorealism and ‘Prompt2med’ for medical imaging. Table 4 presents the details of public models. Evaluation Metrics. We assess both fidelity and utility using two established metrics [1,24,40]: Fréchet Inception Distance (FID) and downstream classification accuracy. Specifically, we generate 60,000 synthetic images for evaluation. Our experimental setup is grounded in the standardized evaluation framework proposed by DPImageBench [1].
5
26.1 30.2 25.0 25.0 30.9 46.2 27.4 28.1 26.3 25.0 26.3 42.9 26.6 26.4 25.0 27.3 25.0 38.5 39.1 36.8 26.2 25.0 27.5 43.4
• RQ2. How does saliency-aware parameter selection in public models benefit the DP image synthesis? • RQ3. How do the hyperparameters introduced by DP-SAPF affect the performance of synthetic images?
5.1
Performance of Synthetic Datasets (RQ1)
This RQ evaluates the utility and fidelity of synthetic images generated by DP-SAPF relative to five baselines. Using the four datasets detailed in Section 4, we perform a comparative analysis under privacy budgets of ε = {1, 10}. Table 5 presents the FID and Acc (%) of DP-SAPF and baselines. Figure 3 shows visualization examples of comparison between synthetic images generated by DP-SAPF and real images across four studied datasets, under ε = 10. We summarize the takeaways from this RQ as follows. In most cases, DP-SAPF generates synthetic images with higher utility and fidelity compared to baselines. Table 5 shows that DP-SAPF achieves the highest accuracy across nearly all settings; the sole exception is on OCTMNIST under ε = 1 when using ‘Realistic-v6’ as the public model, however, all methods do not work in this case. DP-SAPF achieves an accuracy of 25.0%, trailing the top-performing baseline (which is 26.3%) by only a narrow margin. Besides, in terms of FID, DP-SAPF achieves lower (superior) scores on CIFAR-10 and CelebA datasets compared to
Experiment Analysis
This section investigates the effectiveness of DP-SAPF through answering three research questions (RQs) as follows. 2 https://huggingface.co/Manojb/stable-diffusion-2-1-base 3 https://huggingface.co/SG161222/Realistic_Vision_V6.0_ B1_noVAE 4 https://huggingface.co/Nihirc/Prompt2MedImage 5 https://huggingface.co/stabilityai
8
OCTMNIST
CelebA
Camelyon
Real
Synthetic
CIFAR-10
Figure 3: Visualization examples of comparison between synthetic and real images across four datasets. The synthetic images are generated using DP-SAPF, under the privacy budget ε = 10.0. Table 6: The Acc (%) and FID of synthetic images for CIFAR-10 and CelebA, under ε = 10.0. We only finetune the query parameter matrices of cross-attention. The public model is ‘Stable-Diffusion-v1-5.’ Datasets
DP-LoRA FID Acc
DP-LDM FID Acc
DP-Finetune FID Acc
CIFAR-10 CelebA
25.4 34.0
36.2 40.6
132.7 198.0
72.0 85.8
66.0 80.8
the public model is highly complex, manually selecting the parameter matrices for fine-tuning is labor-intensive and consumes additional privacy budget [33]. However, DP-SAPF requires no manual selection and automatically avoids training collapse. For example, under ε = 10.0 and using ‘Stable-Diffusion-v1-5’ as the public model, the FID is reduced by 83.1%, and the accuracy increases by 31.5% compared to DP-LoRA averaged across four sensitive datasets.
32.3 61.1
5.2
fine-tuning-free methods such as PE and Aug-PE. Specifically, since these fine-tuning-free methods preserve the pretrained weights without modification, the synthetic images strictly adhere to the strong prior distribution of the public model, resulting in high visual fidelity and lower FID scores. However, they lack deep adaptation to the label information of specific downstream tasks. When the downstream domain, such as the OCTMNIST medical imaging dataset, exhibits a distribution shift from the natural images used in pretraining, PE-based methods struggle to capture the critical discriminative features necessary for effective classification, leading to inferior accuracy compared to DP-SAPF. DPImageBench [1] reports a similar phenomenon, noting that while the FID of synthetic images from PE is lower than that of state-of-the-art methods, its Acc is also consistently lower.
Strengths of Parameter Selection (RQ2)
We explore the strengths of leveraging the parameter-selection mechanism introduced by DP-SAPF to select critical parameter matrices for private finetuning. We compare the performance of DP-SAPF with the following five invariants of DP-SAPF under ε = 10.0. In this RQ, we use the ‘StableDiffusion-v1-5’ as the public model. • ‘Random’ denotes randomly selecting the parameter matrices without the saliency-aware mechanism introduced by DP-SAPF, to study the importance of the saliency-aware. • ‘Noisy’ indicates that the unselected parameters are replaced with random values, allowing us to analyze how much the original generative capability of the public model contributes to the final synthetic performance.
Fine-tuning all attention layers within public models often leads to training instability or model collapse. Table 6 shows that DP-LoRA, DP-LDM, and DP-Finetune are consistently poor-performing when fine-tuning all attention layers within the public models. As discussed in Section 3.1, some attention layers in public models are highly sensitive yet contribute little to fine-tuning; updating these layers can easily destabilize the model and cause it to collapse. Through careful manual selection of parameter matrices, Table 6 shows that restricting fine-tuning to the query matrices of the cross-attention attention layers [10] consistently enhances the quality of the generated images. However, because
• ‘w/o LoRA’ means that after the saliency-aware mechanism, we finetune the select parameter matrices directly using DP-SGD, without incorporating LoRA. • ‘Layer-Level’ means conducting layer-wise selection, instead of the matrix-wise selection, to validate our analysis in Section 3.1. • ‘All Parameter (Param.)’ indicates that the saliency-aware selection mechanism is applied to all fine-tunable parameter matrices, rather than being limited to the attention layers. 9
Random
CIFAR-10
Layer-Level
80
150 FID
146.3
100
121.6
100
76.3
71.3 65.3
63.0 30.1
50
0
0
CelebA 100
74.6
60
69.3 55.2 41.0
40 20
28.7
23.6
All Param.
CIFAR-10
188.9
Acc (%)
200
26.6
w/o LoRA
CelebA 200
231.5
FID
Noisy
16.2
Acc (%)
DP-SAPF
90.2
88.8 76.3
75 60.4
64.3
64.0
50 25
16.0
0
0
Figure 4: FID and Acc (%) of DP-SAPF and five variants with ε = 10.0. ‘Random’ indicates no saliency-aware mechanism. ‘Noisy’ replaces the unselected parameters with random values. ‘w/o LoRA’ finetunes the select parameter matrices directly using DP-SGD, without incorporating LoRA. ‘Layer-Level’ means conducting layer-wise selection. ‘DP-SAPF’ is our work that conducts matrix-wise saliency-aware parameter selection and finetunes the selected parameters using DP-SGD with LoRA. 34
Acc (%)
80 75.9
32
60
30
40 20 0
100
CelebA
27
90.6
80
Acc FID
0.2 1.0 5.0 10 15 Privacy Budget ε
25.9
20
26
20 0
0.2 1.0 5.0 10 15 Privacy Budget ε
74.6
150 100
20 22.7
50
23
CelebA 250
90.2
80 60
40
24 Acc FID
200 100
Acc FID
60
25
40 28
80
26
60
CIFAR-10
100
40
200
100
20 23.6
25.9
0
20
5 10 20 30 40 50 60 70 Selection Ratio c (%)
150
Acc FID
0
FID
CIFAR-10
100
50
5 10 20 30 40 50 60 70 Selection Ratio c (%)
Figure 5: The Acc (%) and FID of synthetic images generated by DP-SAPF for the sensitive image datasets, CIFAR-10 and CelebA, under privacy budgets ε = {0.2, 1.0, 5.0, 10.0, 15.0, 20.0}, and selection ration c = {5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%}. The public model is ‘Stable-Diffusion-v1-5’. The dashed lines indicate the best value. Figure 4 presents the FID and Acc (%) of DP-SAPF and five variants. We summarize the takeaways of this RQ as follows.
vious studies [2, 35] show that applying LoRA finetuning with DP-SGD reduces the injected noise compared to standard DP-SGD training. Our experiments shown in Figure 4 empirically validate this claim. On the CIFAR-10 and CelebA sensitive image datasets, our method produces superior image quality compared to applying DP-SGD directly to the selected parameter matrices during fine-tuning.
Matrix-wise saliency-aware selection benefits the synthetic performance. In Figure 4, we first observe that randomly selecting the parameter matrices leads to noticeably worse synthetic images compared to saliency-aware selection. In CIFAR-10, the FID and Acc are 146.3 and 16.2, which are far worse than the 26.6 and 74.6 achieved by DP-SAPF. Random selection is unlikely to identify parameters that meaningfully influence the model’s generative behavior, causing the DP-SGD noise to be injected into uninformative or even irrelevant parts of the model, ultimately harming both fidelity and utility. Additionally, we observe that matrix-wise selection outperforms layer-wise selection, which validates our analysis in Section 3.1 that different parameter matrices contribute unequally even within the same layer. Matrix-wise selection enables DP-SAPF to isolate and finetune the most influential matrices, reducing unnecessary noise injection into less relevant components. On CelebA, the FID and Acc for layer-wise selection are 121.6 and 64.0, which are greatly worse than the 23.6 and 90.2 achieved by matrix-wise selection.
5.3
Hyperparameter Analysis (RQ3)
This RQ explores how different hyperparameter settings affect the system, considering the following two perspectives. • Privacy allocations. We evaluate the performance of DP-SAPF under different parameter selection ratios c = {5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%}, under the privacy budget ε = 10.0. The selection ratio determines the ratio of matrices with the largest noisy gradient norms that are selected for subsequent private fine-tuning from the parameter matrices pool. • Privacy budget. We evaluate the utility and fidelity of the synthetic images under ε ∈ {0.2, 1.0, 5.0, 10.0, 15.0, 20.0}. To ensure a consistent privacy allocation ratio across settings, we adjust the noise scale σs in the parameter-selection
LoRA fine-tuning with DP-SGD outperforms directly applying DP-SGD to the selected parameter matrix set. Pre10
Table 7: Selective fine-tuning configuration across different selection ratios c. Columns grouped by U-Net blocks (Down/Mid/Up). Each cell shows finetuned projections for self-attention (attn1) and cross-attention (attn2) as attn1/attn2 (e.g., v/qkv = finetune v in attn1 and q, k, v in attn2). “–” indicates module not selected. The public model is ‘Stable-Diffusion-v1-5’. ‘Di-A j’ = Down block i, attention layer j; ‘Ui-A j’ = Up block i, attention layer j. ‘M-A0’ = Middle block, attention layer 0. Down Blocks
Ratio 5% 10% 20% 30% 40% 50% 60% 70%
Mid Block
Up Blocks
D0-A0
D0-A1
D1-A0
D1-A1
D2-A0
D2-A1
M-A0
U1-A0
U1-A1
U1-A2
U2-A0
U2-A1
U2-A2
U3-A0
U3-A1
U3-A2
-/v v/v v/v v/v kv/v kqv/kqv kqv/kqv kqv/kqv
-/v/v/v v/v kv/v kqv/v kqv/qv kqv/kqv
-/v -/v v/v v/v v/v v/v kqv/v kqv/qv
-/-/v -/v v/v v/v v/v kv/kv kqv/kv
-/v -/v -/v v/v v/v v/v v/kv v/kv
-/v -/v -/v v/v v/v v/kv v/kv v/kv
-/-/-/v -/v -/v -/v -/v v/kv
-/v -/v -/v -/v -/v -/v v/v v/v
-/v -/v -/v -/v -/v -/v v/v v/v
-/-/v -/v -/v -/v v/v v/v kv/v
-/-/-/v -/v v/v v/v v/v v/kqv
-/-/-/v -/v v/v v/v v/v v/v
-/-/-/v v/v v/v v/v v/v v/kv
-/-/-/v/v v/v v/v v/kqv v/kqv
-/-/v/v/v v/v v/v v/v v/kqv
-/-/v/v v/v v/qv v/qv v/qv v/qv
Table 8: FID and Acc (%) of DP-SAPF on four sensitive datasets with ε = {10, ∞} using two public models ‘Stable Diffusionv1-5’ and ‘Realistic-v6’. ‘ε = ∞’ denotes public model fine-tuning without DP protection.
Method
CIFAR-10 FID Acc
Stable Diffusion-v1-5 OCTMNIST CelebA FID Acc FID Acc
Camelyon FID Acc
CIFAR-10 FID Acc
Realistic-v6 OCTMNIST CelebA FID Acc FID Acc
Camelyon FID Acc
DP-SAPF (ε = 10) DP-SAPF (ε = ∞)
26.6 21.4
77.9 71.8
145.9 118.0
30.8 29.8
81.7 73.3
118.6 139.3
74.6 76.9
46.2 50.3
23.6 19.2
90.2 93.4
79.9 81.2
68.0 70.7
38.5 42.2
19.2 16.0
92.1 95.0
71.7 72.4
Table 9: FID and Accuracy (%) of synthetic images under different noise scale σs = {5.0, 10.0, 20.0, 25.0}.
mechanism so that it matches the allocation used in the (ε = 5, σs = 10) configuration reported in Table 10. • Noise scale σs . As introduced in Section 3.4, DP-SAPF doesn’t tune privacy-budget allocation hyperparameters. The user-specified parameter is the noise scale σs for parameter selection. The privacy allocation is automatically determined via RDP based on the dataset size, the target ε, and σs . We evaluate σs = {5.0, 10.0, 20.0, 25.0} under privacy budget ε = {1.0, 10.0} for CelebA. Table 10 presents the DP-cost ratios (%) between the parameter-selection queries and the DP-SGD fine-tuning stage in DP-SAPF.
CelebA
σs = 5 FID Acc
σs = 10 FID Acc
σs = 20 FID Acc
σs = 25 FID Acc
ε=1 ε = 10
25.0 23.6
25.9 23.8
25.8 23.7
25.8 23.5
84.2 90.2
83.5 91.5
84.8 90.1
83.8 91.1
An ε ≥ 10 provides a practical balance between synthetic performance and privacy preservation. In Figure 5, we observe that increasing the privacy budget ε consistently improves synthetic image quality, as a larger ε corresponds to injecting less DP noise [1, 21]. On both CIFAR-10 and CelebA, performance plateaus when ε ≥ 10, suggesting that ε = 10 provides a practical balance between synthetic performance and privacy preservation. For CIFAR-10, increasing ε from 10 to 20 only reduces the FID from 26.6 to 25.5 (a marginal change of 1.1), while the Acc remains nearly unchanged.
All experiments in this research question are conducted using the public model Stable-Diffusion-v1-5. Figure 5 reports the Acc (%) and FID scores of synthetic images generated by DP-SAPF under different privacy budgets and selection ratios. Table 7 lists the specific parameter matrices selected from the candidate pool for each selection ratio. We summarize the key findings from this RQ as follows. The synthetic performance is relatively insensitive to the selection ratio c within the range of [20%, 50%]. Figure 5 shows that when the c lies within the range of [20%, 50%], both FID and Acc remain nearly unchanged, indicating that our method is largely insensitive to this hyperparameter. Consequently, tuning the selection ratio in practice is straightforward. As the increase in c, DP-SAPF may select some sensitive parameter matrices, and finetune them, which easily collapses the public model. For example, in CelebA, Figure 5 shows a sharp drop in Acc and an increase in FID when the select ratio increases from 60% to 70%. Section D examine the sensitivity of the c on more datasets and public models.
Saliency-aware selection reveals structured sparsity in DP LoRA. As present in Table 7, cross-attention (attn2) and value projection (Wv ) dominate early selection: at c = 5%, only the cross-attention is activated, while self-attention is frozen. Wk and Wq remain frozen until c ≥ 40%, with Wv being the sole projection consistently selected. This highlights the critical role of cross-attention and value projection in DPLoRA, as their gradients carry the strongest learning signals, especially under privacy constraints. Besides, the projection importance is layer-dependent. For example, shallow blocks (D0) require holistic tuning (kqv/kqv at 50% and kv/v at 11
Table 10: RDP cost ratios (%) of parameter-selection query / DP-SGD in DP-SAPF under different noise scale σs for CelebA, under the ε = {1.0, 5.0, 10.0}. Privacy Budget ε=1 ε=5 ε = 10
σs = 5
σs = 10
σs = 20
Table 11: FID and Accuracy (%) of synthetic images generated by public models, with and without fine-tuning on sensitive images under the privacy budget ε = 10.0.
σs = 25
Public Model
12.92 / 87.08 2.71 / 97.29 0.65 / 99.35 0.42 / 99.58 0.57 / 99.43 0.14 / 99.86 0.03 / 99.97 0.02 / 99.98 0.17 / 99.83 0.04 / 99.96 0.02 / 99.98 0.01 / 99.99
SD-v1-5 SD-2-1-base Realistic-v6 Prompt2med
40%), while deeper blocks (Mid and Up) benefit from sparse Wv -only updates. Wq and Wk are consistently excluded from the middle block when c ≤ 60%. We also observe that at lower selection ratios, DP-SAPF tends to prioritize parameter matrices proximal to the input and output layers of the model. Notably, DP-SAPF gets unstable when c = 70% (as shown in Figure 5). This shows that the instability of DP LoRA (as presented in Table 2) is not caused by the whole middle block. Its value projection can also be finetuned stably. The synthetic performance is insensitive to the noise scale σs . As shown in Table 9, increasing σs from 5 to 25 has little impact on CelebA. Under ε = 1, the FID remains within 25.0 to 25.9, and the Acc stays within 83.5% to 84.8%. Under ε = 10, the FID remains within 23.5 to 23.8, and the Acc stays within 90.1% to 91.5%. The selection stage is a oneshot Gaussian query, while the final synthetic performance is mainly determined by the subsequent DP-SGD fine-tuning. As σs increases from 5 to 25, the selection-stage cost drops from 12.92% to 0.42% under ε = 1, and from 0.17% to 0.01% under ε = 10. Hence, a larger σs makes selection noisier but leaves more privacy budget for DP-SGD. Since the top-c high-saliency matrices have relatively strong gradient signals, this extra selection noise mainly affects matrices near the selection boundary and does not substantially change the selected subspace. σs = 5 is a practical default. Thus, Table 13 show that σs = 5 is used across all cases.
6
SD-v1-5 SD-2-1-base Realistic-v6 Prompt2med
Camelyon FID Acc 406.5 416.8 393.5 420.1
61.1 67.2 62.4 62.5
145.9 55.3 118.6 89.4
79.9 78.2 71.7 80.1
74.6) × 100%), 4.1% (= (50.3 − 46.2) × 100%), 3.2% (= (93.4 − 90.2) × 100%), and 1.3% (= (81.2 − 79.9) × 100%) across four sensitive datasets compared to the non-DP setting (ε = ∞). However, we observe that for the OCTMNIST and Camelyon datasets, non-DP image synthesis still performs poorly. This is primarily due to the severe distribution mismatch between the public models and these sensitive datasets. As shown in Table 11. Without fine-tuning on the sensitive images, the synthetic images generated directly from public models exhibit poor FID and Acc. These results indicate the need for further improvements to DP-SAPF.
6.2
Synthesizing Images without Fine-Tuning
Table 11 compares the synthesis quality of four public diffusion models before and after DP fine-tuning via DP-SAPF on sensitive data under ε = 10.0 across four datasets. For every model–dataset pair, fine-tuning yields lower FID and higher accuracy compared to the pretrained models. DP-SAPF consistently improves synthesis quality across four public models, validating its robustness to diverse pretraining priors. The domain bias of public models influences the absolute performance after fine-tuning. ‘Realistic-v6,’ pretrained on high-fidelity human faces, maintains superior CelebA synthesis (19.2 with FID) after adaptation, inheriting its strong facial prior. In contrast, despite being a medical-domain model with the best FID before fine-tuning, ‘Prompt2med’ shows relatively modest gains on OCTMNIST (99.1 with FID) compared to other public models, suggesting its pretraining distribution does not fully align with retinal OCT images. These results also suggest that a public model with strong downstream performance (FID or Acc) does not necessarily have greater benefits after fine-tuning. Selecting the most suitable public model remains an open question for DP-SAPF.
Discussions
This section discusses (1) the impact of DP on the DP-SAPF, (2) the transferability of DP-SAPF, (3) the effectiveness of fine-tuning on sensitive datasets, (4) a comparative analysis of computational overhead relative to baselines, and (5) the limitations of our methods.
6.1
CIFAR-10 OCTMNIST CelebA FID Acc FID Acc FID Acc No Fine-tuning on Sensitive Images 38.2 62.6 239.3 25.0 110.0 89.6 43.1 58.1 263.1 25.0 132.4 88.6 48.3 38.5 313.5 25.0 88.8 71.0 47.8 51.2 128.1 25.0 139.6 87.7 Fine-tuning on Sensitive Images (DP-SAPF) 26.6 74.6 77.9 46.2 23.6 90.2 27.2 72.3 80.0 42.9 24.0 90.5 30.8 68.0 81.7 38.5 19.2 92.1 25.4 72.8 99.1 43.4 26.0 88.5
DP-SAPF in the Non-Private Setting
This experiment evaluates the impact of DP on the generative performance of DP-SAPF. We compare our approach against a non-DP baseline (ε = ∞), in which models are trained using the DP-SAPF without the injection of Gaussian noise. Table 8 shows that, under a privacy budget of ε = 10, when using ‘Stable Diffusion-v1-5’ as the public model, DP-SAPF achieves average Acc reductions of only 2.3% (= (76.9 −
6.3
Computational Resources
This section investigates the computational resource usage of various methods. Table 12 presents the GPU memory and 12
Algorithm PE
Aug-PE
DP-LDM
DP-LoRA
DP-Finetune
DP-SAPF
Stage
Memory
Runtime
Selecting Finetune Synthesis Selecting Finetune Synthesis Selecting Finetune Synthesis Selecting Finetune Synthesis Selecting Finetune Synthesis Selecting Finetune Synthesis
0GB 0GB 31.2GB 0GB 0GB 31.2GB 0GB 33.3GB 17.1GB 0GB 29.0GB 17.1GB 0GB 41.0GB 17.1GB 4.7GB 25.9GB 17.1GB
0h 0h 43.5h 0h 0h 43.5h 0h 4.9h 1.9h 0h 4.7h 1.9h 0h 8.0h 1.9h 0.6h 3.5h 1.9h
Acc (%)
90
305.6
300
78.4
80
400
70
40
61.1
Acc (%)
90.2
90.0
34.6
FID
100
30
0
82
23.6
23.4
84
Acc (%)
40 35
86.9
86
50
31.3GB
90
200 61.2
EM PTR DP-SAPF
92
88
60
Peak Memory
FID
25 20
Figure 6: The synthetic performance of DP-SAPF when using DiT, and alternative DP mechanisms, EM, and PTR.
31.2GB
33.3GB
DP-SAPF already requires per-sample gradients for subsequent DP fine-tuning. Leveraging the same clipped gradients and Gaussian noise enables implementation with clear sensitivity and RDP accounting, without introducing an additional DP mechanism or hyperparameters.
29.0GB
41.0GB
25.9GB
6.5
Limitations
As analyzed in Section 6.1 and Section 6.2, the domain bias between public models and sensitive images influences the absolute performance after fine-tuning. When the public model has a substantial domain mismatch with the sensitive data, fine-tuning alone is insufficient to produce high-quality synthetic images. Selecting the most suitable public model and addressing domain-mismatch challenges remain open questions for DP-SAPF. Besides, the parameter matrix candidate pool in DP-SAPF focuses on the attention layers. Indeed, the design of the matrix candidate pool plays a key role in the effectiveness of DP-SAPF, and exploring how to construct it optimally remains a promising direction for future work. Although gradient magnitude is not a perfect saliency metric for parameter importance [44], many widely used non-DP methods adopt it and achieve strong empirical performance despite these theoretical limitations [45,46]. Alternative selection metrics are possible. We do not adopt them in DP-SAPF as they are typically iterative and more complex under DP. As shown in Section D.1, DP-SAPF achieves comparable or better performance while being simpler and more efficient. The primary contribution of DP-SAPF lies in identifying training collapse when using public models under DP and proposing an effective and practical mitigation strategy. We leave the exploration of more suitable proxies to future work.
runtime usage of baselines and DP-SAPF for the CIFAR-10 using ‘Stable-Diffusion-v1-5’ as the public model. In this table, we observe that fine-tune-free methods, such as PE [5] and Aug-PE [12], require substantial image synthesis and variation using a public model. When the public model-generated image is high resolution (like 256 × 256 resolution for CelebA), the synthesizing process is time-intensive. Thus, the training efficiency is lower than that of fine-tune-based methods. Compared with DP-LoRA, DP-SAPF updates fewer parameter matrices, which in turn reduces both training time and GPU memory consumption. Specifically, DP-SAPF saves about 11.0% (= (29.0 − 25.9)/29.0 × 100%) GPU memory requirements, and 9.1%(= (1.9 + 4.7 − (1.9 + 3.5 + 0.6))/(1.9 + 4.7) × 100%) running time, compared to DP-LoRA [2].
6.4
DP Mechanism (CelebA) DP-LoRA DP-SAPF
100
FID
DiT (CelebA)
Table 12: GPU memory usage and runtime analysis for the CIFAR-10 using ‘Stable-Diffusion-v1-5’ as the public model. ‘Memory’ means the GPU memory usage, and ‘Peak Memory’ means the peak GPU memory usage across all stages.
Transferability of DP-SAPF
This section evaluates whether DP-SAPF can transfer to other NN structures and DP mechanisms. We evaluate DP-SAPF on Diffusion-Transformer (DiT) [41] and alternative DP mechanisms, Exponential Mechanism (EM) [42] and Propose Test Release (PTR) [43]. Figure 6 shows that DP-SAPF transfers well to both alternative model architectures and alternative DP mechanisms. For DiT on CelebA, DP-SAPF improves accuracy from 61.2% to 78.4% and reduces FID from 305.6 to 61.1 compared with DP-LoRA, indicating that saliency-aware selection is not limited to U-Net-based diffusion models. For alternative DP mechanisms on CelebA, EM and DP-SAPF achieve similar accuracy of around 90% and an FID of around 24.0, while outperforming the synthetic performance of PTR. Our choice is motivated by practical alignment with DP-SGD.
7 7.1
Related Work DP Image Synthesis
DP image synthesis has seen various advancements to protect sensitive visual data while generating realistic synthetic images [7, 8, 16, 29, 40, 47–49]. Two main types of methods include (1) using public resources like public datasets 13
or models (like APIs) to improve the synthetic performance [2, 3, 7, 8, 12]; (2) while the public resource is unsuitable, another major paradigm involves training synthesizers solely on sensitive image datasets, without public resources [24, 29, 40, 47, 48, 50, 51].
based methods use gradient-related signals to estimate parameter importance before or during training [19, 55, 56]. However, parameter selection in the non-DP differs from the problem studied in DP. Non-DP methods can freely inspect the training data and choose parameters without accounting for privacy loss. In DP image synthesis, the sensitive images used for parameter selection must also be protected [42]. Low-signal parameters, i.e., those with small gradients, can be easily overwhelmed by injected DP noise. Consequently, training collapse during DP fine-tuning of large public models is a DP-specific, noise-induced failure mode that is much less pronounced in non-DP training [10, 17–19, 55, 56]. DP-SAPF is motivated by these differences and focuses on alleviating the DP-specific training collapse. DP-SAPF performs data-dependent matrix-level parameter selection using clipped and noised gradients, and then composes this selection privacy cost with the DP fine-tuning. In this way, DP-SAPF keeps the efficiency benefit of parameter selection while making the selection stage satisfy DP. Thus, DP-SAPF is not a straightforward adaptation of prior non-DP methods.
Leveraging public dataset. A predominant framework involves using a publicly available dataset for pretraining, followed by fine-tuning the model with sensitive images using DP-SGD [2,3,7,8,51]. This approach leverages public data to provide a strong prior for the synthesizer. Notable methods include PDP-Diffusion [7], which pretrains and finetunes using diffusion models, DP-LDM [3], using latent diffusion models, and DP-LoRA [2], integrating LoRA [16] for parameterefficient fine-tuning. Leveraging public model. Various methods explore using public models to reduce computational intensity, bypassing the need for the extensive computational cost of fine-tuning. The public models are either cloud-based services (e.g., DallE [52]) or local software libraries (e.g., Stable Diffusion [10]). PE [5] is a typical method that iteratively directs APIs to generate synthetic images aligning with sensitive data in the feature space. However, PE’s performance can underperform fine-tuning-based methods when the synthetic data distribution diverges from private data. To mitigate this, SIM-PE [4] adjusts released public simulators (e.g., Google Font [53]) to fit private data under DP, offering an alternative to APIs. Another type of DP image synthesis method using public models is fine-tuning based method [2, 3]. DP-LDM [3] proposes fine-tuning the public models on the sensitive image datasets for high-quality image synthesis. DP-LoRA [2] then introduces the use of LoRA to finetune public models, improving training efficiency.
8
Conclusions
Currently, various works propose leveraging the generative capabilities of public models to improve DP synthetic images. Existing methods commonly use LoRA to reduce the number of trainable parameters. However, we argue that applying LoRA exhaustively across all attention layers of the public model, as done in the current state-of-the-art approach, is suboptimal in a DP setting, as it introduces substantial noise accumulation and harms training stability. This paper introduces DP-SAPF, a saliency-aware method that selects a small set of critical parameter matrices from the attention layer for LoRA fine-tuning under DP. To identify these parameters, we first feed sensitive images into the public model, compute the corresponding gradients, and inject noise into them to satisfy DP. DP-SAPF then selects the most salient parameters, those with the highest noisy gradient magnitudes, for DP fine-tuning. Extensive experiments on four sensitive image datasets show that DP-SAPF improves downstream classification accuracy of synthetic data, reduces FID, and saves computational cost, compared with finetuning approaches that update all attention parameter matrices. Even worse, fine-tuning all attention parameter matrices is highly prone to collapse during private training. DP-SAPF represents the first systematic approach to parameter selection for DP image synthesis leveraging public models, providing a foundation for future research in DP image synthesis.
Selecting Public Datasets for DP Dataset Synthesis. Previous works propose selecting public datasets from available resources to enhance synthetic data generation [3, 8, 54]. Li et al. [8] and Yu et al. [54] propose the selection of a subset of public datasets with distributions aligned to sensitive data for pretraining. SIM-PE [4] proposes a training-free method, which selects a part of images from a large public image dataset to match the distribution of sensitive images. Our paper focuses on leveraging public models and proposes DP-SAPF to address the problem of selecting suitable parameter matrices for private fine-tuning.
7.2 Parameter Selection for Non-DP Synthesis Parameter selection has been studied in non-DP image synthesis. A common line of work reduces the number of trainable parameters by updating only selected modules, such as attention layers or low-rank adapters, instead of fine-tuning the full model [10, 17, 18]. These methods are mainly designed to improve training efficiency, reduce memory cost, and preserve the generative prior of pretrained models. Other pruning-
Ethical Considerations This section examines how each stakeholder is affected during two stages: the research process (data handling) and the 14
dissemination of results (deployment). We then outline mitigation strategies and justify the need for conducting this work.
References
Stakeholders and Process Impact. DP-SAPF involves three stakeholder groups. (1) Data Subjects: individuals represented in sensitive datasets (e.g., faces, medical scans) who rely on strong confidentiality protections during data processing. (2) Data Owners: institutions such as hospitals that manage sensitive images and must navigate regulatory and legal constraints during research collaborations. (3) Researchers and Practitioners: the ML community who depend on reproducible, methodologically reliable DP techniques.
[1] C. Gong, K. Li, Z. Lin, and T. Wang, “Dpimagebench: A unified benchmark for differentially private image synthesis,” in Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, CCS 2025, pp. 4139–4153, ACM, 2025. [2] Y.-L. Tsai, Y. Li, C.-M. Yu, et al., “Differentially private fine-tuning of diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4561–4571, 2025.
Impact of the Research. Releasing DP-SAPF produces both benefits and risks for these groups.
[3] M. F. Liu, S. Lyu, M. Vinaroz, and M. Park, “Differentially private latent diffusion models,” Transactions on Machine Learning Research, 2024.
Positive Impacts. (1) Supporting data sharing (Data Owners & Researchers): DP-SAPF improves the utility of DP-generated images, enabling institutions to share information while preserving privacy, thereby expanding researchers’ access to data. (2) Advancing transparency (Practitioners): open-sourcing our implementation facilitates auditing and replication.
[4] Z. Lin, T. Baltrusaitis, and S. Yekhanin, “Differentially private synthetic data via apis 3: Using simulators instead of foundation model,” arXiv preprint arXiv:2502.05505, 2025.
Negative Impacts. (1) Bias amplification (Data Subjects): DPgenerated data may still reflect or intensify biases present in the original datasets, potentially harming downstream individuals. (2) Misuse risks (Data Subjects & Society): high-fidelity generation methods can be exploited for identity forgery, misinformation, or surveillance applications.
[5] Z. Lin, S. Gopi, J. Kulkarni, et al., “Differentially private synthetic data via foundation model APIs 1: Images,” in The Twelfth International Conference on Learning Representations, 2024. [6] C. Gong, Z. Liu, K. Li, and T. Wang, “Privorl: Differentially private synthetic dataset for offline reinforcement learning,” arXiv preprint arXiv:2512.07342, 2025.
Mitigation. We implement several safeguards and recommend additional measures for real-world use. Methodological Mitigations. (1) Validation checks (Practitioners): we apply strict verification procedures to avoid erroneous or misleading synthetic outputs. (2) Transparency (Data Owners & Researchers): our released codebase enables institutions to audit privacy claims before deployment.
[7] S. Ghalebikesabi, L. Berrada, S. Gowal, et al., “Differentially private diffusion models generate useful synthetic images,” CoRR, vol. abs/2302.13861, 2023.
Recommended Deployment Practices. (1) Fairness auditing (Data Subjects): future deployments should integrate fairness evaluations [57]. (2) Misuse prevention (Society): controlled access and usage monitoring [58], such as restricting model checkpoints, rate-limiting, or requiring authenticated usage, can reduce the risk of malicious exploitation.
[8] K. Li, C. Gong, Z. Li, et al., “PrivImage: Differentially private synthetic image generation using diffusion models with Semantic-Aware pretraining,” in 33rd USENIX Security Symposium (USENIX Security 24), 2024. [9] Z. Liu, P. Luo, X. Wang, and et al., “Deep learning face attributes in the wild,” in 2015 IEEE International Conference on Computer Vision, pp. 3730–3738, 2015.
Justification. We argue that the benefits of DP-SAPF outweigh the risks given the safeguards in place. DP-SAPF addresses a central challenge in privacy-preserving generative modeling: balancing utility and privacy without leveraging public models. By releasing our method and implementation, we promote transparency, facilitate community oversight, and help identify risks in DP image synthesis.
[10] R. Rombach, A. Blattmann, D. Lorenz, et al., “Highresolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022. [11] Z. Liu, C. Gong, T. Y. Zhuo, et al., “Privcode: When code generation meets differential privacy,” arXiv preprint arXiv:2512.05459, 2025.
Open Science We release the replication package on the Github link.6 Besides, the DOI for the artifacts is on Zenodo.7
[12] C. Xie, Z. Lin, A. Backurs, et al., “Differentially private synthetic data via foundation model apis 2: Text,” in International Conference on Machine Learning, 2024.
6 https://github.com/2019ChenGong/DP-SAPF 7 https://doi.org/10.5281/zenodo.20287797
15
[13] H. Wang, Z. Lin, D. Yu, and H. Zhang, “Synthesize privacy-preserving high-resolution images via private textual intermediaries,” arXiv preprint arXiv:2506.07555, 2025.
[25] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in Advances in Neural Information Processing Systems, 2020. [26] A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017.
[14] S. Wang, V. Raunak, A. Backurs, et al., “Struct-bench: A benchmark for differentially private structured text generation,” arXiv preprint arXiv:2509.10696, 2025.
[27] J. Yuan, J. Zhang, S. Sun, P. Torr, and B. Zhao, “Realfake: Effective training data synthesis through distribution matching,” in The Twelfth International Conference on Learning Representations, ICLR 2024.
[15] T. Zou, Z. Lin, S. Gopi, Y. Liu, Y.-Q. Zhang, R. Sim, X. Deng, and S. Yekhanin, “Pe-sgd: Differentially private deep learning via evolution of gradient subspace for text,” in The Fourteenth International Conference on Learning Representations, 2026.
[28] T. Matsumoto, T. Miura, and N. Yanai, “Membership inference attacks against diffusion models,” in IEEE Security and Privacy Workshops (SPW), pp. 77–83, 2023.
[16] E. J. Hu, Y. Shen, P. Wallis, et al., “Lora: Low-rank adaptation of large language models,” in The Tenth International Conference on Learning Representations, ICLR, 2022.
[29] D. Jiang, S. Sun, and Y. Yu, “Functional renyi differential privacy for generative modeling,” in Advances in Neural Information Processing Systems, 2023.
[17] X. Li, F. Tramèr, P. Liang, and T. Hashimoto, “Large language models can be strong differentially private learners,” in International Conference on Learning Representations, ICLR 2022.
[30] Z. Lin, A. Jain, C. Wang, G. Fanti, and V. Sekar, “Using gans for sharing networked time series data: Challenges, initial promise, and open questions,” in Proceedings of the ACM internet measurement conference, 2020.
[18] N. Kumari, B. Zhang, R. Zhang, et al., “Multi-concept customization of text-to-image diffusion,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[31] Y. Yin, Z. Lin, M. Jin, G. Fanti, and V. Sekar, “Practical gan-based synthetic ip header trace generation using netshare,” in Proceedings of the ACM SIGCOMM 2022 Conference, pp. 458–472, 2022.
[19] C. Wang, G. Zhang, and R. B. Grosse, “Picking winning tickets before training by preserving gradient flow,” in 8th International Conference on Learning Representations, ICLR 2020, 2020.
[32] F. Tramèr, G. Kamath, and N. Carlini, “Position: Considerations for differentially private learning with largescale public pretraining,” in Forty-first International Conference on Machine Learning, 2024.
[20] X. Yao, H. Qian, X. Hu, et al., “Theoretical insights into fine-tuning attention mechanism: Generalization and optimization,” in Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2025.
[33] A. Koskela and T. D. Kulkarni, “Practical differentially private hyperparameter tuning with subsampling,” Advances in Neural Information Processing Systems, vol. 36, pp. 28201–28225, 2023.
[21] C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of Cryptography: Third Theory of Cryptography Conference,, pp. 265–284, 2006.
[34] X. Li, F. Tramèr, P. Liang, and T. Hashimoto, “Large language models can be strong differentially private learners,” in The Tenth International Conference on Learning Representations, ICLR, 2022.
[22] M. Abadi, A. Chu, I. J. Goodfellow, and et al., “Deep learning with differential privacy,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318, 2016.
[35] Z. Li, Q. Hu, Y. Chen, P. Wang, Y. Zhang, and J. Cheng, “Lorada: Low-rank direct attention adaptation for efficient llm fine-tuning,” in Findings of the Association for Computational Linguistics: EMNLP 2025.
[23] I. Mironov, K. Talwar, and L. Zhang, “Rényi differential privacy of the sampled gaussian mechanism,” CoRR, vol. abs/1908.10530, 2019.
[36] I. Mironov, “Renyi differential privacy,” CoRR, vol. abs/1702.07476, 2017.
[24] K. Li, C. Gong, X. Li, Y. Zhao, X. Hou, and T. Wang, “From easy to hard: Building a shortcut for differentially private image synthesis,” in 2025 IEEE Symposium on Security and Privacy (SP), pp. 3656–3674, 2025.
[37] A. Krizhevsky, “Learning multiple layers of features from tiny images,” tech. rep., University of Toronto, 2009. 16
[38] J. Yang, R. Shi, D. Wei, et al., “Medmnist v2-a largescale lightweight benchmark for 2d and 3d biomedical image classification,” Scientific Data, p. 41, 2023.
[51] F. Harder, M. Jalali, D. J. Sutherland, and et al., “Pretrained perceptual features improve differentially private image generation,” Trans. Mach. Learn. Res., 2023.
[39] P. Bándi, O. Geessink, Q. Manson, and et al., “From detection of individual metastases to classification of lymph node status at the patient level: The CAMELYON17 challenge,” IEEE Trans. Medical Imaging, vol. 38, no. 2, pp. 550–560, 2019.
[52] A. Ramesh, P. Dhariwal, A. Nichol, et al., “Hierarchical text-conditional image generation with clip latents,” arXiv:2204.06125, 2022.
[40] T. Dockhorn, T. Cao, A. Vahdat, et al., “Differentially private diffusion models,” Transactions on Machine Learning Research, 2023.
[54] D. Yu, S. Gopi, J. Kulkarni, et al., “Selective pre-training for private fine-tuning,” Transactions on Machine Learning Research.
[41] W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023.
[55] Z. Zhang, Q. Zhang, Z. Gao, et al., “Gradient-based parameter selection for efficient fine-tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28566–28577, 2024.
[53] Google, “Google fonts,” https://github.com/google/fonts, 2022.
[42] C. Dwork, A. Roth, et al., “The algorithmic foundations of differential privacy,” Foundations and Trends® in Theoretical Computer Science, pp. 211–407, 2014.
[56] H. He, J. Cai, J. Zhang, D. Tao, and B. Zhuang, “Sensitivity-aware visual parameter-efficient fine-tuning,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 11825–11835, 2023.
[43] C. Dwork and J. Lei, “Differential privacy and robust statistics,” in Proceedings of the forty-first annual ACM symposium on Theory of computing, pp. 371–380, 2009.
[57] S. Barocas, M. Hardt, and A. Narayanan, Fairness and machine learning: Limitations and opportunities. MIT press, 2023.
[44] M. Ancona, E. Ceolini, C. Öztireli, and M. Gross, “Towards better understanding of gradient-based attribution methods for deep neural networks,” in 6th International Conference on Learning Representations, ICLR 2018.
[58] M. Tang, A. Dai, L. DiValentin, A. Ding, A. Hass, N. Z. Gong, Y. Chen, et al., “{ModelGuard}:{InformationTheoretic} defense against model extraction attacks,” in 33rd USENIX Security Symposium, 2024.
[45] N. Lee, T. Ajanthan, and P. H. S. Torr, “Snip: single-shot network pruning based on connection sensitivity,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
[59] P. de Jorge, A. Sanyal, H. S. Behl, et al., “Progressive skeletonization: Trimming more fat from a network at initialization,” in 9th International Conference on Learning Representations, ICLR 2021.
[46] C. Wang, G. Zhang, and R. B. Grosse, “Picking winning tickets before training by preserving gradient flow,” in 8th International Conference on Learning Representations, ICLR 2020.
[60] H. Tanaka, D. Kunin, D. L. K. Yamins, and S. Ganguli, “Pruning neural networks without any data by iteratively conserving synaptic flow,” in Advances in Neural Information Processing Systems NeurIPS 2020.
[47] F. Harder, K. Adamczewski, and M. Park, “DP-MERF: differentially private mean embeddings with random features for practical privacy-preserving data generation,” in AISTATS, pp. 1819–1827, 2021.
A
[48] S. P. Liew, T. Takahashi, and M. Ueno, “PEARL: data synthesis via private embeddings and adversarial reconstruction learning,” in The Tenth International Conference on Learning Representations, 2022.
Details of Rényi DP in DP-SAPF
We use the Rényi DP (RDP) paradigm [23] to account for the cumulative privacy costs, for fairness comparison with previous DP image synthesis methods [1, 2, 5, 24]. RDP provides a rigorous accounting schedule for tracking privacy loss across multiple operations.
[49] C. Gong, K. Li, Z. Lin, and T. Wang, “From easy to hard++: Promoting differentially private image synthesis through spatial-frequency curriculum,” arXiv preprint arXiv:2601.06368, 2026.
Definition 2 (Sub-sampled Gaussian Mechanism (SGM [23])). Let f : Dsub ⊆ D → Rd be a function with sensitivity ∆ f = maxD≃D′ ∥ f (D) − f (D′ )∥2 . Parameterized with a sampling rate q ∈ (0, 1] and noise standard deviation σ > 0, the SGM Q is defined as, ∆ Q f ,q,σ (D) = f (S) + N 0, σ2 ∆2f I
[50] Y. Yang, K. Adamczewski, and et al, “Differentially private neural tangent kernels for privacy-preserving data generation,” CoRR, vol. abs/2303.01687, 2023. 17
where S = {x |x ∈ D selected independently with probability q} and f (∅) = 0. The privacy loss of SGM can be tracked through Rényi DP [23] (Definition 3). RDP can quantify the privacy loss of SGM accurately, as introduced in Theorem 3.
Privacy Composition. The sequential execution of multiple DP operations requires a unified approach to determine the global privacy guarantee. An advantage of the RDP framework is its linear compositionality. Formally, if a sequence of k mechanisms {Q1 , Q2 , . . . , Qk } is applied, where each Qi independently satisfies (α, γi )-RDP, the resulting composite mechanism is guaranteed to satisfy (α, γ)-RDP with γ = ∑ki=1 γi . This additive property streamlines privacy accounting across the iterative updates of DP-SGD and the multi-stage pipeline of DP-SAPF. The total privacy cost accumulates as, td γtotal = γs + ∑i=1 γd,i . Once the cumulative RDP cost (α, γtotal ) is determined, it is mapped back to the standard (ε, δ)-DP through Theorem 4 by optimizing over the order α.
Definition 3 (Rényi DP [23]). The Rényi divergence between two probability M and N is, Dα (M∥N) = h distributions i 1 α−1 ln Ex∼N
M(x) α N(x)
, where α > 1 and α ∈ R. A randomized mechanism satisfies (α, γ)-RDP if, for any neighboring datasets D, D′ , and algorithm Q, it holds that, Dα (Q(D)∥Q(D′ )) ≤ γ. In DP-SAPF, prior to fine-tuning, we perform a gradientbased screening to select a subset of parameters from the public model. For each image xi ∈ Ds , we compute the persample gradient gi , clip it by Cs , and inject Gaussian noise as shown in Section 3.2. Based on the sensitivity analysis in Section 3.2, the ℓ2 -sensitivity of the average operator is ∆s = Cs /N ∗ , where N ∗ means the expected size of the sensitive datasets. Consequently, this selection phase satisfies (α, γs )-RDP, where, γs = 2σα2 , as shown in Theorem 1. s Here, σs is the noise multiplier specifically allocated for the parameter selection. The resulting privacy cost γs remains independent of both the dataset size n and the model dimensionality, providing a robust bound for the selection stage. Then, the selected parameters are optimized using DP-SGD. For each training iteration with sampling ratio q, the privacy loss γd,i is characterized by the SGM, defined in Definition 2. Note that Cs in the selection stage and C in the training SGM correspond to the clipping bounds in their respective phases.
B
Proof of Theorem 1. The averaged clipped gradient query [S1 (Ds ), ..., SK (Ds )] has global ℓ2 sensitivity ∆s = Cs /N ∗ . For any Rényi order α > 1, adding Gaussian noise N 0, σ2s ∆2s I to Sk (Ds ) for the weight matrices {W1 , . . . , WK } ensures that the resulting mechanism satisfies (α, γs )-RDP for γs = 2σα2 . s
Proof. Consider two neighboring sensitive datasets under the add/remove adjacency: Ds = {x1 , . . . , xN } and D′s = {x1 , . . . , xN−1 }. For each sample xi , let gi = [vec(∇W1 Li ), . . . , vec(∇WK Li )] denote the concatenated gradient across all K candidate matrices, and let ḡi = ΓCs (gi ) be the jointly clipped gradient. By the definition of the joint clipping operator, ∥ḡi ∥2 ≤ Cs . Let F(Ds ) denote the vector-valued query that stacks the averaged clipped gradients for all matrices:
Theorem 3. Let p0 = N(0,C2 σ2 ) and p1 = N(1,C2 σ2 ) denote the probability density functions of two Gaussian distributions. A single DP-SGD training step satisfies (α, γi )-RDP for any γi such that: γi ≥ Dα ((1 − q)p0 + qp1 ∥ p0 ) .
Missing Proof
F(Ds ) = [S1 (Ds ), . . . , SK (Ds )] =
1 N ∑ ḡi . N ∗ i=1
Similarly, F(D′s ) = N1∗ ∑N−1 i=1 ḡi . Since the two datasets differ in only one sample xN , their results satisfy, F(Ds ) − F(D′s ) = 1 N ∗ ḡN . Taking the ℓ2 norm and using ∥ḡN ∥2 ≤ Cs gives,
(1)
Here the unit mean shift (0 → 1) follows the normalized analysis in [23]. Theorem 3 facilitates the characterization of the incremental privacy cost γi by evaluating the Rényi divergence across a sub-sampled mixture distribution. The mixture distribution arises because in SGM the differing example is included in the sub-batch with probability q. leading to a mixture of the two output distributions. To report these guarantees within the standard (ε, δ)-DP paradigm, we utilize the following analytical conversion.
∥F(Ds ) − F(D′s )∥2 =
1 Cs ∥ḡN ∥2 ≤ ∗ . N∗ N
Thus the global ℓ2 sensitivity is ∆s = NCs∗ . We refer to implementation in DPImageBench [1], we adopt the approximation N ∗ ≈ N and ignore the additional privacy cost associated with estimating the dataset size. By the Rényi DP guarantee of the Gaussian mechanism, for any Rényi order α > 1, F̃(Ds ) satisfies (α, γs )-RDP with [36],
Theorem 4 (From (α, γ)-RDP to (ε, δ)-DP [36]). A mechanism A satisfying (α, γ)-RDP also satisfies (ε, δ)-DP for any 0 < δ < 1, where, ε = γ + ln(1/δ) α−1 .
γs =
By adjusting the noise variance σ2 , the privacy cost ε = γ + ln(1/δ) α−1 can meet the target privacy budget ε. In practice, ε is obtained by optimizing over α, typically evaluated over a predefined grid (e.g., α ∈ [1.01, 2, 4, ..., 256]).
α∆2s α(Cs /N ∗ )2 α = = 2. ∗ 2 2(σsCs /N ) 2(σsCs /N ∗ )2 2σs
All averaged gradients {Sk (Ds )}Kk=1 are jointly protected because their concatenation F(Ds ) = [S1 (Ds ), . . . , SK (Ds )] has global sensitivity ∆s , and we add independent Gaussian noise 18
Table 13: Hyperparameter settings of DP-SAPF. For different public models, we use the same hyperparameter settings. hyperparameter Noise scale σd Noise scale σs Selection grad. norm Cs Fine-tuning epoch Fine-tuning iterations td Select ratio c Fine-tuning learning rate λ Fine-tuning Batch size Fine-tuning grad. norm C Fine-tuning sample rate q LoRA rank r
CIFAR-10
ε = 1.0 OCTMNIST CelebA
Camelyon
CIFAR-10
ε = 10.0 OCTMNIST CelebA
Camelyon
21.2 5.0 1.0 91 1000 30% 5e−4 4096 1.0 9.1e−2 4
15.6 5.0 1.0 74 1000 30% 5e−4 4096 1.0 7.4e−2 4
2.4 5.0 1.0 15 1000 30% 5e−4 4096 1.0 1.5e−2 4
1.7 5.0 1.0 91 1000 30% 5e−4 4096 1.0 9.1e−2 4
1.5 5.0 1.0 74 1000 30% 5e−4 4096 1.0 7.4e−2 4
0.7 5.0 1.0 15 1000 30% 5e−4 4096 1.0 1.5e−2 4
4.2 5.0 1.0 28 1000 30% 5e−4 4096 1.0 2.8e−2 4
0.8 5.0 1.0 28 1000 30% 5e−4 4096 1.0 2.8e−2 4
Table 14: Prompts used for class-conditional generation across four datasets. Dataset
The Number of Category
Prompt
CIFAR-10
10
“An image of an airplane”, “An image of an automobile”, “An image of a bird”, “An image of a cat”, “An image of a deer”, “An image of a dog”, “An image of a frog”, “An image of a horse”, “An image of a ship”, “An image of a truck.”
CelebA
2
“An image of a female face”, “An image of a male face”
Camelyon
2
“A normal lymph node image”, “A lymph node histopathology image”
4
“An optical coherence tomography (OCT) image for retinal disease 1”, “An optical coherence tomography (OCT) image for retinal disease 2”, “An optical coherence tomography (OCT) image for retinal disease 3”, “An optical coherence tomography (OCT) image for retinal disease 4”
OCTMNIST
of variance σ2s ∆2s I to each block. This is equivalent to applying a Gaussian mechanism to the full vector F(Ds ), and therefore the released set {Sk (Ds )}Kk=1 satisfies the same (α, γs )-RDP. Proof of Theorem 2. Given a fixed clipping threshold C, noise multiplier σd , sampling rate q, and number of iterations td , the (ε, δ)-DP guarantee of DP-SGD remains identical regardless of whether the training is performed on the full parameter set Θ or a low-rank subspace ΘLoRA . Proof. For the objective L(θ; x), let g(θ; x) = ∇θ L(θ; x) denote the gradient. The clipping operator ΓC is defined as, ∥g∥2 ΓC (g) = g max 1, , C
in Section 2.1) is,
∑ ΓC (gi ) − ∑′ ΓC (g j )
∆2 = max
Ds ,D′s i∈D s
j∈Ds
. 2
By the Triangle Inequality and the definition of ΓC , ∆2 = max ∥ΓC (g(θ; x))∥2 ≤ C. x∈Ds ∆D′s
Crucially, the bound C is a scalar invariant to the dimensionality of the parameter vector space d, i.e., ∀d ∈ {m × h, r × (m + h)} : ∥ΓC (g)∥2 ≤ C. Consequently, the RDP budα∆2
get γd,i = 2σ2 C2 2 = 2σα2 remains independent of the number of
Let Mfull be the DP-SGD over the full parameter space Θ ∈ Rm×h and Mlora be the mechanism over the LoRA subspace ΘLoRA ∈ Rr×(m+h) . The m and h are the sizes of the weight matrix, and r ≪ min(m, h) is the rank of the decomposition. For identical hyperparameters {C, σd , q,td }, it holds that,
d
d
trainable parameters. For simplicity, we omit privacy amplification here. Thus, the dimensionality reduction inherent in LoRA does not compromise the DP. LoRA and full-parameter DP-SGD satisfy the same (ε, δ)-DP under equivalent noise.
PrivacyCost(Mfull ) = PrivacyCost(Mlora ).
C
Considering a single iteration, the aggregated clipped gradient for the sample xi in a sub-batch Dsub = {xi }Bi=1 is, s ∑i∈B ΓC (gi ). The gi means the gradient for xi . The ℓ2 sensitivity of G̃t for neighbor datasets D, D′ (as introduced
C.1
Implementation Details Hyperparameter Settings of DP-SAPF
This section introduces the hyperparameter settings in DPSAPF. As presented in Table 13, most parameters remain fixed 19
40
Acc FID
40
0
CelebA
Acc (%)
90
5
10 20 30 40 50 60 70
90.2
80
Acc (%)
0 5
92.5
30
18.4
5 40
Acc FID
Acc FID
100 26.0
5
38.5
300 40 Acc FID
30
Acc FID
200
30 25
73.8
5
10 20 30 40 50 60 70
Selection Ratio (%)
0
10 20 30 40 50 60 70
46.5
30
20
0
90.9
60
10 20 30 40 50 60 70
35
Acc FID
10 20 30 40 50 60 70
70
60
40
40
5
80
Acc FID
10 20 30 40 50 60 70
45.4
0
200
80
24.0
24.9
90
70
5
46.5
10 20 30 40 50 60 70
90
60
10 20 30 40 50 60 70
50
20
10 20 30 40 50 60 70
Acc FID
100
29.3
90.6
80
23.6
5
OCTMNIST
5
70
60
50
0
90 Acc FID
70
26.4
Acc FID
40
20
25.9
150
60 Acc FID
40
20
20
76.2
FID
Acc FID
Prompt2med 80
71.8
60
60
60
Realistic-v6
80
73.6
FID
Acc (%)
CIFAR-10
SD-2-1-base 80
74.6
FID
SD-v1-5 80
76.3
5
77.9
10 20 30 40 50 60 70
5
Selection Ratio (%)
94.5
10 20 30 40 50 60 70
5
100
10 20 30 40 50 60 70
Selection Ratio (%)
Selection Ratio (%)
Figure 7: The Acc (%) and FID of synthetic images generated by DP-SAPF for the sensitive image datasets. Table 15: Performance of synthetic images using alternative saliency metrics under privacy budget ε = 10.0. We use ‘Stable-Diffusion-v1-5’ as the public model.
across all datasets and privacy budgets. Only three parameters vary by dataset. The sample rate q is dataset-dependent and derived from q = batch size/dataset size. Following previous works [2,3], we ignore the privacy cost of querying the dataset size. The noise scale of DPSGD σd is computed via the DP composition theorem to satisfy the given privacy budget. Finetuning epoch naturally differs across datasets due to their varying training set sizes, while the number of fine-tuning steps remains strictly unified. It is noticed that for different public models, we use the same hyperparameter settings.
Datasets CIFAR-10 CelebA
We did not tune the shared hyperparameters between DPLDM, DP-LoRA, DP-Finetune, and DP-SAPF to avoid consuming additional privacy budget [33] and instead used the default settings presented in the previous work [1].
C.2
FID
SNIP Acc Time
FID
FORCE Acc Time
FID
27.1 24.0
74.2 89.8
29.2 27.1
73.8 89.0
26.3 27.5
1.2h 1.5h
6.0h 7.5h
SynFlow Acc Time 73.8 89.5
1.8h 2.2h
FID 26.6 23.6
D
Additional Experimental Analysis
D.1
Alternative Saliency Metric
DP-SAPF Acc Time 74.6 90.2
0.6h 0.8h
We conduct experiments using three alternative saliency metrics, SNIP [45], FORCE [59], and SynFlow [60]. As shown in Table 15, these metrics achieve similar FID and accuracy, suggesting that the benefit mainly comes from selecting a high-saliency parameter subspace. However, DP-SAPF is consistently faster. It takes only 0.6h/0.8h on CIFAR-10/CelebA, compared with 1.2h/1.5h for SNIP, 1.8h/2.2h for SynFlow, and 6.0h/7.5h for FORCE. This speedup comes from the scoring procedure: DP-SAPF directly ranks matrices using the clipped and noised gradients already computed for the private selection query, whereas SNIP, FORCE, and SynFlow require additional metric-specific saliency computation before DP fine-tuning. Therefore, alternative saliency metrics are feasible, but our gradient-norm criterion is simpler and more time-efficient while preserving comparable or better synthetic performance. We emphasize that the main contribution of this paper is identifying a training collapse when using public
Text Prompt
Since we fine-tune public text-to-image generative models (e.g., Stable Diffusion [10]), text prompts are conditional inputs during DP finetuning. For each dataset, we convert class labels into natural-language descriptions to form (prompt, image) pairs for fine-tuning, as shown in Table 14. The prompts for CIFAR-10, CelebA, and Camelyon follow the mappings in DPImageBench [1]. For the OCTMNIST dataset, we adopt neutral numeric labels (type 1–4) rather than clinical abbreviations because we find that pretrained text encoders lack reliable semantic grounding for these rare ophthalmic terms. 20
models under DP and proposing an effective, straightforward mitigation method.
D.2
Analysis for Parameter Selection Ratio
This section evaluates the sensitivity of parameter selection ratio c for the datasets CIFAR-10, CelebA, and OCTMNIST, using public models ‘Stable-Diffusion-v1-5’, ‘Stable-Diffusion-21-base’, ‘Realistic-v6’ and ‘Prompt2med’. Figure 7 shows that when c lies within the range of [20%, 50%], both FID and accuracy remain largely stable across most cases, indicating that our method is relatively insensitive to this hyperparameter. Although for OCTMNIST, when using ‘Stable Diffusion 2.1-base’ as the public model, the accuracy increases from 41.9% to 45.4% (a modest gain of 3.5%), both FID and accuracy remain stable in the other cases. Consequently, tuning the selection ratio in practice is simple, and 30% or 40% both serves as a suitable default choice when applying DP-SAPF to new sensitive datasets and public models. This paper does not tune this hyperparameter and instead directly adopt 30% as the default setting.
21