Public Diffusion Models, Private Images: Key-Controlled Inversion for Conditional Reconstruction Lijunxian Zhang, Weihai Li∗ , Bin Liu and Zikai Xu
arXiv:2606.22988v1 [cs.CR] 22 Jun 2026
School of Cyber Science and Technology, University of Science and Technology of China, Hefei, China, 230026 Email: {ljxzhang@mail., whli@, flowice@, ustcxzk@mail.}ustc.edu.cn
Abstract—Diffusion models are often deployed in settings where model parameters are publicly accessible (e.g., open-source libraries or released checkpoints). This white-box scenario creates a serious security risk: any user who obtains an intermediate latent representation can invert the process to recover the original input image. Most prior work on access control for generative models assumes a black-box model (i.e., parameters are kept secret), typically under an honest-but-curious adversary. By contrast, we address the more challenging and realistic white-box setting where all parameters are public. We present a key-controlled inversion framework that turns the inherent error propagation of diffusion models, which exponentially amplifies small perturbations, into a security asset. By injecting key-dependent noise into the inversion formula, we ensure that only a user with the correct key can reconstruct the original image; any other key yields unrecognizable output. Theoretically, by leveraging existing error-propagation theory for diffusion models, we prove that the resulting ciphertext distribution is IND-CPA secure and derive that the adversary’s advantage is exponentially small in a tunable security parameter, hence negligible for any probabilistic polynomial-time (PPT) adversary. Experimentally, we validate these security guarantees across several models and datasets and further demonstrate cross-model robustness, that the injected key noise does not amplify the performance drop caused by model discrepancies.
I. I NTRODUCTION Diffusion models (DMs) are used in commercial image generation services and open-source libraries. A user who obtains an intermediate latent representation can invert the process to recover the original input. This creates a security risk when the input is sensitive (e.g., a face or a medical image). Existing protections, such as image watermarking [1] or adversarial perturbations [2], are reactive. They detect or respond after the fact rather than prevent misuse before it. For example, image watermarking can prove ownership, but does not stop an adversary from inverting a latent to recover the original image. This motivates the need for an active access control mechanism that determines who can invert a latent to recover the original input. The first, studied in the context of privacy-preserving diffusion models [3], uses homomorphic encryption or secure multi-party computation to protect a user’s prompt while generating a plaintext latent image. These schemes assume the diffusion model parameters remain black-box to the user. More importantly, they are designed for text-to-image generation and do not address image-to-image reconstruction. The second ∗ Corresponding Author
direction fine-tunes the VAE decoder [4] so that a secret key is required to obtain a perceptually meaningful image. Both lines of work rely on keeping the model inaccessible. This assumption fails in open-source settings or when model checkpoints are publicly released, a common practice in the research community. In this white-box setting, where all model parameters are public, an adversary can simply download the model and invert any latent they obtain, rendering black-box protections ineffective. How can one enable active access control without hiding the parameters? Our solution rests on two key insights: First, there exists a strictly invertible sampling formula (e.g., the O-BELM sampler [5]) that allows exact reconstruction when the forward and backward paths are aligned. This gives us a correctness guarantee: with the correct key, the inversion process recovers the original input without error. Second, diffusion models exhibit exponential error propagation [6]: a small perturbation injected early in the reverse process accumulates as the chain proceeds. This mirrors the avalanche effect in symmetric cryptography, where a tiny change in the key produces a completely different ciphertext. We turn this traditionally undesirable property into a security asset: by injecting a key-dependent noise into the noise prediction term, any mismatch (i.e., an incorrect key) is exponentially amplified, making the reconstructed output unrecognizable. Consequently, we utilize the strictly reversible sampler based on O-BELM and inject a key-dependent noise into the noise prediction term during inversion. With the correct key, the noise cancels exactly during decryption, yielding lossless reconstruction. With any other key, the mismatch is exponentially amplified along the chain, producing completely unrecognizable output. Thus, by leveraging the error propagation of diffusion models, we prove that the distinguishing advantage of any PPT adversary in the IND-CPA game is exponentially small in a tunable security parameter, hence negligible. Also, experiments on three datasets confirm that correct keys enable recognizable reconstruction, while wrong keys yield severely distorted outputs. An adaptive adversary cannot gain a non-negligible advantage even after thousands of queries. Overhead is less than 5% compared to standard inversion. In summary, the main contributions of this work are listed as follows:
performed with different models. This property is crucial for practical deployment for frequently updated or finetuned models. II. BACKGROUND AND P RELIMINARIES (a) w/o Protection
A. Diffusion Models and DDIM Inversion Diffusion models are a class of generative models inspired by non-equilibrium thermodynamics [9]. They operate by gradually destroying the data structure via a forward diffusion process, then learning to reverse it to restore the original data distribution [10]. Therefore, diffusion models involve two fundamental processes: forward diffusion and reverse denoising. DDPM [11]formulates them as a probabilistic Markov chain, while DDIM [12] reframes them into an ordinary differential equation (ODE). For DDIM, the forward process is defined as: √ √ xt = ᾱt x0 + 1 − ᾱt ϵ, ϵ ∼ N (0, I), Qt where ᾱt = i=1 αi with αi = 1 − βi following a predefined schedule. The standard reverse (sampling) step is: p √ xt−1 = ᾱt−1 x̂0|t + 1 − ᾱt−1 ϵθ (xt , t),
(b) Post-hoc Watermarking [1]
(c) Adversarial Perturbation [7]
where x̂0|t satisfies:
(d) Privacy-Preserving Model [8]
√ 1 x̂0|t = √ xt − 1 − ᾱt ϵθ (xt , t) . ᾱt For image-to-image (I2I) reconstruction, we need the inversion process: given an image x0 , we compute a sequence of latents x1 , x2 , . . . , xT that can later be used to reconstruct x0 . Unlike the forward process, inversion uses the model’s predicted noise ϵθ (xt , t) instead of a random sample. The inversion step from t to t + 1 is: p √ xt+1 = ᾱt+1 x̂0|t + 1 − ᾱt+1 ϵθ (xt , t),
(e) Fine-tuning-based Access Control [4]
where x̂0|t is defined as above. This expression is obtained by assuming the same noise prediction ϵθ (xt , t) would have been used in the forward direction. Although deterministic, the forward-reverse pair is not strictly reversible; i.e., applying the reverse step followed by the inversion does not exactly recover the original x0 [13].
(f) Our Method
Fig. 1: Existing methods for image protection.
Key-controlled inversion for white-box diffusion models. We introduce a strictly invertible sampling formula that embeds a secret key into the noise prediction term, enabling active access control without hiding model parameters. • Provable security via error propagation. Leveraging the exponential error amplification property of diffusion models, we conduct a formal security analysis for the keycontrolled framework and prove IND-CPA security with adversary advantage exponentially small in the effective inversion depth. • Comprehensive experimental validation. We evaluate the correctness and security through validation experiments across multiple datasets and models. The results further demonstrate cross-model robustness, i.e., the system remains effective even when encryption and decryption are
•
B. Strictly Invertible Sampling The irreversibility stems from a simple asymmetry: the inversion step uses ϵθ (xt , t), while the reverse (sampling) step would need ϵθ (xt+1 , t + 1) to exactly undo it. These two predictions are generally not equal, so the forward-reverse pair is not strictly reversible. Hence, inversion capability is distinct from true reversibility. To achieve exact reversible inversion, subsequent works have resorted to fine-tuning [14] or redesigned sampling formulations [15], [16]. In this paper, we adopt O-BELM [5] for diffusion sampling and inversion. It is a bidirectional explicit sampler that guarantees mathematically exact reversibility. Its optimal coefficients, derived from minimizing local truncation error, keep reconstruction error negligible.
2
Intuitively, Etmod measures how accurately the module (i.e. εθ (xt , t)) predicts the true posterior. And both errors are non-negative and ETcum = 0 because pθ (xT ) = q(xT ) (standard Gaussian). Based on these definitions, Li and van der Schaar [6] established how modular errors accumulate along the chain, leading to the following propagation inequality.
Let γi denote the noise-schedule coefficient used in the O√ BELM sampler, defined as γi = αi , where αi is the standard DDIM parameter above. In the following, we consistently use γi to avoid confusion with the αt in DDIM. The O-BELM sampler and its exact inversion are given by: xi−1 = ai xi+1 + bi xi + ci εθ (xi , i),
(1)
xi+1 = a′i xi−1 + b′i xi + c′i εθ (xi , i).
(2)
Theorem 1 (Error propagation inequality [6]). Under mild conditions (the network output ϵθ is approximately standard Gaussian and the entropy of pθ (xt ) decreases with t), the cumulative error Etcum and modular error Etmod satisfy
The coefficients ai , bi , ci and a′i , b′i , c′i are step-dependent functions of γi , derived from minimizing local truncation error: h2i γi−1 h2i+1 γi+1 h2 − h2i γi−1 bi = i+12 (3) hi+1 γi hi (hi + hi+1 ) γi−1 ci = − hi+1 h2 γi+1 a′i = i+1 h2i γi−1 h2 − h2 γi+1 (4) b′i = i 2 i+1 hi γi hi+1 (hi + hi+1 ) c′i = γi+1 hi p where hi = σ̄i − σ̄i−1 , with σ̄i = (1 − γi )/γi being the scaled noise level. These coefficients satisfy the following mutual inverse relations naturally: ai =
ai a′i = 1,
ci + a′i bi = 0,
c′i + a′i ci = 0.
cum Etcum ≥ Et+1 + Etmod ,
∀t ∈ [1, T ].
(8)
Equivalently, define the amplification factor µt by cum Etcum − Etmod = µt Et+1 .
(9)
Then the inequality implies µt ≥ 1 for every t. Empirical Results. Li and van der Schaar [6] empirically estimated the cumulative error using the maximum mean discrepancy (MMD) [17] with T = 1000 sampling steps. Their experiments show that the MMD value remains stable during early steps but grows rapidly as the reverse chain approaches t = 0, exhibiting a clear exponential amplification trend. This observation directly supports the existence of a threshold step x0 after which errors are exponentially magnified, providing an empirical foundation for our Assumption 3. III. T HREAT M ODEL We consider a public diffusion model (white-box setting) where all parameters are known. A secret key k is used to (δ) encrypt an image x0 into a latent xT via our key-controlled inversion. Only a user with the correct key can recover x0 ; any other key yields invalid. Adversary Capabilities: We follow Kerckhoffs’s principle: the adversary A knows the entire diffusion model, the inversion/sampling algorithms, and the encryption scheme, except the secret key k. A is probabilistic polynomial time (PPT) and can adaptively query an encryption oracle O: on input (δ) a plaintext image x0 , O returns xT generated with a fresh random key noise. A can make up to q = poly(λ) queries. Adversary Goals: The adversary A aims to: • Distinguish which of the two chosen plaintexts corresponds to a given ciphertext (IND-CPA). • Recover the secret key k or the original image x0 from a ciphertext. We focus on the first goal, which implies the others. Security Goal: IND-CPA: Let Π = (Gen, Enc, Dec) be our scheme. For a PPT adversary A, define the IND-CPA advantage as:
(5)
Consequently, applying Eq. (1) followed by Eq. (2) recovers the original xi+1 exactly, establishing strict reversibility. C. Error Propagation in Diffusion Models Diffusion models are sequential: each denoising step takes the output of the previous step. Consequently, a small error at an early step can propagate and potentially amplify along the chain. This phenomenon, known as error propagation, has been formalized by Li and van der Schaar [6]. We briefly recall their framework, which we will later use to analyze the security of our key-controlled inversion. Definition 1 (Modular and cumulative errors). For a diffusion model with T steps, let Etmod denote the modular error of the t-th denoising module pθ (xt−1 | xt ) compared with the true denoising process q(xt−1 | xt ): h i Etmod = Ext ∼pθ (xt ) DKL pθ (xt−1 | xt ) q(xt−1 | xt ) . (6) Similarly, the cumulative error Etcum captures the total discrepancy up to step t: Etcum = DKL pθ (xt−1 ) q(xt−1 ) , (7)
AdvIND−CPA (λ) = Pr[A wins] − Π,A
1 , 2
where the game proceeds as follows: 1) A is allowed to make up to q = poly(λ) adaptive queries to an encryption oracle O. On each query with a
where pθ (xt−1 ) is the marginal distribution induced by the sampling chain and q(xt−1 ) is the true marginal from the forward process.
3
Algorithm 1 Encryption (Key-controlled Inversion) Require: Original latent x0 , secret key k, total steps T , seed −1 n, variance schedule {σi2 }Ti=0 Ensure: Protected latent xT,δ 1: for i = 0 to T − 1 do 2: δi ← PRNG(k, i, n) ∼ N (0, σi2 Id ) 3: xi+1,δ ← a′i xi−1,δ + b′i xi,δ + c′i (ϵθ (xi,δ , i) + δi ) {Eq. (11)} 4: end for 5: return xT,δ Algorithm 2 Decryption (Key-controlled Sampling) Require: Protected latent xT,δ , secret key k, total steps T , −1 seed n, variance schedule {σi2 }Ti=0 Ensure: Reconstructed latent x0,δ 1: for i = T − 1 to 0 do 2: δi ← PRNG(k, i, n) ∼ N (0, σi2 Id ) 3: xi−1,δ ← ai xi+1,δ + bi xi,δ + ci (ϵθ (xi,δ , i) + δi ) {Eq. (10)} 4: end for 5: return x0,δ
Fig. 2: Overall framework. A secret key k drives a PRNG to produce noise δt that is injected into the inversion process. Only the correct key enables exact reconstruction. (δ)
plaintext x0 , O returns xT ← Enc(k, x0 ) using freshly generated key noise. (0) (1) 2) A then chooses two equal-length plaintexts x0 , x0 that were not queried to O. 3) The challenger picks b ∈ {0, 1} uniformly, computes (δ) (b) xT ← Enc(k, x0 ), and sends it to A. 4) A outputs a guess b′ and wins if b′ = b.
Key noise generation.: To make δi reproducible yet unpredictable, we generate it using a cryptographically secure pseudorandom generator (CSPRNG). Let k be a secret key, n a pseudorandom but content-aware seed, and σi2 a step-dependent variance. Then
Π is IND-CPA secure if for every PPT A, AdvIND−CPA (λ) Π,A is negligible in the security parameter λ, where λ depends on both the injected noise variance σN and the effective amplification depth T (the number of steps after which errors grow exponentially). We have proved the existence of such a λ, and its practical values are empirically given through our experiments.
δi = PRNG(k, i, n) ∼ N (0, σi2 Id ),
where d is the dimension of xi . The variance σi2 can be a constant or a scheduled function; it trades off security against reconstruction fidelity. Encryption and decryption algorithms.: Algorithm 1 performs encryption (key-controlled inversion) by iterating Eq. (11) from i = 0 to T − 1, while Algorithm 2 performs decryption (key-controlled sampling) by iterating Eq. (10) backward from i = T − 1 to 0. The next sections analyze the correctness (Proposition 1) and the IND-CPA security (Theorem 3) of this construction.
IV. P ROPOSED F RAMEWORK A. Key-controlled Image Reconstruction Our construction starts from the deterministic O-BELM sampler (Eqs. (1) and (2)), which provides mathematically exact reversibility. To turn this invertible sampler into a key-controlled encryption scheme, we exploit the fact that the predicted noise term ϵθ (xi , i) propagates through the inversion chain. By augmenting this term with a key-dependent perturbation δi , we obtain a conditional inversion and its corresponding reverse sampling: xi−1 = ai xi+1 + bi xi + ci ϵθ (xi , i) + δi , xi+1 = a′i xi−1 + b′i xi + c′i ϵθ (xi , i) + δi .
(12)
B. Correctness Analysis The classical correctness criterion requires: ∀k ∈ K, m ∈ M
Pr(Dec(k, Enc(k, m)) = m) = 1
(13)
where k is a key from the key space K and m is a plaintext from the plaintext space M . In our scheme, however, perfect reconstruction is relaxed due to the diffusion model’s generative noise εθ . We therefore define correctness as:
(10) (11)
Definition 2 (ε-Correctness). Our key-controlled inversion scheme is ε-correct if for any secret key k and original latent x0 , the latent x′0 reconstructed via the conditioned sampling (Alg. 2) from the protected latent xδ,T (Alg. 1) satisfies:
The coefficients ai , bi , ci and a′i , b′i , c′i satisfy the mutual inverse relations of Eq. (5). Consequently, applying Eq. (11) (encryption) followed by Eq. (10) (decryption) with the same δi recovers the original x0 exactly.
Pr (∥x′0 − x0 ∥ ≥ ε) ≤ negl(λ),
4
For two Gaussians with identical isotropic covariance, the KL divergence is
where the probability encompasses all randomness in the noise injection and model sampling, and negl(·) is negligible in the security parameter λ.
∥µ1 − µ2 ∥2 . DKL N (µ1 , σ 2 I) ∥ N (µ2 , σ 2 I) = 2σ 2 Thus, the increase in KL divergence caused by δi is
Def. 2 collapses to perfect (deterministic) correctness as ε = 0. With the explicit reversibility of O-BELM, we can establish the following ideal-case guarantee:
∆DKL (xi , δi ) =
Proposition 1 (Ideal Correctness). Under ideal conditions (no numerical error, deterministic model outputs), our scheme achieves 0-Correctness for any noise variance σt and coefficient scheduling algorithm.
(18)
Taking expectation over independent xi ∼ pθ (xi ) and δi ∼ 2 N (0, σδ,i Id ) eliminates the cross term (E[δi ] = 0), yielding
Proof. We substitute Eq. (11) into Eq. (10) under Eq. (5). Then, we can notice that all key-noise terms δ are cancelled. Hence, x′0 = x0 holds, reaching the ideal correctness. As Prop. 1 establishes perfect reconstruction ideal conditions, we can infer that the reconstruction error under the correct key originates only from practical non-idealities such as rounding errors and floating-point calculation errors. This makes ε-correctness an applicable measure in real-world scenarios.
∆i := E[∆DKL ] =
c′2 i 2 · d · σδ,i . 2β̃i
(19)
This quantity is the expected increase in modular error contributed by the key noise δi in a single inversion step. It is exact under the Gaussian assumption and depends only on the O-BELM coefficient c′i , the true posterior variance β̃i , the 2 latent dimension d, and the noise variance σδ,i . Error Propagation Across Steps: Recall from Def. 1 that the modular error of step i is the expected KL divergence between the model’s conditional distribution and the true posterior. The previous part derived this expectation under the key noise δi ; that value is exactly ∆i as defined in Eq. (19). Hence, after injecting key noise, the modular error becomes ∆i . Now we invoke the error propagation inequality of Li and van der Schaar (Thm. 1), which holds for the true nonlinear diffusion model:
V. S ECURITY A NALYSIS A. Dynamics of Error Propagation Single-Step KL Divergence Increase: We analyze how the key noise δi injected at each inversion step propagates through the chain. Before deriving the incremental modular error, we state a standard assumption used in the error propagation literature [6]. Assumption 1 (Distributional Consistency of Noiseless Inversion). Applying the inversion process to a plaintext latent x0 without added key noise yields a final latent xT that approximately follows a standard normal distribution: xT ∼ N (0, I). Moreover, the model’s noise prediction ϵθ (xt , t) is trained to fit independent Gaussian noise. Therefore, its output distribution is also approximately N (0, I) for any input xt that is close to the training distribution.
cum Etcum ≥ Et+1 + Etmod ,
∀t ∈ [1, T ],
where Etcum is the cumulative error at step t and Etmod is the modular error. When key noise is injected, the modular error Eimod is replaced by ∆i . Iterating the inequality from t = T − 1 down to 0 and assuming ETcum = 0 (because the backward chain starts from pure Gaussian noise), we obtain the following lower bound on the final cumulative error. Proposition 2 (Cumulative error lower bound under key noise). Under the conditions of Thm. 1, injecting key noise that increases the modular error at step i to ∆i yields
This assumption is consistent with the training objective of diffusion models and is widely adopted in theoretical analyses (e.g., [6], [14]). Consequently, the conditional distributions q(xi+1 | xi ) (true posterior) and pθ (xi+1 | xi ) (model prediction) can be well approximated by isotropic Gaussians with a known variance β̃i , which is determined by the noise schedule and can be explicitly demonstrated by αi . Now consider the O-BELM inversion (encryption) step (Eq. (11)), where δi is the key-dependent noise, independent of xi . Under Assumption 1, we have: q(xi+1 | xi ) = N µtrue (14) i , β̃i Id , pθ (xi+1 | xi ) = N µi , β̃i Id , (15)
E0cum =
T −1 Y i X i=0
µs ∆i ,
s=0
where µs ≥ 1 are the amplification factors. cum + Etmod and Proof. We start from the inequality Etcum ≥ Et+1 rewrite it using the amplification factor µt defined by cum Etcum − Etmod = µt Et+1 .
Thm. 1 guarantees µt ≥ 1 for all t. Substituting the modular error Etmod with the key-noise induced value ∆t (and noting that the original modular error is zero in the ideal model, or that ∆t already accounts for the increment), we obtain
where d is the dimension of xi , and µi = E[a′i xi−1 + b′i xi + c′i ϵθ (xi , i)].
c′2 c′ i ⊤ ∥δi ∥2 + i (µi − µtrue i ) δi . 2β̃i β̃i
(17)
(16)
cum Etcum = µt Et+1 + ∆t .
Injecting δi shifts the mean to µ′i = µi + c′i δi .
5
Iterating this relation from t = T −1 down to 0 with ETcum = 0 gives T −1 Y i X E0cum = µs ∆i . i=0
amplification factor of the true nonlinear model (Theorem 1). Under the same conditions as Theorem 1 (Li & van der Schaar), the spectral radius satisfies ρ(Mi ) ≥ 1 for every step i.
s=0
Proof. Theorem 1 guarantees µi ≥ 1 for every step. This means that any error injected at step i cannot be reduced by the subsequent dynamics; it is at least preserved when propagated to the next step. Now suppose, for contradiction, that ρ(Mi ) < 1. Then, for sufficiently small initial errors, the linearized system would predict exponential contraction over successive steps. Since the true nonlinear dynamics is a first-order perturbation of the linearized system (the difference consists of higher-order terms in the error), the actual evolution would also contract. This directly contradicts the fact that µi ≥ 1 (which holds for arbitrarily small errors). Hence our assumption ρ(Mi ) < 1 is impossible, and we must have ρ(Mi ) ≥ 1.
Since each µs ≥ 1, the right-hand side is at least the sum of the ∆i . This establishes the bound. This proposition shows that even small per-step perturbations ∆i lead to an exponentially huge final cumulative error when the key is incorrect, as long as the amplification factors satisfy µt > 1 for most steps. Consequently, under such conditions, the ciphertext distributions corresponding to different plaintexts become statistically indistinguishable. B. Linearized Differential Analysis QtProp. 2 shows that the cumulative error grows at least as s=0 µs when key noise is injected. However, to obtain a quantitative bound on the adversary’s IND-CPA advantage, we need a more explicit expression for the covariance of the ciphertext distribution. For this purpose, we consider a differential analysis for the linearized approximation of the error dynamics. Given that each key noise δt is i.i.d. from N (0, σt2 Id ), we can analyze the general attack scenario by considering a null noise estimate (δ = 0). Leveraging the reversibility of OBELM, the error propagation is therefore traced through the encryption process. Considering the unconditional inversion equation Eq. (2) and the key-conditioned version Eq. (11), let ei denote the latent error at step i, i.e., ei = xi,δ −xi . Subtracting these two equations gives the dynamical equation for error propagation:
Prop. 3 shows that the linearized approximation is consistent with the true nonlinear dynamics, ensuring that the tractable error bounds obtained from the linear model are meaningful and provide a valid basis for security analysis. Explicit form of linearized errors. Now we solve the linear recurrence. The total error eT after T iterations can be expressed as a closed-form sum of the noises injected at each step, each amplified by the propagation gain Ψt : eT =
Ψt δ t ,
(24)
t=0
Ψt = P Φ(T, t) Nt (25) QT −1 where Φ(T, t) = i=t Mi is the cumulative state transition from step t to T , P = [0 I] extracts the error component from the full state vector, and Nt = [0 c′t I]T maps the injected noise δt into the state space. Thus, Ψt quantifies how the noise injected at step t contributes to the final error after forward propagation. Substituting Eq. (25) into Eq. (24), we directly obtain the following bound on the overall error magnitude:
ei+1 = a′i ei−1 + b′i ei + c′i [εθ (xi,δ , i) − εθ (xi , i)] + c′i δ (20) We assume εθ (x, t) is continuously differentiable in x. The property is achievable via differentiable activations. This allows us to simplify Eq. (20) through a first-order approximation: ei+1 = a′i ei−1 + (b′i I + c′i Ei )ei + c′i δ (21) where Ei is the Jacobian matrix of εθ , given by: εθ (xi,δ , i) − εθ (xi , i) ≈ Ei (xi,δ − xi ) = Ei ei .
T −1 X
(22)
This forms a second-order linear system governed by Eq. (21). To analyze its signal amplification, we rewrite it in vector form as shown below. ′ ei+1 bi I + c′i Ei a′i I ei ′ δ = + ci (23) ei I 0 ei−1 0
∥eT ∥ =
T −1 X
P Φ(T, t) Nt δt
t=0
≤ κ · max ∥δt ∥ · t
T −1 X
∥Φ(T, t)∥,
(26)
t=0
where κ = ∥P∥ maxt ∥Nt ∥ is a constant determined by coefficient schedule. This leads to a useful characterization of error growth:
Eq. (23) can be decomposed into its homogeneous part and driven part. The homogeneous part is governed by the statetransition matrix Mi . Connection to nonlinear amplification. Noting the relationship between the linearized approximation and the true nonlinear model, we have the following proposition.
Corollary 1 (Error Propagation Bound). In the linearized dynamics, the norm of the total latent error ∥eT ∥ satisfies: ! T −1 X ∥eT ∥ ≤ C · ∥Φ(T, t)∥ (27)
Proposition 3. Let Mi be the state transition matrix of the linearized error dynamics (Eq. (23)) and let µi be the
t=0
6
where C = κ · maxt ∥δt ∥ is a constant factor. The product of the maximum injected noise magnitude and the sum of the propagation-gain norms ∥Ψt ∥. Consequently, the error amplification over the entire path is governed by the sum of ∥Φ(T, t)∥.
where Φ(·) denotes the cumulative distribution function (CDF) of the standard normal distribution, and dCT is the Mahalanobis distance between the two distributions. Consequently, under this setting, the adversary’s advantage is upper-bounded by: = AdvIND−CPA A,Π
C. Indistinguishability of Ciphertext Distributions In the standard IND-CPA game, an adversary A with oracle access to Enc(k, ·) submits two plaintexts (m0 , m1 ) and receives a challenge ciphertext c∗ ← Enc(k, mb ) for a randomly chosen b ∈ {0, 1}. The adversary’s advantage is = Pr[b′ = b] − 21 , where b′ is defined as AdvIND−CPA A,Π A’s guess. The scheme is IND-CPA secure if AdvIND−CPA A,Π is negligible for any probabilistic polynomial-time A. Since the challenge ciphertext c∗ is the protected latent (b) xT,δ , the adversary’s distinguishing task reduces to a statistical hypothesis test (Def. 3). To validate this claim, we analyze the character of the ciphertext distribution. For any plaintext m corresponding to the original latent x0 , we use Eq. (24) and (25) to relate the ciphertext xT,δ and the noiseless inversion result xT,0 as follows. T −1 X
PΦ(T, t)Nt δt
Assumption 2 (Exponential Amplification in the Nonlinear Model). There exists a threshold step x0 < N such that for all steps t > x0 , the Q amplification factor satisfies µt > 1. t Moreover, the product s=x0 +1 µs grows exponentially. This property is consistent with the empirical results of [6] and ensures that errors injected at earlier steps are exponentially magnified. The following assumption is a linearized counterpart, which is justified by Proposition 3.
(28)
Assumption 3 (Exponential Growth in the Linearized Model). For a fixed maximum inversion step N , there exists a threshold x0 < N such that for any x > x0 and t > 0,
t=0
Moreover, as we assume that each δt is i.i.d. sampled from N (0, σt2 Id ), the distribution of the ciphertext can be closely approximated by: xT,δ ∼ N (xT,0 , ΣT )
∥Φ(N, x + t)∥ ∼ ρ t ∥Φ(N, x)∥,
(29)
with ρ > 1, while for 1 ≤ x ≤ x0 , ∥Φ(N, x)∥ ≈ k∥Φ(N, 1)∥ for some constant k.
(30)
This exactly matches the error propagation pattern observed in [6] when the key-induced noise is treated as part of the modular error. Based on these assumptions, we now state the following propositions. Their proof is given in the Appendix.
where ΣT satisfies: ΣT =
T −1 X t=0
σt2 ΨTt Ψt ≤ (max σt )2 t
T −1 X
ΨTt Ψt .
(35)
The diffusion model’s noiseless inversion approximates a highdimensional normal sample, so the adversary’s advantage is essentially bounded by the covariance ΣT . To quantify this bound, we introduce two related assumptions. The first captures the essential nonlinear error propagation property, while the second translates it into a form suitable for linearized analysis.
xT,δ = xT,0 + eT = xT,0 +
1 ∗ . − Perr 2
t=0
Therefore, given the ciphertext distribution, we define the hypothesis testing problem as follows:
Theorem 2. Under Asm. 3, there exists a noise schedule such that distinguishing the ciphertext distributions corresponding to two distinct plaintexts is computationally hard. More precisely, the distinguishing advantage is exponentially small in the number of effective steps N − x0 .
Definition 3 (Hypothesis Test for IND-CPA Distinguishing). (0) (1) Let P0 = N (xT,0 , ΣT ) and P1 = N (xT,0 , ΣT ) denote the two possible ciphertext distributions corresponding to the (0) (1) encryption of messages x0 and x0 , respectively. Given a (b) single observed ciphertext xT,δ , b ∈ {0, 1}, the adversary’s task is to decide between: H0 : xT,δ ∼ P0 ,
(b)
(31)
(b) H1 : xT,δ ∼ P1 .
(32)
Theorem 3. If Thm. 2 holds and the PRNG is cryptographically secure (pseudorandom), then there exists a sampling configuration for which the key-controlled inversion scheme is IND-CPA secure. And the adversary’s advantage is bounded by exp(−Ω(λ)), which is negligible for any PPT adversary.
This constitutes a hypothesis test for two multivariate Gaussian distributions with a common covariance matrix ΣT .
D. Security Boundaries and Parameters
According to Def. 3, we can find that the minimum error probability is: Z x 2 dCT 1 ∗ Perr = Φ(− ) Φ(x) = √ et /2 dt (33) 2 2π −∞ q (0) (1) (0) (1) dCT = (xT,0 − xT,0 )T Σ−1 (34) T (xT,0 − xT,0 )
We now examine the practical security guarantees of our scheme, returning to the threat model defined in Section III. Multi-query indistinguishability. Our IND-CPA analysis previously focused on a single challenge. However, the adversary in the IND-CPA game is allowed to make up to q = poly(λ) adaptive queries before receiving the challenge ciphertext.
7
Because each encryption uses fresh, independently generated pseudo-random noise {δt } (derived from a secret key k and a seed n to prevent replay), the ciphertexts from different queries are independent. This independence enables a standard hybrid argument. Define hybrid experiments Hyb0 , Hyb1 , . . . , Hybq as follows: in Hybi , the first i queries are answered with the true encryption of the adversary’s chosen plaintexts, while the remaining q − i queries are answered with encryptions of a fixed reference plaintext (e.g., the all-zero image). The difference between Hybi−1 and Hybi lies exactly in the i-th query, which reduces to the single-challenge distinguishing scenario. Hence, the advantage of distinguishing Hybi−1 from Hybi is at most the single-challenge advantage ε = exp(−Ω(λ)). By the triangle inequality, the total advantage in distinguishing the real game (Hybq ) from the all-reference game (Hyb0 ) is bounded by q · ε. Since q = poly(λ) and ε is exponentially small, the overall advantage remains negligible. Thus our single-challenge security analysis directly extends to the full multi-query IND-CPA setting. Key & Plaintext Recovery. The key noise is generated by a CSPRNG; extracting the key from a ciphertext is as hard as breaking the CSPRNG. Plaintext recovery is implied by IND-CPA security: a successful plaintext recovery would directly yield a distinguishing advantage. Thus, both attacks are computationally infeasible under the same assumptions. Practical determination of the security parameter. The theoretical analysis establishes that the adversary’s advantage is bounded by an exponentially decaying function of λ = N − x0 , where x0 is the step after which the amplification factors µt become strictly greater than 1. In practice, however, the exact value of x0 depends on the noise schedule, the model architecture, and the statistics of the training data. Moreover, the total number of steps N cannot be arbitrarily large because the diffusion model’s own prediction error accumulates over long chains, potentially deteriorating the reconstruction quality for legitimate users. Therefore, while the existence of a security parameter λ is guaranteed, its concrete numerical value must be determined empirically. We provide a set of experiments in Sec. VI-D that measure the reconstruction error under correct and incorrect keys for various N and noise variances, thereby offering practical guidance for choosing λ to achieve a desired security level.
inversion and sampling process of the O-BELM sampler for encryption and decryption, respectively. The default number of steps is set to 20 for both processes, and the noise schedule follows the squared cosine (squaredcos_cap_v2) scheduler [22]. Key Noise Generation. The key noise δt is sampled from a Gaussian distribution N (0, σδ2 Id ). To meet the fixed input length requirement of the PRNG and to cryptographically bind the noise to the secret key, the step index, and the query, we first compute a hash: seedt = Hash(k ∥ t ∥ n), where k is the secret key, t the step index, n a query-specific seed (to prevent replay attacks), and Hash is a cryptographic hash function (e.g., SHA-256). This seed is then used to instantiate a cryptographically secure PRNG (AES-128 in CTR mode) to generate δt ∼ N (0, σδ2 Id ). Unless stated otherwise, we set σδ = 1 and vary it in Sec. VI-B to study the effect on legitimate reconstruction. Evaluation Metrics. To assess reconstruction quality, we adopt three widely used metrics: •
Peak Signal-to-Noise Ratio (PSNR) measures pixel-wise fidelity. For an original image x0 and its reconstructed version x̂0 , both with values in [0, 1], we define 1 1 , MSE = ∥x0 − x̂0 ∥2 , PSNR = 10 log10 MSE d
where d is the number of pixels. Higher PSNR indicates better reconstruction. • Structural Similarity Index (SSIM) evaluates perceptual similarity. It combines luminance, contrast, and structure terms: SSIM =
(2µx µx̂ + C1 )(2σxx̂ + C2 ) , 2 (µx + µ2x̂ + C1 )(σx2 + σx̂2 + C2 )
where µ, σ, σxx̂ denote local means, standard deviations, and cross-correlation, and C1 , C2 are small constants. SSIM ranges in [0, 1], with 1 meaning identical. • Fréchet Inception Distance (FID) [23] quantifies the distributional distance between real and reconstructed images. FID is computed as FID = ∥µr − µg ∥2 + Tr Σr + Σg − 2(Σr Σg )1/2 ,
VI. E XPERIMENTS A. Experimental Setup
where (µr , Σr ) and (µg , Σg ) are the mean and covariance of features extracted from real and generated images by a pre-trained Inception network. Lower FID indicates higher fidelity and diversity.
Datasets and Models. We conduct experiments on three standard image datasets: ImageNet-1K [18], MS-COCO [19], and CelebA-HQ [20]. We randomly select 200 images with the size of 512 × 512 from each dataset. Moreover, we employ multiple public latent diffusion models: Stable Diffusion [21] v1.4, v1.5, v2.0, and v2.1 to evaluate cross-model robustness. Environment and Parameters. All experiments are performed on an RTX 4090 GPU with PyTorch 2.7. We use the
We also report computational efficiency as the total runtime for a complete encryption-decryption cycle. Additional security-specific metrics (e.g., distinguishing accuracy under adaptive queries) are presented in Sec. VI-D.
8
(a) PSNR - σ
computation: high-intensity noise loses low-order information during floating-point operations, and Sec. V indicates that multi-step inversion amplifies this low-order error to a level that noticeably affects reconstruction quality. We also note that even with σ = 0, perfectly reversible inversion is hardly attainable. Besides the floating-point errors mentioned above, the reconstruction error introduced by the VAE is another significant contributing factor.
(b) SSIM - σ
C. Cross-Model Robustness
(c) FID - σ
To assess the robustness against potential model updates or fine-tuning as a common scenario in practice, we evaluate its performance when encryption and decryption are performed with different diffusion model variants. We symmetrically test all pairwise combinations among four Stable Diffusion checkpoints (v1.4, v1.5, v2.0, and v2.1). For each pair (encmodel, dec-model), we generate ciphertexts and attempt reconstruction across the same ranges of step counts T and noise intensities σ 2 as in Sec . VI-B.
(d) MSE - σ (ε-correctness)
Fig. 3: Variation of quality metrics with the noise variance σ
(a) σ = 0
(b) σ = 0.5
(c) σ = 1
(d) σ = 2
(e) σ = 5
(f) σ = 10
Fig. 4: Visual effects of quality degradation with the increase of σ. Here we hold sampling steps T = 20.
B. Quality of Conditional Reconstruction (a) Near-version models
This experiment evaluates the empirical ε-correctness of the framework by measuring the reconstruction quality under correct decryption across different sampling steps T and noise intensities σ 2 . The latent Mean Squared Error (MSE) is computed to estimate the practical reconstruction error bound ε. For experimental clarity, we set the key noise variance to a time-invariant constant σ 2 across all sampling steps. The baseline (σ 2 = 0) corresponds to standard O-BELM without protection. The quantitative trends across parameters are shown in Fig. 3. Example reconstructions under different noise levels are provided in Fig. 4. Experimental results show that as the number of sampling steps and the noise intensity increase, the quality of the reconstructed images decreases, while the MSE between the latent variables increases, that is, the ε-correctness error coefficient ε rises. This is caused by floating-point errors in
(b) Distant-version models
Fig. 5: Evaluation of cross-model robustness. As in the Sec. VI-B, we adopt three metrics: PSNR (top row), SSIM (middle row), and FID (bottom row). The results are aggregated into Fig. 5, and visualized as a heatmap (Fig. 6) for a representative T = 20 and σ = 1. We observe that when using closely related model versions (e.g., encrypting with Stable Diffusion v2.0 and decrypting with v2.1), the robustness performance is better compared to using versions with larger differences. As a controlled comparison, we also test reconstruction from ciphertexts perturbed by a small amount of additive noise but using the same model for both encryption and decryption, resulting in PSNR ≈ 6.41dB and FID ≈ 606.51. The result is irrelevant to the model version. Thus, the impact
9
Fig. 6: The heatmap of cross-model robustness, where P indicates PSNR and S indicates SSIM. Method PCDiff RNDMask Our Scheme
Condition
PSNR/dB
SSIM
FID
corr. key wr. key w/o key corr. key wr. key w/o key corr. key wr. key w/o key
30.13 25.35 21.49 25.51 13.37 7.51 25.41 5.84 5.84
0.91 0.85 0.82 0.76 0.35 0.08 0.73 0.03 0.03
8.54 34.81 39.37 27.39 299.68 423.61 9.05 576.90 640.98
TABLE I: Conditional Access Control with different existing methods, where for wrong-key (wr. key) and no-key (w/o key) settings the desired trends are opposite to those for the correct key (corr. key): lower PSNR/SSIM and higher FID indicate stronger protection.
(a) ∥Φ(T, t)∥
Fig. 7: Parameter trends across sampling steps, where each row corresponds to a different scheduling method: squared linear (first row), squared cos (second row), linear (third row).
of cross-model robustness is empirically isolated from general ciphertext sensitivity. Finally, we provide a theoretical analysis of cross-model robustness based on the error-propagation dynamics in the decryption/sampling path. Details of this analysis are discussed in Sec. VII-C.
existing methods for image encryption or access control in diffusion models(e.g., PCDiff [4], RNDMask [24]). As verified in Tab. I, due to the performance limitations of the VAE and the presence of floating-point numerical errors, the quality of correct reconstruction is constrained. However, under both wrong key and no key conditions, our method prevents unauthorized reconstruction by an adversary more effectively compared to the other two approaches.
D. Security Evaluation This section provides a comprehensive empirical assessment of the security properties, designed to complement the theoretical analysis. The evaluation consists of three targeted subexperiments: Perceptual Security: We compare reconstruction quality under three conditions: with the correct key (corr. key), with a wrong key (wr. key, under single bit flip), and without any key (w/o key). Results are benchmarked against representative
Steps
Avg. Cost (s)
AdvIND−CPA A
Attack time (h)
10 20 50 100 200
1.01 1.85 4.35 8.58 16.92
3.44 × 10−11 6.98 × 10−11 1.95 × 10−10 3.91 × 10−10 6.17 × 10−10
4.08 × 106 3.68 × 106 3.10 × 106 3.04 × 106 3.80 × 106
(b) λ2 (ΣT )
Validation of Theoretical Bounds: To empirically support Asm. 3, we measure the norm of the error propagation operator ∥Φ(T, t)∥ and the square of the eigenvalue of the ciphertext variance matrix λ2 (ΣT ) across different noise schedules and maximum step counts N , under a controlled setting where the model’s noise prediction ci Ei is disabled. As shown in Fig. 7, the two parameters exhibit a highly consistent growth pattern: they remain relatively stable in the early sampling stages and then undergo a transition to exponential growth later. Based on these observations, we conclude that Asm. 3 is well-founded. If Assumption 3 holds, the ciphertext distributions for two different plaintexts are approximately Gaussian with the same (0,0) (0,1) covariance matrix ΣT but different means xT and xT . In this case, the squared Mahalanobis distance between the two distributions is given in Eq. 34. Consequently, as proved
TABLE II: Encryption-Decryption Time
10
Steps 10 20 50 100 200
Ours 1.01 1.85 4.35 8.58 16.92
Models & Methods O-BELM[5] DDIM[11] [25] 0.85 1.21 2.48 4.71 9.05
0.85 1.21 2.47 4.70 9.03
14.82 31.95 70.92 137.66 262.10
Whitening [26]
with whitening w/o whitening
2030.15 3220.91 8352.14 >10000 >10000
PSNR/dB
SSIM
MSE
27.52 27.82
0.8321 0.8620
2.8 × 10−3 2.5 × 10−3
TABLE IV: Comparison experiment of whitening
TABLE III: Inference Time Comparison (s) in Appendix ??, d2CT satisfies the following upper bound: d2CT ≤
36 dim(x0 ) , λmin
where λmin denotes the smallest eigenvalue of ΣT . Thus, by estimating λmin (ΣT ) from linearized error propagation, we can compute a concrete upper bound on the distinguishability of the two ciphertext distributions. Meanwhile, according to Eq. (30), the variance of the injected noise σt2 directly influences ΣT : increasing σt enlarges λmin (ΣT ) quadratically, thereby reducing the adversary’s advantage at a quadratic rate. This provides an additional tunable parameter for security. Computational Security & Practical Implications: We report the wall-clock time for a full encryption-decryption cycle in Tab. II. Combining this with the theoretical upper bound for the adversary’s IND-CPA advantage above, we estimate the computational effort required for a successful attack through the equation below: T =
1 AdvIND−CPA A,Π
×
tC 2
(a) with whitening
(b) w/o whitening
Fig. 8: Visual Quality of the whitening experiment VII. D ISCUSSION & I MPLEMENTATION D ETAILS A. Avoiding Side-Channel Analysis The precise inversion in O-BELM relies on two consecutive latents, i.e., xi−1 and xi , which could open an opportunity for side-channel analysis aimed at inferring information from their relationship. Inspired by classical block-cipher designs, we introduce a whitening step, thereby effectively hardening the scheme against such side-channel attacks. The purpose of whitening is to transform xi−1 using a simple method so that its distribution matches that of xi . An existing approach is to use the PRNG to generate noise δW satisfying:
(36)
where tC is the duration of one encryption–decryption cycle. An adversary mounting an attack would need to spend at least tC /2 per query to the encryption oracle. Although experiments show that under our given computational constraints, an attack would require at least 3 × 106 h ≈ 342.47years, from a long-term perspective, the system presents a latent risk if an adversary’s computational resources are sufficiently advanced. This allows us to derive practical guidelines for setting access frequency limits and key refresh policies to maintain security in deployed scenarios. Moreover, we compare the total runtime of a complete encryption-decryption cycle against several baselines: nonencrypted direct reconstruction (using both standard DDIM and O-BELM) and representative privacy-preserving diffusion model approaches. The results are summarized in Table III. It shows that our scheme incurs higher time overhead than standard DDIM and O-BELM. The main bottleneck is the generation of the key noise δt ; as shown in [5], the time cost of O-BELM itself is similar to that of DDIM. However, compared to privacy-preserving diffusion models that rely on homomorphic encryption or secure multi-party computation, our approach offers a significant speedup. Those cryptographic primitives introduce high computational and communication overhead, often making them impractical for real-time applications.
δW ∼ N (µxi − µxi−1 , σx2 i − σx2 i−1 )
(37)
such that the addition of δW to xi−1 results in the same mean and variance as xi while ensuring sufficient masking of the noise. We also propose an additional experiment to verify that the implementation of whitening does not significantly degrade the reconstruction or decrypted image, as is displayed in Tab. IV and Fig. 8. In the additional experiment, we set σ = 1.0 and sampling steps T = 20 as a typical setting for application. However, it is necessary to securely transfer the mean and variance of xi−1 for this process. Inspired by existing steganography algorithms, we derive a basic method as described in the Alg. 3, which applies perturbations to the whitened latent x̂i−1 so that the encrypted mean and variance information can be embedded. B. Error Propagation on Sampling Process Similar to the security analysis, we examine the crossmodel robustness from the perspective of Error Propagation Dynamics. This is because robustness across different models can also be modeled as error induced by the discrepancy in model-predicted noise. However, unlike the key-induced
11
Algorithm 3 Whitening with Secret Embedding Input: latents xi−1 , xi , key k, step i, seed n Output: whitened latent x̂i−1 1: Compute mean and variance of xi−1 , xi . 2 2: Generate δW ∼ N (µxi − µxi−1 , σx − σx2 i−1 ) using i PRNG(k, i, n). 3: x̂i−1 ← xi−1 + δW . 4: Generate a 128-bit mask B ← PRNG(k, i, n). 5: Encrypt the 128-bit parameter M as E = M ⊕ B, embedding E into the first 128 entries of x̂i−1 . 6: For each bit B[j], if B[j] = 0 then flip the sign of the j-th entry of x̂i−1 ; otherwise leave it unchanged. 7: return x̂i−1
during the decryption process under Asm. 3. This implies that if the initial noise is identical, the disturbance introduced by the sampling process is often bounded. Notably, this intuition is formally supported by the properties of O-BELM below. Definition 4 (Zero-stability). A system described in Eq. (41) is said to be zero-stable if, when its driving term is zero, i.e., ci = 0, there exists a constant K such that for any two sequences N {xi }N i=0 and {yi }i=0 generated by the same sampling scheme but with different initial values, the following inequality holds: ∥xi − yi ∥ ≤ K max ∥xj − yj ∥ Theorem 4. [5] O-BELM is zero-stable. In Sec. VII-B, we have shown that the error introduced by the model discrepancy can be analyzed theoretically using a similar approach as in Sec. V. Likewise, the total error can also be written as the sum of noise injected at each step with (D) a list of specific coefficients Ψt as Eq. (24), which is shown below: T −1 X (D) (t) (D) Ψt ε̂ED (44) e0 =
error, the model-prediction errors are introduced during the sampling/decryption process. Consequently, the behavior of model-prediction errors may differ from the analysis presented in Sec . V. Our error propagation analysis again starts from Eqs. (11) and (10), but here we modify the model prediction noise term εθ . Let θ(E) and θ(D) denote the parameters of the encryption model and the decryption model, respectively. By substituting Eq. (11) into Eq. (10), we can get the model-prediction error below: (D)
(D)
(D)
ei−1 = ai ei+1 +bi ei
(E)
(D)
(D)
+ci [εθ (xi , i)−εθ (xi
t=0
To simplify the analysis, we assume that the effect of this noise becomes negligible in the subsequent predictions of the diffusion model, i.e., there exists a negligible ε > 0 such that (D) ∥ci Ei ∥ < ε. Thus, we obtain the following proposition:
, i)] (38)
Proposition 4. Under the above assumption, the propagation of the model-prediction error noise remains bounded.
(D)
Consistent with Eq. (20), we denote ei as the error at step (D) (D) (D) i, i.e., ei = xi − xi , where xi is the latent at step i (D) utilizing the decryption model θ . Without losing generality, we add a ”mixed prediction noise” for linear approximation, i.e., Eq. (39). (E)
(D)
(D)
(D)
(D)
εθ (xi , i) − εθ (xi , i) + εθ (xi , i) − εθ (xi
Proof. Under this assumption, each term in the summation of Eq. (44) can be regarded as the scenario where noise is introduced only at step i. Consequently, the effect of the error can be viewed as the combined contribution of N independent sampling processes with sampling lengths of 1, 2, · · · , N steps, respectively. Each sampling process is initialized with an error (Eq. (45)).
, i) (39)
(D) (D) (D) (D) (D) (D) (D) εθ (xi , i) − εθ (xi , i) ≈ Ei (xi − xi ) = Ei ei
(40)
(i)
∥xi − yi ∥ = ∥ci ε̂ED ∥
Thus, similar to Eq. (21), we can obtain a second-order linear system through a linear approximation of the decryption model (Eq. (40)). However, the driving term is now replaced by (i) ε̂ED given in Eq. (42). This term represents the difference between the noises predicted by the encryption model and the decryption model for the same latent. (D) (D) (D) (D) (i) ei−1 = ai ei+1 + (bi + ci Ei )ei + ci ε̂ED (i) (E) (D) ε̂ED = εθ (xi , i) − εθ (xi , i)
(43)
i<j≤N
(45)
According to the zero-stability of O-BELM (Thm. 4), we can infer that there exists a constant Ki to bound the contribution of i-th model-prediction error, i.e. (D) (i) (i) ε̂ED ∥ ≤ ci Ki ∥ε̂ED ∥
∥Ψi
(46)
Substituting Eq. (46) into Eq. (44) will result in the goal of the proof, that is:
(41) (42)
(D)
∥e0 ∥ ≤
Consequently, the error introduced by model discrepancy can be analyzed theoretically using a method similar to Sec. V. However, it should be noted that the final error (D) expression is e0 , since the error is introduced during the decryption/sampling process.
T −1 X t=0
(D) (t) ε̂ED ∥ ≤
∥Ψt
T −1 X
(t)
ct Kt ∥ε̂ED ∥
(47)
t=0
which indicates a specific upper bound. Based on Prop. 4, we conclude that the cross-model robustness essentially stems from the boundedness of the modelprediction error. This analysis is supported by the experimental observation in Sec. VI-C that the reconstruction quality with the key is positively correlated with the quality without the key.
C. From Zero-stability to Cross-Model Robustness Since the reversibility between encryption and decryption, it is intuitively expected that the impact of noise error decreases
12
Existing encryption methods. Some works encrypt only the final latent xT [29], leaving the entire denoising (or inversion) pipeline unprotected. Others use DNN-based encryption [30], [31] without rigorous security reductions. None of the above addresses the white-box scenario where model parameters are fully public, and an adversary can simply download the model and invert any obtained latent.
D. Practical Limitations: Chosen-Ciphertext Security Our cross-model robustness analysis reveals a fundamental property of the sampling process: it satisfies zero-stability (Def. 4 and Thm. 4). For the O-BELM sampling, the distance between two decryption paths should satisfy Eq. (43). In the (δ) context of decryption, if the ciphertext xT is perturbed, the (δ) resulting error at any intermediate step xi cannot exceed a constant multiple of the initial perturbation. Consequently, our scheme does not aim to provide chosen-ciphertext (CCA) security, and intermediate latents should be treated as sensitive. Practical attack: partial decryption. An adversary who (δ) gains access to xi (e.g., through a side channel or memory dump) can treat it as a new ciphertext with fewer remaining steps. Since the noise has already been partially applied, the remaining steps may not provide sufficient amplification to hide the key mismatch. A concrete countermeasure is to never expose intermediate latents to untrusted parties and to run the entire decryption in a trusted environment. Why IND-CPA remains unaffected. The IND-CPA security of our scheme relies on the difficulty of distinguishing two ciphertexts corresponding to two different plaintexts. From the perspective of a chosen-plaintext adversary, the only available (δ) information is the final ciphertext xT . Constructing a pair of plaintexts whose ciphertexts are statistically close is computationally hard because the forward (encryption) direction exhibits exponential error amplification. The zero-stability property does not help an adversary in the CPA setting because the adversary cannot query the decryption oracle or obtain intermediate states. Hence, the IND-CPA guarantee stands independent of the CCA weakness.
B. Our Positioning To overcome the limitations above, we propose a framework that embeds cryptographic control directly into the diffusion inversion process. Compared to existing methods, our framework has the following features: • White-box setting. It assumes the diffusion model parameters are public; security relies only on the secret key, not on hiding the model. • Exact reversibility. Based on the strictly invertible O-BELM sampler (Sec. II-B), we inject key-dependent noise δt into the noise prediction term. The correct key cancels the noise exactly; an incorrect key leads to exponential amplification of error. • Provable IND-CPA security. In Sec. V, we derive an upper bound on the adversary’s distinguishing advantage, which is exponentially small in the number of steps. • Lightweight computation. Compared to HE/MPC-based solutions, our method only adds PRNG operations and a few operations per step. Runtime is comparable to standard inversion (O-BELM). • Designed for image-to-image reconstruction. The scheme targets reconstructing the original image from a latent, serving as a building block for downstream applications such as editing.
VIII. R ELATED W ORKS
IX. C ONCLUSION
A. Securing Generative Content
We presented a key-controlled inversion framework for diffusion models that enables conditional image reconstruction without hiding model parameters. By injecting key-dependent noise into the noise prediction term of a strictly invertible sampler, our scheme guarantees exact reconstruction with the correct key and exponential error amplification otherwise. We analyzed its security by reducing IND-CPA attacks to a practical hypothesis test and deriving an advantage bound that depends on the accumulated error variance ΣT , thereby ensuring the existence of a security parameter λ. We also empirically validate it through experiments and further discuss cross-model robustness, where decryption quality depends primarily on the inter-model differences, and key-noise injection does not substantially amplify the performance drop inherent to cross-model reconstruction. Our results demonstrate that error propagation in diffusion models can be turned into a powerful security asset. Future work may explore access-conditioned image editing and leverage its cross-model robustness to develop data hiding in the encrypted domain [32] and watermarking with client-side embedding [33], further expanding its application scenarios.
Current image protection methods for diffusion models can be broadly classified into three categories: post-hoc tracing, proactive defenses, and privacy-preserving computation. Post-hoc tracing. Techniques such as image watermarking [27] embed invisible marks into generated content for ownership verification. Model watermarking [1] fingerprints the diffusion model itself. These methods are reactive: they detect misuse after the fact but cannot prevent unauthorized reconstruction or inversion. Proactive defenses. Adversarial perturbation methods (e.g., EditShield [7]) disrupt specific attacks but do not offer a generic key-controlled encryption mechanism. Other access-control frameworks [4], [24], [3] rely on fine-tuning the decoder or modifying the sampling process, yet they lack formal security analysis. Privacy-preserving diffusion. A separate line of work uses homomorphic encryption (HE) [25], [8] or secure MPC [28] to protect text prompts or model weights. These approaches assume a black-box setting, incur huge overhead, and are designed for text-to-image generation. More importantly, the non-linear operations in the U-Net (activation functions) make HE-based image-to-image (I2I) reconstruction infeasible.
13
R EFERENCES
[23] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems, vol. 30, 2017. [24] T. Tanaka and K. Nakamura, “Access control for diffusion models by random masking the covariance of initial noise distribution,” in Asia Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, 2025, pp. 2062–2067. [25] Y. Chen and Q. Yan, “Privacy-preserving diffusion model using homomorphic encryption,” arXiv preprint arXiv:2403.05794, 2024. [26] G. He, Y. Ren, G. Li, G. Feng, and X. Zhang, “Private image synthesis of latent diffusion model with the ciphertext of prompt,” Neural Networks, p. 107678, 2025. [27] J. Zhu, R. Kaplan, J. Johnson, and L. Fei-Fei, “Hidden: Hiding data with deep networks,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 657–672. [28] X. Zhao, X. Chen, X. Chen, H. Li, T. Fan, and Z. Zhao, “Cipherdm: Secure three-party inference for diffusion model sampling,” in European Conference on Computer Vision. Springer, 2024, pp. 288–305. [29] M. Kaur and V. Kumar, “A comprehensive review on image encryption techniques.” Archives of Computational Methods in Engineering, vol. 27, no. 1, 2020. [30] Y. Guo, J. Chang, Y. Zhang, J. Zhang, and S. Ma, “Image encryption and compression based on reversed diffusion model,” in 2024 Picture Coding Symposium (PCS). IEEE, 2024, pp. 1–5. [31] L. Huang, C. Qin, G. Feng, X. Luo, and X. Zhang, “New framework of robust image encryption,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 21, no. 3, pp. 1–22, 2025. [32] Y. Ke, M. Zhang, X. Zhang, J. Liu, T. Su, and X. Yang, “A reversible data hiding scheme in encrypted domain for secret image sharing based on chinese remainder theorem,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 4, pp. 2469–2481, 2021. [33] X. Xiao, Y. Zhang, Z. Hua, Z. Xia, and J. Weng, “Preview helps selection: Previewable image watermarking with client-side embedding,” IEEE Transactions on Dependable and Secure Computing, 2025.
[1] P. Fernandez, G. Couairon, H. Jégou, M. Douze, and T. Furon, “The stable signature: Rooting watermarks in latent diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22 466–22 477. [2] Y. Liu, K. Xu, X. Chen, and L. Sun, “Stable unlearnable example: Enhancing the robustness of unlearnable examples via stable errorminimizing noise,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 4, 2024, pp. 3783–3791. [3] L. Lei, K. Gai, J. Yu, L. Zhu, and Q. Wu, “Secure and efficient watermarking for latent diffusion models in model distribution scenarios,” in International Joint Conference on Artificial Intelligence, 2025. [4] K. Gai, Z. Shen, J. Yu, L. Zhu, and Q. Wu, “Pcdiff: Proactive control for ownership protection in diffusion models with watermark compatibility,” arXiv preprint arXiv:2504.11774, 2025. [5] F. Wang, H. Yin, Y.-J. Dong, H. Zhu, H. Zhao, H. Qian, C. Li et al., “Belm: Bidirectional explicit linear multi-step sampler for exact inversion in diffusion models,” Advances in Neural Information Processing Systems, vol. 37, pp. 46 118–46 159, 2024. [6] Y. Li and M. van der Schaar, “On error propagation of diffusion models,” arXiv preprint arXiv:2308.05021, 2023. [7] R. Chen, H. Jin, Y. Liu, J. Chen, H. Wang, and L. Sun, “Editshield: Protecting unauthorized image editing by instruction-guided diffusion models,” in European Conference on Computer Vision. Springer, 2024, pp. 126–142. [8] G. He, Y. Ren, X. Cai, G. Feng, and X. Zhang, “Private sampling of latent diffusion models for encrypted prompt,” IEEE Transactions on Circuits and Systems for Video Technology, 2025. [9] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International conference on machine learning. pmlr, 2015, pp. 2256– 2265. [10] R. Jiang, G.-C. Zheng, T. Li, T.-R. Yang, J.-D. Wang, and X. Li, “A survey of multimodal controllable diffusion models,” Journal of Computer Science and Technology, vol. 39, no. 3, pp. 509–541, 2024. [11] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840– 6851, 2020. [12] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502, 2020. [13] Y. Huang, J. Huang, Y. Liu, M. Yan, J. Lv, J. Liu, W. Xiong, H. Zhang, L. Cao, and S. Chen, “Diffusion model-based image editing: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. [14] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or, “Nulltext inversion for editing real images using guided diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 6038–6047. [15] B. Wallace, A. Gokul, and N. Naik, “Edict: Exact diffusion inversion via coupled transformations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 532–22 541. [16] G. Zhang, J. P. Lewis, and W. B. Kleijn, “Exact diffusion inversion via bidirectional integration approximation,” in European Conference on Computer Vision. Springer, 2024, pp. 19–36. [17] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A kernel two-sample test,” The journal of machine learning research, vol. 13, no. 1, pp. 723–773, 2012. [18] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255. [19] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision. Springer, 2014, pp. 740–755. [20] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” arXiv preprint arXiv:1710.10196, 2017. [21] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10 674–10 685, 2021. [22] A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in International conference on machine learning. PMLR, 2021, pp. 8162–8171.
14