DIME: Query-Efficient Framework for Membership Inference on Diffusion Models

DIME: Query-Efficient Framework for Membership Inference on Diffusion Models

arXiv:2608.22824v1 [cs.LG] 24 Aug 2026

Tue Do, Daniel Alabi University of Illinois at Urbana-Champaign

Abstract

independent of the specific content involved: if a model is trained on medical imaging, biometric data, or any dataset implicitly associated with a sensitive category, confirming that a specific individual’s data was included can constitute a privacy harm on its own [21]. Second, verifying whether a model still reflects a specific individual’s data after a deletion request is a practical necessity under data-protection regulation [1, 4, 17], and one an external party generally cannot resolve without some way to test the model’s behavior directly. Third, and increasingly salient specifically for generative image models, diffusion models have been shown not merely to learn generalizable visual concepts from their training data, but in some cases to memorize and reproduce specific training examples [6, 43], raising direct concerns about the copyright status and provenance of generated content that have already surfaced in litigation [48]. Membership inference offers a formal, auditable way to address each of these concerns: given a trained model and a candidate data point, determine whether that point was used during training [42]. A successful membership inference attack against a deployed diffusion model would allow an individual, artist, or auditor to establish, without access to the training pipeline itself, if a specific image was used, making membership inference a central tool both for attacking and for auditing the privacy properties of these models. Existing membership inference attacks against diffusion models [13, 27, 31, 34] have made substantial empirical progress, but share a common limitation: the test statistics they employ, including loss values and single-step reconstruction error are chosen based on intuition and empirical validation, without a derived connection to what the best possible statistic would be for this task. This leaves open a basic question:

Membership inference attacks expose whether individual records were used to train a model, yet existing attacks on diffusion models are largely heuristic and can require substantial query budgets. We introduce DIME (Denoiser Ideal Membership Error), a theoretically grounded and queryefficient framework for membership inference on diffusion models. Our starting point is an exact characterization of the optimal diffusion denoiser for a finite training set, which reveals that membership leakage is governed by the denoiser’s implicit reconstruction error. This error decomposes into two complementary signals: a bias term, capturing reconstruction accuracy, and a previously unexplored local crowding term, capturing the geometry of nearby training examples. Both admit efficient estimators using only model queries, yielding a practical attack with as few as two queries. Across CIFAR10/100, STL10-U, CelebA, and ImageNet, DIME consistently outperforms prior attacks at comparable or substantially lower query cost, improving TPR at 1% FPR by up to 3×; remarkably, its two-query variant can outperform existing 30-query baselines. Finally, we suggest, discuss, and evaluate specific defenses to counteract such powerful membership tests.

1

Introduction

Diffusion models [20, 44, 45] have become one of the dominant paradigms for high-dimensional generative modeling, powering widely deployed systems for image synthesis [33, 36, 39], and increasingly extending to audio, video, and scientific domains. In addition to sampling quality, diffusion models beat earlier pixel-level autoregressive approaches [46, 47] in sampling speed and throughput. As with most large-scale machine learning systems, the specific composition of a deployed model’s training data is typically not disclosed, making it difficult for an individual, creator, or auditor to independently verify whether particular data was used in training. This opacity raises privacy concerns of several distinct kinds. First, whether a particular individual’s data was used in training can itself be a sensitive fact

Given a diffusion model’s specific training objective and architecture, what does an optimal membership inference statistic actually look like, and can existing empirical attacks be understood as approximations to it? We answer this question directly. Our first contribution is 1

theoretical, and proceeds in three steps. First, we derive an exact, closed-form characterization of the idealized denoiser: the MSE-optimal noise-prediction function a diffusion model would compute at any query point, given a finite training set. We show it has an explicit, interpretable form: a responsibility-weighted average over training examples, where each example’s weight is a softmax over its distance to the query. Every prior diffusion membership inference attack evaluates the real denoiser at a single point and treats its output as an empirically useful signal; we instead start from what the optimal denoiser provably computes, and derive a test statistic from that closed form directly. Second, we show this idealized denoiser admits an exact bias-variance decomposition of a natural estimation error: how far a candidate point’s model-implied reconstruction deviates from the candidate itself. The bias term is exactly the response magnitude used, in various forms, by every existing attack we compare against. The variance (crowding) term, measuring how densely the candidate is surrounded by other similar training points, has no analog anywhere in the diffusion membership inference literature. Because this analytical decomposition is exact rather than heuristic, both terms are available from the very same closed-form object used to justify the first term alone, meaning existing attacks have been discarding a mathematically principled, complementary source of signal we capture. See Figure 1. Third, we show both terms admit efficient estimators from forward evaluations of the trained network alone, requiring no gradient access, no shadow models, and no full generative sampling. The bias term is estimated by Monte Carlo averaging, and the variance (crowding) term via a Hutchinson trace estimator applied to the denoiser’s local Jacobian, using the same perturbed queries already spent on the first estimate. This construction is what makes the theory practically actionable. Our second contribution is practical: we introduce DIME (Denoiser Ideal Membership Error), a test statistic derived directly from this decomposition, together with a queryefficient estimator of each component. We measure query cost as the number of forward passes through the trained model; consistent with prior diffusion membership inference work [13, 27, 31, 34], reflecting a gray-box threat model in which the adversary has query access to the model’s noiseprediction output at chosen inputs and timesteps, but no access to weights, gradients, or training infrastructure (Section 2.4). A realistic adversary querying a hosted or commercially deployed diffusion model faces per-query cost, rate limits, or usage monitoring that a query-heavy attack may not survive in practice. This also distinguishes our threat model from substantially more expensive alternatives: shadow-model attacks require the adversary to train multiple auxiliary models [42], and reconstruction-based attacks require full iterative sampling via the reverse diffusion process per candidate point [6, 32]; orders of magnitude more expensive than a

single forward pass, and correspondingly harder to deploy unnoticed or at scale. The same efficiency matters symmetrically for the auditing use case motivating this work: a privacy auditor verifying a deletion request or investigating potential memorization benefits from an attack that is cheap and fast to run repeatedly, not one that requires retraining models or generating samples per candidate point. We show (Section 5) that DIME achieves SoTA attack performance with as few as 2 total queries, and increasing in effectiveness as much as query budget allows. Across five diffusion checkpoints and datasets with substantially different structure, DIME outperforms the strongest prior baseline on every standard metric (ASR, AUC, and TPR@1%FPR) at matched or lower query cost; we observe DIME to improve TPR@1%FPR by up to 3× on CIFAR-10 from SoTA methods while remaining query efficient. We additionally evaluate DIME against differential privacy [14, 15], the standard formal algorithmic defense against privacy adversaries, and find that differential privacy remains an effective defense even against our empirically strongest attack. Contributions. In summary, this work makes the following contributions: • An exact, closed-form theoretical characterization of the MSE-optimal denoiser for a diffusion model trained on a finite dataset (Section 4), including an asymptotic isolation result explaining the behavior and the limitations of prior single-query attacks, and an exact bias-variance decomposition of a novel estimation error statistic. • DIME, a query-efficient membership inference attack derived directly from this decomposition, requiring only one reference model query and no generation of full samples, gradient access, or shadow-model training (Algorithm 1). • An extensive empirical evaluation across five diffusion checkpoints spanning DDPM [20] and Guided Diffusion [12] architectures, demonstrating that DIME outperforms existing baselines at matched or lower query cost across every evaluated metric, and an evaluation of DIME’s robustness under DP-SGD (Section 5).

2 2.1

Preliminaries and Background Notation

We write x ∼ D to denote x a sample drawn from distribution D. For some randomized mechanism M , we also can denote x ∼ M as an instance outputted by M . For finite set D = {x1 , . . . , xn }, we write x ← D to denote x assigned to a uniformly random element of set D . We typically will elect to use a subscript xi to refer to an indexed dataset member, and superscript x∗ to refer to an attack point of inference to glean a 2

Figure 1: Intuition for DIME’s two signals. For a candidate x∗ , mlocal is the responsibility-weighted mean of nearby training points. An isolated member’s mlocal coincides with x∗ (low bias, low crowding); an isolated non-member’s deviates substantially (high bias). A non-member inside a dense cluster can have mlocal coincidentally close to x∗ (low bias) while the contributing points remain widely spread (high crowding, shaded ellipse). membership decision from. We use superscript x(t) to denote a sample from a particular timestep in the diffusion process. We use capital letters (i.e. X) to denote random variables.

2.2

learned Gaussian transitions starting at pθ (x(T ) ) ∼ N (0, Id ) and pθ (x(t−1) |x(t) ) ∼ N (µθ (x(t) ,t), Σt ), where   1 βt µθ (x(t) ,t) := √ ε̂θ (x(t) ,t) , x(t) − √ αt 1 − αt

Diffusion Models

Diffusion models [20, 44, 45] are a class of latent-variable generative models that represent data generation through a sequence of intermediate random variables, emerging as a leading approach for high-dimensional generative modeling, particularly in continuous domains such as images, audio, and scientific data. We focus on the discrete DDPM formulation of diffusion models, closely following [20]. Supposing x(0) ∈ Rd is a data sample drawn from unknown data distribution D to be learned, a diffusion model introduces latent random variables

Σt :=

1 − αt−1 βt . 1 − αt

Here, ε̂θ (x(t) ,t) is a trained noise prediction network that minimizes the expected noise for some Dtrain ∼ D∗ : h i p Lt := Ex←Dtrain ,z∼N (0,Id ) ∥ε̂θ ( αt x + σt z,t) − z∥2 . (1)

2.3 Membership Inference as Hypothesis Testing Given a model f (·; θ) trained on a dataset Dtrain and a candidate point x∗ , membership inference is the task of deciding whether x∗ ∈ Dtrain . A membership inference attack (MIA) is formalized as a (possibly randomized) function A that, given x∗ and some level of access to f (·; θ), outputs a bit b̂ ∈ {0, 1}, representing a guess as to whether x∗ was a training member. In particular, membership inference implicitly invokes a classical hypothesis testing framework. Given some observations from a trained model, our goal is to distinguish between two statistical hypotheses:

X 1:T = (X 1 , . . . , X T ), so that the diffusion forward process repeatedly adds Gaussian noise such that p X t |X t−1 ∼ N ( 1 − βt X t−1 , βt Id ), where for all t ∈ [T ] we have fixed constants βt ∈ (0, 1) called the noise schedule. For notational convenience, we write √ αt = 1 − βt and αt = α1 · · · αt . We might sometimes use σt = 1 − αt . From induction, we can also show that for any t (see Equation 4 in [20]): p X t |X 0 ∼ N ( αt X0 , (1 − αt )Id ).

• H0 : x∗ ∈ / Dtrain (Null Hypothesis) • H1 : x∗ ∈ Dtrain (Alternate Hypothesis) Membership inference, across every domain in which it has been studied, reduces to a common methodological backbone: a test statistic computed from a candidate point and the

Subsequently, joint (parameterized) distribution pθ (x0:T ) is called the reverse process and defined as a Markov chain with 3

Example 2. Given candidate image x∗ , the adversary chooses t√= 10, samples 10 i.i.d. zk ∼ N (0, Id ), and constructs gk = αt x∗ + σt zk . The adversary sends the 10 pairs (gk ,t) to the model and receives 10 vectors of the form ε̂θ (gk ,t). This process counts as 10 queries.

target model, thresholded to produce a binary membership decision. That is, given some test statistic T computed from observation, an induced membership inference adversary A with threshold τ outputs membership decision:

A = 1[T ≤ τ].

(2) We summarize our formal threat model below:

For any given test statistic T , prior membership inference work in [38] suggests for the Bayes-optimal τ to be calibrated on a held-out validation set. For each test statistic evaluated in this work we follow that protocol for optimal performance while adhering to our specified threat model. We give the formal security game underlying this task, and the specific access A is granted in this work, in Section 2.4.

2.4

Envisioned attacker. The adversary in Aq has grey-box access to the target model. The adversary additionally has query access to the underlying data distribution D (used to construct a held-out calibration set for selecting attack hyperparameters), but not to the training set D itself. That is, as is standard in empirical membership inference evaluation [13, 27, 31, 34], we allow the adversary auxiliary calibration data, generated independently of the challenge bit, which may contain labeled examples of known members and non-members of the target model. This information may be used to select hyperparameters and decision thresholds, but does not reveal the membership of the challenge point itself. The adversary’s goal is a binary decision: whether a specific, adversary-supplied challenge point x was a member of D .

Threat Model

We define the adversarial model Advq via security game, drawing from the framework presented in [5]. Definition 1 (Membership inference security game). The game proceeds between a challenger C and adversary A i. The challenger samples a training dataset D ∼ D∗ from underlying data distribution D, and trains a model f (·; θ) ∼ M (D ) on training set D ,

Threat surface. The attack targets the deployment phase of the ML lifecycle: it requires only query access to an alreadytrained model exposed via an inference interface (e.g., a hosted diffusion-model API, or any product surface that allows a caller to supply a noised input and timestep and receive the corresponding denoising output), and makes no assumptions about and requires no access to the data curation or training phases. The attack surface is specifically the model’s noise-prediction function as exposed at inference time, not the training pipeline or model weights at rest.

ii. The challenger flips a bit b, and if b = 0 samples a fresh challenge point x ∼ D (such that x ∈ / D ). Otherwise, the challenger selects a point from training set x ← D . iii. The challenger sends x to the adversary. iv. The adversary gets q queries to the model f (·; θ) and query access to the distribution D, outputs a bit b̂ ∼ A D, f (x). We can think of In corresponding to bit 1 and Out corresponding to bit 0.

Generality. The attack makes no assumption about model architecture, resolution, or conditioning beyond the standard diffusion forward/reverse process and training objective; both the theoretical construction (Section 4) and the resulting statistic apply universally across diffusion architecture. We demonstrate this empirically (Section 5) across five checkpoints spanning a 64× range in input dimensionality and both unconditional and class-conditional settings, without any architecture-specific modification to the attack.

v. Output 1 if b̂ = b, and 0 otherwise. We specialize Definition 1 to the diffusion setting by taking f (·; θ) to be the trained noise-prediction network εθ (·, ·), and by making explicit what a “query” means: the adversary submits a pair of noised input and timestep (g,t) and observes εθ (g,t). This is the same gray-box query interface assumed by every prior attack we compare against [13, 27, 31, 34], so that our comparison to prior work reflects a difference in test statistic and query efficiency, not a difference in assumed adversarial capability. By grey-box, we mean that the adversary has query access to εθ (g,t) for adversary-chosen (g,t), but no access to model weights, gradients, or internal activations, and no ability to backpropagate through the model.

Practicality. The attack requires only q + 1 total model queries: a single shared baseline query plus q perturbed queries, each as a single forward pass through the denoising network, with no gradient computation, no model retraining, and critically, no generation of full samples via the reverse diffusion process. Our attack stands in contrast to shadow-model attacks [42] (requiring the adversary to train multiple auxiliary models) and reconstruction-based attacks [32] (requiring full iterative sampling per query, orders of magnitude more expensive than a single forward pass). We show (Section 5)

Example 1. Given candidate image x∗ , the adversary chooses t = 10, and sends (x∗ ,t) to the model and receives the predicted noise vector ε̂θ (x∗ , 10). This entire vector counts as one query. 4

3

that in some settings our attack achieves above SoTA results with as few as 2 total queries, making it deployable against a realistic, rate-limited or pay-per-query inference API where an adversary’s query budget is a genuine, binding constraint.

2.5

3.1

Related Work Membership Inference

Model inversion [6, 7, 18, 19, 50] and membership inference attacks (MIA) are a well studied privacy problem that predate the introduction of the diffusion process as a generative architecture. The notion of a test statistic for membership inference predates machine learning; Homer et al. [21] first demonstrated this kind of inference showing that an individual’s presence in a pooled genomic mixture could be detected from a measurable statistical bias in comparing their known genotype against the mixture’s aggregate allele frequencies. In the machine learning context, MIA was first defined for general supervised classification tasks [42] with shadow model training methods. Subsequently, the literature has moved to MIA for generative image models [8] starting with generative adversarial networks (GANs) for its clear privacy and intellectual property implications.

Evaluation of MIAs on Diffusion Models

Because ground-truth membership is not observable for a model deployed in practice, evaluating a membership inference attack requires a controlled setting in which membership is known by construction. Following standard practice, we partition a pool of candidate points into a member set (points drawn from Dtrain ) and a held-out set (points drawn from the same underlying distribution D but excluded from Dtrain ), and report attack performance over this labeled evaluation set. This labeling is an artifact of evaluation, not a capability granted to the adversary in Definition 1, who must still output b̂ from x∗ alone. Given a test statistic T and this labeled evaluation set, recall the induced decision rule in Equation (2), i.e. a smaller value of T indicates membership. Varying τ over all thresholds traces out a pair of rate functions,     TPR(τ) := P T ≤ τ | H1 , FPR(τ) := P T ≤ τ | H0 ,

3.2

Membership Inference for Diffusion

Originally, MIA in machine learning was studied in the “black-box” threat model, where attacks are performed without knowledge of the model weights or structure, relying only on input and output pairs from a deployed model. Prior work [38] shows that under standard supervised classification, a single scalar is asymptotically sufficient for determining membership, that is more model access to intermediate-layer activations and gradients is uninformative as in a “white-box” threat model. The most common class of diffusion model MIAs, “grey-box” attacks span between these two extrema, generally having access to weights and/or internal representations, and crucially may query the model for particular test points. One of the first works in the diffusion MIA space Loss takes the training loss as membership signal [31]. Similarly, follow-up works in SecMI and PIA use other diffusion quantities of reconstruction step error and ground-truth trajectories and observe practical improvements in performance [13, 27]. Finally, SimA improves on prior work by introducing Monte Carlo sampling for attack stability, electing to use the score prediction as the test statistic [34]. These prior methods are restricted to norm-based statistics, motivated heuristically or empirically. Our contribution goes beyond just another test statistic, we uniquely begin from the theoretical characterization of the query object (the denoiser) under empirical risk minimization and derive an overlooked membership relevant quantity in the crowding term.

estimated empirically over the labeled evaluation set. We report three standard metrics derived from these rates throughout this work: • ASR (attack success rate), the best balanced accuracy achievable over all
thresholds, ASR := maxτ 21 TPR(τ) + (1 − FPR(τ)) . • AUC, the area under the resulting ROC curve (FPR(τ), TPR(τ)), equal to the probability that T assigns a lower value to a randomly chosen member than to a randomly chosen non-member. • TPR@1%FPR, the true-positive rate at the most permissive threshold whose false-positive rate does not exceed 1%, TPR@1%FPR := max{TPR(τ) : FPR(τ) ≤ 0.01}. Unlike AUC and ASR, TPR@1%FPR evaluates attack performance specifically in the low-false-positive regime: the setting in which an adversary makes confident, individuallydefensible membership claims, which requires a low falsepositive rate regardless of average-case performance elsewhere on the ROC curve [5]. These metrics, and the evaluation protocol above, are not specific to any model class; we apply them to the diffusion setting without modification, consistent with the generality of the attack itself. What differs across the model checkpoints we evaluate is only the underlying population data distribution D and the resulting training member/held-out pools, described per dataset in Section 5.

3.3

Misc. Diffusion Security

Other adversarial work studies backdoor attacks on diffusion models, engineering a compromised training process so when a certain trigger is present, the model emits attacker-chosen 5

content [9]. We evaluate our attacks exclusively in the image generation domain, but prior work has investigated MIA in other data modalities, including text-to-image and text-tovideo generation [24, 29, 41]. Unlike our evaluated methods, these methods invoke the text modality and rely on tokenlevel likelihood signals that are specific to auto-regressive generation. Prior work shows that a few poisoned samples can corrupt specific target prompt outputs in text-to-image diffusion generation [40]. In contrast, our proposed method is tailored to diffusion models in the image-to-image generative domain and invokes the specific training loss objective. The theoretical underpinning to our metric is not directly applicable to cross-modalities, but we believe our methodology can be extended to detect training members of text-conditioned diffusion models.

4

the loss: h

√ g αt n ε̂ (g,t) = − ∑ ri (g)xi , σt σt i=1 ∗

where ri (g) = ∑n si (g) , and si (g) := exp(− 2σ1 2 ∥g − j=1 s j (g) t √ αt xi ∥2 ). Proof. We can write the loss function

Lt =

Membership Inference Frameworks: Algorithms and Theory

p 1 n Ez∼N (0,I) [∥ε̂θ ( αt xi + σt z,t) − z∥2 ]. ∑ n i=1

Expanding the expectation, and writing φ(z) = (2π)−d/2 exp(− 12 ∥z∥2 ) as the standard normal density, we have 1 n ∑ n i=1

Z

p ∥ε̂θ ( αt xi + σt z,t) − z∥2 φ(z)dz

√ √  2   g − αt xi g − αt xi dg ε̂θ (g,t) − φ σt σt σtd √ √   2   Z n g − αt xi 1 g − αt xi = d ∑ ε̂θ (g,t) − φ dg, σt σt nσt i=1 1 n = ∑ n i=1

Roadmap of the Framework

A perfectly trained MSE denoiser computes a posteriorweighted average over possible training examples. For members, at low noise, this posterior concentrates on the member itself. This suggests measuring reconstruction error. The reconstruction error has two pieces: displacement of the posterior mean (“bias”) and dispersion of training examples (“crowding”). The first is essentially what norm-based attacks [13, 27, 31, 34] already see; the second is new. Both can be estimated from model queries, and their combination gives DIME.

4.2

i

can be explicitly written as

In this section we first introduce and characterize the asymptotic diffusion denoiser to ground our new membership inference attack. Subsequently, from our theoretical analysis, we motivate and define a novel diffusion training test statistic which takes advantage of the unique membership error quantity we derive.

4.1

p

Lt := Ex←Dtrain ,z∼N (0,Id ) ∥ε̂θ ( αt x + σt z,t) − z∥2

Z



where the first equality follows from the change of variables √ √ g− ᾱt xi g = ᾱt xi + σt z. Indeed, z = σt , and since g ∈ Rd , the Jacobian determinant of this transformation is σtd . Hence dz = σ1d dg. Recall that we want to find the denoiser function t

ε̂∗ (·,t) that minimizes the loss Lt , so we can minimize the integrand pointwise, which means that for any g, we want  √  2  √  φ g− σtαt xi . to minimize ℓg := ∑ni=1 ε̂θ (g,t) − g− σtαt xi dℓ

Clearly, this expression is quadratic, so setting d ε̂g = 0, we θ get √ √    n  g − αt xi g − αt xi ∗ ·φ =0 ∑ ε̂ (g,t) − σt σt i=1

Idealized Denoiser

We term the idealized denoiser as the theoretical function that minimizes the empirical diffusion training risk described in Equation 1. We characterize an idealized denoiser ε̂θ∗ that minimizes Lt defined in (1) over all continuous functions described as f : Rd ×[T ] → Rd . The Universal Approximation Theorem [11, 22, 23] tells us that any continuous function can be approximated with arbitrary closeness by a neural network, meaning that given enough training over the appropriate loss objective, a learned model approaches the function described in Lemma 1.

√ g − αt xi si (g) σt i=1 i=1 √ g n αt n ∗ =⇒ ε̂ (g,t) = ri − ∑ ∑ ri (g)xi σt i=1 σt i=1 √ g αt n = − ∑ ri (g)xi , σt σt i=1 n

n

=⇒ ε̂∗ (g,t) · ∑ si (g) = ∑

where the last equality comes from the fact that ∑ni=1 ri = 1, which gives us the desired statement. □

Lemma 1 (Idealized Denoiser). Given a fixed training dataset Dtrain = {x1 , . . . , xn }. The optimal denoiser ε̂∗ that minimizes 6

We show that the idealized denoiser scales the input with an offset described by the posterior dataset “responsibilities” ri (g), which are precisely the posterior probability that training point xi was the source of the noisy observation g, under the same generative process the diffusion model was trained to invert. Structurally, ri (g) is a softmax over negative squared distances, acting as a smooth, probabilistic version of nearestneighbor lookup. In other words, the optimal denoiser attends over the training set. We show that as the diffusion noise σt → 0 asymptotically, the softmax becomes sharply peaked, and nearly all responsibility concentrates on whichever training point is closest to g.

the idealized denoiser at low timesteps approaches jump discontinuity at Voronoi boundaries, which we find empirically to not be well captured by actual neural networks, resulting in a gap from our theoretical contribution, and our attacks tend to peak in performance early but not at the beginning of the diffusion timescale (see Figure 3).

4.3

The responsibility distribution r(g) gives a direct, intuitive account of why detecting membership requires more than checking a single number as in prior attacks. Consider three cases, distinguished only by the local geometry of the training set around a candidate x∗ . Case 1: an isolated member. If x∗ ∈ D and no other training point lies nearby, responsibility concentrates almost entirely on x∗ itself: rx∗ (g) → 1 as t → 0 (Lemma 2). The model’s implicit reconstruction of the candidate’s source is, in expectation, exactly x∗ , and because responsibility is concentrated on a single point, there is essentially no disagreement across the (one) candidate source either. Case 2: an isolated non-member. If x∗ ∈ / D but sits near a single, otherwise-isolated training point x j ̸= x∗ , the posterior instead concentrates on x j : r j (g) → 1. Responsibility is still sharply concentrated on one point, but the model’s reconstruction is now centered on the wrong point, and it differs substantially from x∗ . This mismatch alone is enough to correctly flag x∗ as a non-member, which is the basis of prior diffusion membership inference attacks [13, 27, 31, 34]. Case 3: a non-member inside a dense cluster – the case that motivates our construction. Suppose instead x∗ ∈ / D, but several training points surround it closely and roughly symmetrically. No single point dominates the posterior, and responsibility spreads across the nearby cluster. Critically, the weighted average of these several nearby points can land, by simple geometric coincidence, very close to x∗ itself, even if x∗ matches none of them individually. The model’s reconstruction therefore looks, at a glance, just as accurate as in the first case: a statistic that only checks how close the reconstruction lands to x∗ cannot tell these two cases apart, which motivates our new attack method.

Definition 2. Fix some dataset D = {x1 , . . . , xn } where xi ∈ Rd . Denote di (x) = ∥x − xi ∥ for any x ∈ Rd . Denote Vi = {x : di (x) = min j d j (x)}. Vi is exactly the Voronoi cell associated with training point xi in relation to the rest of D . Lemma 2 (See Lemma 4). Consider a sequence ( j)

s

( j)

(g)

( j)

i functions ri (g) = , where si (g) = ( j) ∑ k  sk (g)  p exp − 2(σ1( j) )2 ∥g − a( j) xi ∥2 , a( j) = 1 − σ( j)2 , and

of

Test Statistic

( j)

σ( j) → 0+ . Then ri (g) −→ 1Vi (g) a.e. As the diffusion noise tends to zero, the soft posterior responsibilities become a hard nearest-neighbor rule. Intuitively, Lemma 1 states that a perfectly-trained denoiser does not just guess noise in the abstract. For any noisy input, its best possible prediction is built by implicitly asking “which training images could plausibly have produced this?” and blending their identities together, weighted by the posterior of how well each data member can produce the forward noised input. If the noisy input came from an image the model was actually trained on, and the added noise is small, that image is overwhelmingly the best match: the model’s responsibility blend collapses almost entirely onto it, and its prediction becomes highly accurate, nearly canceling out the true noise exactly. If the input instead came from an image the model has never seen, there is no single, exact match to fall back on, but only an approximate blend of similar-looking training images, differing from the actual noise added to the non-member. This mismatch is not just larger on average for non-members; the same underlying construction lets us measure a second, complementary signal, particularly how much that blend of “similar-looking” images is spread out, or crowded together, around the query, giving two independent, theoretically grounded handles on membership. Lemma 2 shows that as the diffusion noise tends to zero, the idealized denoiser becomes a simple affine function of the input, precisely offset by the nearest dataset member, suggesting the best membership test lies in early timesteps. See Appendix B for plots of the idealized denoiser on synthetic data. However,

Membership as estimation error. The idealized denoiser derived above is, by construction, the function that most accurately recovers a training example from its noised observation. For a genuine training point xi , this gives it a distinguishing √ property: the denoiser’s responsibility distribution r( αt xi ) concentrates on xi itself in the small-noise limit (Lemma 2), meaning the model’s implicit reconstruction of the identity of the source point is, in expectation, exactly correct. For a point x∗ that was not in the training set, no such guarantee exists; instead the model’s best reconstruction is some responsibilityweighted average of whichever training points happen to lie nearby, which do not necessarily coincide with x∗ itself. 7

This suggests a direct membership statistic: rather than testing attack point x∗ ’s response against a hypothesized distribution, we measure how far the idealized model’s implicit reconstruction of x∗ deviates from x∗ itself, e.g. an estimation error. Members should exhibit small estimation error, whereas non-members derive larger error.

in both: an isolated member drives both terms toward zero as t → 0 (Lemma 2), and we find that a non-member’s estimation error can be large through either channel, from a poorly-reconstructed average location and/or a crowded neighborhood (See Figure 4). Finite-sample estimators. Neither B nor V is directly observable; both are defined in terms of an expectation over the full training set. We estimate each using only forward √ evaluations of ε̂∗ at q randomly perturbed points around g = αt x∗ . iid For step size γ > 0 and z1 , . . . , zq ∼ {−1, +1}d (Rademacher), we can define B̂ (estimator of B) as:

Defining the estimation error. Denote I ∼ r(g) the random variable indexing a training point drawn√according to the model’s responsibility distribution at g = αt x∗ . That is, I is defined as the random variable such that P(I = i|g) = ri (g), √ and let Y = ᾱt xI be the corresponding (scaled) training point random variable. We define the estimation error of the optimal denoiser at x∗ as p   M(x∗ ,t) = E ∥Y − αt x∗ ∥2 , (3)

B̂(x∗ ,t, γ; q) =

Exact decomposition. The standard bias-variance identity [26] applied to Y gives p M(x∗ ,t) = ∥E[Y ] − αt x∗ ∥2 + tr(Cov(Y )) . (4) | {z } | {z }

Both estimators share the same q perturbed queries and a single base query at g, for a total query cost of q + 1 per evaluated (t, γ). See Lemmas 5 and 6 in Appendix A for more detailed analysis of our finite-sample estimators of the membership-error bias-variance decomposition.

variance

Example 3. Consider a dataset with two points {−1, 1}. For x∗ = 0, Y = −1 with probability 1/2, and Y = 1 with probability 1/2, so E[Y ] = 0 and so the bias is zero despite x∗ not being a training point. But the variance of Y is 1. This motivating example necessitates the variance/crowding term.

From the ideal quantity to a practical statistic. Defined above are finite-sample estimators B̂ and V̂ of B and V in Equation (5), each requiring only forward evaluations of ε̂θ . We find empirically that allowing a free weight hyperparameter φ which tunes the relative weight of the two estimation error components improves attack performance at low tuning cost; in practice we calibrate the relative weighting of the two error components directly against held-out data, remaining in line with the adversarial model’s knowledge of the data distribution (Definition 1), rather than fixing it at its population-exact value, which finally yields DIME (Denoiser Ideal Membership Error):

From Corollary 1 in Appendix A, ε̂∗ (g,t) = σ1t (g − E[Y ]), so the bias term is exactly σt2 ∥ε̂∗ (g,t)∥2 =: σt2 B2 , where B is the response magnitude of the optimal denoiser at x∗ . The variance term is tr(C) =: V , where C = Cov(Y ) = √ CovI∼r(g) [ αt xI ], which is the expected squared distance of a responsibility-sharing training point from the weighted centroid of its neighborhood, which we show has a closed-form estimator via the Hutchinson trace estimator [25] applied to ∇g ε̂∗ (g,t). Thus M(x∗ ,t) = σt2 B2 +V,

(6)

a Monte Carlo estimate of the expected response magnitude at radius γ. From Lemma 6 in Appendix A, we can define V̂ estimating an affine transformation of V using a Hutchinson trace estimator [25] of the gradient of the idealized denoiser tr(∇g ε̂∗ (g,t)), incurring only O(γ) estimator bias. We define explicitly
1 q z⊤ k ε̂θ (g + γσt zk , t) − ε̂θ (g,t) ∗ V̂ (x ,t, γ; q) = ∑ . (7) q k=1 γσt

the expected squared distance between x∗ and a training point drawn in proportion to how much the model currently attributes it to x∗ . This is a natural, direct measure of reconstruction quality: it is small precisely when the model’s implicit best guess at x∗ ’s identity is reliably close to x∗ itself, and large when it is not.

bias2

1 q ∑ ε̂θ (g + γσt zk , t) , q k=1

DIME(x,t, γ, φ) = cos φ · B̂2 + sin φ · V̂ ,

(8)

the calibration-weighted combination of the two estimation error components of the ideal estimation error, with t, γ, and φ selected by grid search on a held-out split. Because (cos φ, sin φ) is the standard parameterization of every point on the unit circle and only the direction (ratio), affects test statistic performance, restricting the combination w1 B̂ + w2V̂ to unitnorm weights loses no achievable ranking while reducing

(5)

exactly; the estimation error decomposes, with no approximation, into two independently estimable and individually meaningful quantities: 1. how far the average nearby training point sits from x∗ , and 2. how spread out the nearby training points are around one another. Membership manifests 8

aim to empirically answer the following research questions: RQ1. Our attack introduces a membership error variance crowding term which captures the denoiser’s local structure around an attack point, which has no analog in prior membership inference literature. Does a test statistic grounded in idealized membership error carry membership-relevant information beyond previous simple query-based methods? RQ2. How does the performance of diffusion-model membership inference attacks change under different query budgets, and which attack provides the best trade-off between privacy leakage detection and query cost (e.g., inference time or computation) when the number of queries is limited? RQ3. The theoretical characterization makes no assumption about resolution, conditioning, dataset structure, or training procedure. Does the resulting attack’s advantage over prior methods persist when these assumptions are stressed, i.e. at substantially higher resolution and under class-conditioning (Guided Diffusion / ImageNet), under non-standard dataset geometry (STL10-U, CelebA), and under a training procedure with a formal privacy guarantee (DP-SGD)? As a quick summary, we find the answer to all three research questions is yes. On RQ1, our DIME test statistic carries substantially more signal than any single-query statistic in prior work; at matched query cost, our method improves TPR@1%FPR by up to 3× over the strongest baseline (SimAMC) on CIFAR-10 (50.8% vs. 16.1%). On RQ2, this advantage holds even at minimal query budgets: our cheapest setting (2 queries) already matches or exceeds every baseline’s most expensive setting (30 queries) on multiple datasets, including a 1.5× TPR@1%FPR improvement over SimA-MC’s best multi-query result on ImageNet (14.3% vs. 9.33%) at a fraction of the query cost. On RQ3, this advantage persists under substantial architectural and distributional stress, maintaining the best per-query performance across dataset and model fidelity while disappearing entirely under DP-SGD: our method, like every baseline, collapses to nearly chance-level performance (AUC within 48-53%) at every tested privacy budget, confirming that DP-SGD remains an effective defense even against our strongest attack evaluated in this work.

Algorithm 1: DIME: Denoiser Ideal Membership Error Input: candidate x∗ ; timestep t; perturbation radius γ; query budget q; calibrated combination angle φ; calibrated threshold τ Output: decision b̂ ∈ {0, 1} √ membership ᾱt x∗ 1 g← 2 εbase ← ε̂θ (g,t) // 1 query 3 bias_sum ← 0; crowd_sum ← 0 4 for k = 1 to q do 5 Draw zk ∼ Rademacher({−1, +1}d ) 6 gk ← g + γσt zk 7 εk ← ε̂θ (gk ,t) // 1 query 8 bias_sum ← bias_sum + ∥εk ∥ z⊤ (εk − εbase ) 9 crowd_sum ← crowd_sum + k γσt 10 B̂ ← bias_sum/q 11 V̂ ← crowd_sum/q 2 12 score ← cos φ · B̂ + sin φ · V̂ 13 b̂ ← 1[score ≤ τ] 14 return b̂

the search from two free parameters down to one. Unlike the fixed-weight construction, this combination adapts to standard estimation error as well as which error component is, in practice, more reliably estimable at the query budget available, while remaining a direct estimator of the same quantity: how far the optimal denoiser’s implicit reconstruction of x∗ deviates from x∗ itself. We provide a precise algorithm of our DIME method in Algorithm 1. As a brief aside, our implementation standardizes the B̂, V̂ terms in hyperparameter calibration search for numerical convenience. With an explicit definition, it remains to evaluate the effectiveness of our new attack on diffusion membership inference. In the next section, we design and run experiments to quantify the membership signal contained in our idealized denoising estimation error test statistic.

5.1 5

Experiments: Setup and Evaluation

Datasets and Checkpoints

We evaluate previous baselines and our attack over five benchmark datasets spanning a deliberate range of scale and domain, namely CIFAR-10/100 [28], STL10-U [10], CelebA [30], and ImageNet-1k [37]. We briefly describe how model weights and data splits for each dataset are obtained for reproducibility. All experiments are done on a single NVIDIA A40 GPU.

Although membership inference has been extensively studied for discriminative and generative models, diffusion models have a fundamentally different iterative denoising process and query interface, which may affect both the design and effectiveness of membership inference attacks. Understanding these differences is important for accurately assessing privacy risks in diffusion models. Our work on the idealized denoiser (Lemma 1) characterizes the model outputs, raising specific testable questions about both when existing attacks should be expected to succeed or fail, and what additional structure an idealized attack could exploit that prior heuristics do not. We

• CIFAR-10/CIFAR-100: We reuse the public model checkpoints and dataset member/held-out splits (25k/25k) drawn from [13]. • STL10-U: We trained a model on a single NVIDIA A40 GPU node from scratch for 18k training steps with 9

Metrics. We use the following metrics in line with prior literature: ASR (attack success rate), AUC (Area Under ROC Curve), TPR@1% FPR (fraction of true members correctly identified as members where at most 1% of non-members are misclassified), which was introduced in [5]. We also note the number of model queries each attack uses in our tables, especially pertinent to SimA and our own attacks which improve with more model queries.

10k/10k member/heldout splits. • CelebA: We trained a model from scratch on a single NVIDIA A40 GPU node for 60k training steps with 30k/30k member/heldout splits. • ImageNet-1k: Similarly to prior literature [34], we reuse public Guided Diffusion model [12] checkpoints trained on ImageNet-1k, and use the validation set ImageNetV2 [35] as the held-out set.

Model architectures. For CIFAR-10, CIFAR-100, STL10U, and CelebA, we use the standard DDPM U-Net architecture of [20] (base channels 128, channel multipliers [1, 2, 2, 2], 2 residual blocks per resolution, self-attention at 16×16, GroupNorm normalization and Swish activations throughout), the same architecture used by SecMI [13], PIA [27], and SimA [34] in their respective evaluations. This model is unconditional: it predicts noise given only a noised image and timestep, at 32×32 resolution (d ≈ 3,072). For ImageNet-1k, we use Guided Diffusion [12], a substantially larger, publicly released checkpoint operating at 256×256 resolution (d ≈ 196,608, a 64× increase in input dimensionality). Unlike our DDPM checkpoints, Guided Diffusion is class-conditional: the class label is injected as an additional conditioning signal alongside the timestep embedding, and must be supplied at every query. Both architectures instantiate the same underlying forward diffusion process and noise-prediction training objective.

Baselines. We use prior membership inference attack baselines Loss [31], PIA [27], SecMI [13], SimA / SimA-MC [34]. These attacks compute test statistics based on quantities in the diffusion process such as diffusion trajectories, estimated score norm, and reconstruction error, in comparison to our idealized denoiser-based attack. We define their test statistics below. All tests are based on tunable hyperparameter thresholds for determining membership. These baselines are chosen in particular for their SoTA performance in diffusion training membership inference, across varying query budgets [34]. In each of the methods below, ε is independently sampled standard Gaussian noise used to compute the test statistic. The Loss criterion proposed in [31] uses the training loss function, or: p TLoss (x,t) = ∥ε − ε̂θ ( αt x + σt ε,t))∥ SecMI of [13] takes instead the reconstruction error of a noising and denoising step, taking the following statistic: p p TSec (x,t) = ∥ 1 − αt (ε̂θ (x,t)− ε̂θ ( αt+1 x+σt+1 ε,t +1))∥

5.2

PIA introduced by [27] uses the ground-truth diffusion trajectory to determine membership, using: p TPIA (x,t) = ∥ε̂θ (x, 0) − ε̂θ ( αt x + σt ε̂θ (x, 0),t)∥

Results and Discussion

DDPM. Table 1 reports DIME against Loss, PIA, SecMIstat , SimA, and SimA-MC across four DDPM checkpoints (CIFAR-10, CIFAR-100, CelebA, STL10-U). We compare at matched, or near-matched, query budgets. At every dataset and every matched tier, DIME outperforms the strongest baseline on all evaluation metrics simultaneously. The gap is most pronounced on TPR@1%FPR, the metric prior work [5] has argued is most operationally meaningful for membership inference: at 11 queries, DIME achieves 47.23% TPR@1%FPR on CIFAR-10, compared to SimA-MC’s best result of 16.14% at 30 queries, displaying roughly 3× the detection rate at a third of the query cost. The same pattern holds on CelebA (14.08% at 11 queries vs. 12.91% at the highest tier for SimA) and is most dramatic on STL10-U, where DIME reaches 97.44% TPR@1%FPR at 11 queries against SimAMC’s best 80.92%, which is near-total detection. On CIFAR100, DIME improves TPR@1%FPR from SimA-MC’s 19.03% to 26.93%; we note a modest, isolated dip in TPR@1%FPR at the highest query tier on this dataset not mirrored in AUC or ASR (both of which continue to improve), consistent with the greater sensitivity of tail-probability estimates to sampling noise at this scale, and report the 11-query tier as CIFAR100’s representative operating point.

Finally, SimA [34] uses the norm of the predicted score as the decision criterion, that is:

TSimA (x,t) = ∥ε̂θ (x,t)∥ p and we also evaluate a more stable Monte-Carlo variant p 1 TSimA (x,t) = ∑ ∥ε̂θ (xt ,t)∥, xt = αt x + σt εn , N n where εn ∼ N (0, I). We re-implement all of the methods for experimental consistency, drawing mainly from the authors’ provided codebases. Evaluation protocol. Member/held-out sets are split into disjoint calibration and test halves. All test hyperparameters (including timestep selection) are selected by maximizing the AUC on the calibration half only; all reported metrics are computed once on the held-out test half, which is a stricter protocol than in prior work [34]. 10

Table 1: Performance of MIA attacks on DDPM across four benchmark datasets. DIME outperforms all other methods in all metrics at comparable query costs, with the biggest gains in TPR@1%FPR. CIFAR-10 (%) Method

CIFAR-100 (%)

STL10-U (%)

CelebA (%)

#Query↓

ASR↑

AUC↑

TPR@1%FPR↑

ASR↑

AUC↑

TPR@1%FPR↑

ASR↑

AUC↑

TPR@1%FPR↑

ASR↑

AUC↑

TPR@1%FPR↑

Loss PIA SecMIstat SimA (ℓ4 ) SimA-MC (ℓ4 , #mc=10) SimA-MC (ℓ4 , #mc=30)

1 2 ∼12 1 10 30

77.30 81.67 81.45 83.60 84.26 85.15

84.38 88.58 88.51 90.43 91.34 92.00

6.59 14.20 10.45 14.07 16.95 16.14

75.31 79.14 80.85 82.40 83.22 83.74

82.51 86.34 88.05 89.40 90.55 90.99

9.44 15.26 13.34 13.64 19.03 18.87

93.85 95.28 93.98 95.42 97.25 96.74

97.95 98.77 98.20 98.61 99.31 99.23

52.14 70.54 57.76 62.68 80.92 79.84

67.32 72.63 71.08 71.09 74.33 75.03

73.50 80.06 78.03 78.21 82.05 82.96

5.67 9.75 8.64 8.91 9.91 12.91

DIME (Ours) DIME (Ours, mc=10), DIME (Ours, mc=30),

1+1 10+1 30+1

84.18 86.43 87.30

91.50 93.35 93.89

30.88 47.23 50.77

83.00 85.01 85.20

90.02 91.75 91.92

22.79 26.93 24.83

96.87 98.65 98.80

99.39 99.78 99.80

86.16 97.78 98.00

73.54 76.67 78.14

81.02 84.55 86.31

11.63 14.08 18.78

Table 2: Performance on ImageNet-1k / Guided Diffusion (256x256, class-conditional). DIME outperforms all other methods in all metrics at comparable query costs, with the biggest gains in TPR@1%FPR. Method

#Query↓

ASR↑

AUC↑

TPR@1%FPR↑

Loss PIA SecMIstat SimA (ℓ4 ) SimA-MC (ℓ4 , mc=10) SimA-MC (ℓ4 , mc=30)

1 2 ∼12 1 10 30

74.62 67.56 66.67 83.84 84.13 82.58

80.64 69.88 50.59 86.35 89.41 87.45

1.60 1.07 0.93 7.20 9.33 1.87

DIME (Ours) DIME (Ours, mc=10) DIME (Ours, mc=30)

1+1 10+1 30+1

86.45 86.96 87.33

92.50 92.58 92.65

14.27 15.27 15.27

all three metrics outright: 86.96% ASR vs. 84.13%, 92.58% AUC vs. 89.41%, and 15.27% TPR@1%FPR vs. 9.33%. We note two additional observations specific to this setting. First, DIME’s TPR@1%FPR plateaus between 11 and 31 queries (15.27% at both), while ASR and AUC continue to improve marginally (86.96%→87.33%, 92.58%→92.65%), which is unlike the clean, monotonic gains from additional queries observed on our DDPM checkpoints, suggesting the marginal value of additional averaging diminishes faster at this scale, plausibly reflecting the smaller evaluation set available at ImageNet’s higher per-query cost. Second, SimA-MC itself is non-monotonic in query count here: its TPR@1%FPR at mc=30 (1.87%) is substantially lower than at mc=10 (9.33%), indicating this instability is not specific to our method but reflects a broader difficulty in reliably estimating tail-probability statistics at this resolution. Taken together, these results indicate that the theoretical grounding motivating DIME, derived from the exact optimaldenoiser solution for a finite training set (Lemma 1), yields consistent practical improvements across a substantial architectural and scale gap, not only on the smaller checkpoints used in our primary evaluation. Illustratively, Figure 2 shows the clear statistical separation between member and held-out data points in the DIME statistic across all datasets and trained checkpoints. Although we report the best calibrated timestep t in our main tables, we plot the robustness of DIME to varying timesteps in Figure 3. Lemma 2 prescribes a stronger membership signal for earlier timesteps in the idealized denoiser, which is reflected strongly in the Guided Diffusion ImageNet experiment, but not in the DDPM experiments on the other datasets. We explain this discrepancy simply. While our DDPM model checkpoints use ∼ 35M parameters, the Guided Diffusion checkpoint substantially expands with ∼ 550M parameters, suggesting that the heavier architecture with its additional expressivity conforms better to our theoretical object in the idealized denoiser. Our experiments demonstrate that our optimal denoiserbased membership inference attacks outperform previously considered methods across a variety of dataset fidelities, model architectures, and evaluation metrics.

These improvements hold consistently whether comparing against SimA-MC (and other methods) at matched query count or, more conservatively, against its best result at any query count in our evaluation; we observe that DIME’s cheapest tier alone (2 queries) already exceeds every baseline’s TPR@1%FPR on CIFAR-10 and CIFAR-100. Guided-Diffusion. To test whether these results generalize beyond the 32×32 DDPM checkpoints above, we evaluate DIME against Guided Diffusion [12], a substantially larger, 256×256, class-conditional model, using a stratified sample of ImageNet-1k (member) and ImageNetV2 [35] (held-out). This setting differs from our DDPM checkpoints not only in resolution and parameter count, but in architecture (classconditioning) and input dimensionality (d ≈ 196,608, a 64× increase over CIFAR-scale inputs), which is a meaningfully more demanding test of whether our theoretical construction transfers beyond the setting it was developed in. Table 2 reports the resulting comparison. DIME outperforms every baseline on ASR, AUC, and TPR@1%FPR simultaneously, at every query budget we evaluate. Even at its cheapest setting (2 queries), DIME’s TPR@1%FPR (14.27%) exceeds every baseline’s best result at any query cost, including SimA-MC’s best 10-query setting (9.33%), showing a 1.5× improvement at roughly 1/5th the budget. At a directly comparable cost (11 queries vs. SimA-MC’s 10), DIME wins 11

Figure 2: DIME test statistic distributions for samples from member and held-out sets. The vertical dashed line denotes the selected threshold τ for each figure.

Figure 3: Robustness of the DIME test to timestep across five datasets. DDPM models remain stable across all time-steps, while the Guided Diffusion model is most vulnerable to membership inference at early timesteps.

6

Defenses

adversary under a differentially private training algorithm. Lemma 3. For fixed FPR α, if training mechanism M satisfies (ε, δ)-DP, then the TPR is bounded above by eε α + δ.

Differential privacy (DP) [15, 16] is the standard formal defense against membership inference: Differentially Private Stochastic Gradient Descent (DP-SGD) [2] is a standard machine learning algorithm that implements differential privacy. It bounds the influence that any single training example can have on the final model, giving a provable guarantee independent of the specific attack used against it. Prior work evaluating diffusion-model memorization under DP-SGD finds that existing attacks are rendered ineffective at standard privacy budgets on small-scale fine-tuning datasets [32]. We evaluate whether this holds against DIME as well: a defense that only defeats weaker, prior attacks is a substantially less interesting result than one that also defeats a stronger, theoreticallymotivated one.

Proof. Denoting arbitrary z := Dtrain and z′ := Dtrain ∪ {x} adjacent, we can rewrite FPR = P[b̂ = 1|H0 ] = P[A (M (z)) = 1], TPR = P[b̂ = 1|H1 ] = P[A (M (z′ )) = 1], giving us that TPR ≤ eε FPR + δ. □ Remark 1. The TPR@1%FPR metric is bounded above by 0.01eε + δ.

6.2

Experimental setup

We train a CelebA checkpoint, separately to our main results, with DP-SGD (via Opacus [49]) at three privacy budgets, ε ∈ {1, 4, 10} and δ = 10−5 , alongside a non-private (‘Vanilla’) baseline trained identically but without DP-SGD. All four checkpoints share the same architecture, training recipe, and 500-epoch budget, differing only in the presence and strength of DP-SGD, with n=1000 member images. We use an identity-aware member/held-out split: no identity’s photos appear on both sides, preventing the defense evaluation from being confounded by identity-level correlation that DP-SGD’s per-example guarantee does not address.

6.1 Differential Privacy: Definition and Implications One of the most ubiquitous definitions of privacy due to its rigorous guarantees, differential privacy, referred to as DP [14, 15], quantifies the worst-case privacy risk admitted by randomized algorithms. Definition 3 (Differential Privacy). A (randomized) mechanism M satisfies (ε, δ)-differential privacy if for all neighboring datasets z, z′ , i.e. differing in at most one row, it holds that for all events S, P[M (z) ∈ S] ≤ eε P[M (z′ ) ∈ S] + δ.

6.3

Results and Discussion

Table 3 reports ASR, AUC, and TPR@1%FPR for all five methods against all four checkpoints. The Vanilla checkpoint shows substantial, consistent memorization across every method: AUC ranges from 77.7% (SecMIstat ) to 86.9%

To illustrate the practical privacy effects of such a theoretical guarantee, we present a simple utility bound for any 12

Table 3: Attack accuracy under DP-SGD defense (CelebA, n=1000, identity-aware split, 500 epochs). All five attack methods’ accuracy collapses under DP-SGD, regardless of ε. ‘Vanilla’ means without DP-SGD. DIME’s #Query is qref +1 (11 at qref =10, the reported tier, roughly comparable to SimA-MC’s 10). The highest accuracy in each column is bolded. Loss

ε=1 ε=4 ε = 10 Vanilla

PIA

SecMIstat

DIME (Ours)

AUC↑

T@1%F↑

ASR↑

AUC↑

T@1%F↑

ASR↑

AUC↑

T@1%F↑

ASR↑

AUC↑

T@1%F↑

ASR↑

AUC↑

T@1%F↑

0.530 0.539 0.519 0.690

0.521 0.518 0.509 0.754

1.00 1.40 1.40 10.20

0.526 0.506 0.506 0.753

0.516 0.481 0.483 0.829

1.60 0.60 0.80 15.40

0.517 0.522 0.523 0.710

0.500 0.488 0.519 0.777

1.80 1.20 1.00 13.60

0.517 0.515 0.510 0.745

0.505 0.502 0.489 0.827

0.40 0.20 1.00 16.20

0.529 0.525 0.537 0.798

0.517 0.515 0.529 0.869

0.40 1.20 0.80 22.40

(DIME), with DIME achieving the strongest result on every metric (79.8% ASR, 86.9% AUC, 22.4%TPR@1%FPR), which confirms this checkpoint is a genuine, non-trivial target before any defense is applied. Under DP-SGD, every method collapses to near-chance performance at every tested ε, including DIME. AUC falls into a narrow 48.1-52.9% band across all fifteen (method, ε) combinations, and TPR@1%FPR drops from Vanilla’s 10.222.4% range to at most 1.8% under any DP-SGD setting. Notably, we do not observe a consistent monotonic trend across ε = 1, 4, 10 for any method; performance at ε = 1 is not reliably lower than at ε = 10, which is explained with all three privacy budgets already suppressing memorization signal below what our evaluation scale can reliably distinguish, rather than ε producing a graded effect in this regime. These results indicate that DP-SGD remains an effective, complete defense against membership inference on diffusion models even against an attack with substantially higher power in the non-private setting: DIME’s 8-15 percentage point AUC advantage over the strongest baseline on the Vanilla setting (Table 1) does not survive contact with DP-SGD at any tested privacy budget.

7

SimA (ℓ4 )

ASR↑

That structure decomposes, exactly, into two complementary, independently estimable sources of membership signal: how far a candidate’s model-implied reconstruction deviates from the candidate itself, and how densely the training points responsible for that reconstruction are crowded around one another. The first channel is what prior diffusion membership inference work measures; the second has no analog in that literature, materializing from our theoretical analysis of the idealized denoiser. We showed both channels admit queryefficient estimators computable from forward evaluations of the trained network alone, sharing the same perturbed queries and requiring no gradient access, shadow models, or generative sampling. We complete the story by developing DIME, an efficient and motivated membership error derived attack. Across five diffusion checkpoints spanning a large range in input dimensionality and containing both unconditional and class-conditional architectures, DIME outperforms every baseline attack we compare against on ASR, AUC, and TPR@1%FPR simultaneously, at matched or substantially lower query cost. We observe cases where our cheapest setting exceeds existing baselines’ own most query expensive setting. We additionally evaluated DIME against DP-SGD, the standard formal defense against privacy adversaries, and found that our attack, like the tested prior baselines, is reduced to near-chance performance at every tested privacy budget. This is itself an informative result: DP-SGD’s guarantee holds even against an attack with substantially higher power in the undefended setting.

Conclusion

We derived an exact, closed-form characterization of the idealized denoiser; the MSE-optimal noise-prediction function a diffusion model would compute at any query point, given a finite training set, showing it takes the form of a responsibilityweighted average over training examples, with weights determined by a softmax over distance to the query. This posterior responsibility structure is the exact mechanism by which a finite training set leaves a detectable trace in the model’s output. When responsibility concentrates on a single training point, the model’s implicit reconstruction reveals that point’s identity with near-certainty; when it is divided among several, the reconstruction instead reflects an aggregate that need not correspond to any single training example. Framing membership inference through this lens replaces an empirically-motivated search for informative statistics with a question that has a derivable answer: what does the training set’s structure make observable at a given query.

Limitations and future work. Our theoretical characterization concerns the idealized denoiser; a trained neural network approximates this object rather than realizing it exactly, and while our empirical results suggest this approximation is close enough to be exploitable, formalizing the gap remains an open problem. Furthermore, the bias and crowding decomposition has a natural coding-theoretic interpretation: the training set acts as a Euclidean codebook and the diffusion denoiser as a soft decoder. The bias term measures displacement from the decoded codeword, while the crowding term measures ambiguity among nearby plausible codewords. Exploring this connection, and its relation to privacy, is important future work [3].

13

References

[10] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215–223. JMLR Workshop and Conference Proceedings, 2011.

[1] 104th United States Congress. Health insurance portability and accountability act of 1996. Pub. L. No. 104-191, 110 Stat. 1936, 1996. URL: https://www.govinfo.gov/content/pkg/ PLAW-104publ191/pdf/PLAW-104publ191.pdf.

[11] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989.

[2] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016.

[12] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS ’21, Red Hook, NY, USA, 2021. Curran Associates Inc.

[3] Daniel Alabi. The existence of error-correcting codes implies privacy lower bounds. IEEE BITS the Information Theory Magazine, pages 1–12, 2026. doi: 10.1109/MBITS.2026.3683802.

[13] Jinhao Duan, Fei Kong, Shiqi Wang, Xiaoshuang Shi, and Kaidi Xu. Are diffusion models vulnerable to membership inference attacks? In International Conference on Machine Learning, pages 8717–8730. PMLR, 2023.

[4] California State Legislature. California consumer privacy act of 2018, as amended by the california privacy rights act of 2020. Cal. Civ. Code SS1798.100 et seq., 2018. URL: https://leginfo.legislature.ca. gov/faces/codes_displayText.xhtml?lawCode= CIV&division=3.&title=1.81.5.&part=4.

[14] Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pages 1–12. Springer, 2006. [15] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, pages 486–503, 2006.

[5] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pages 1897–1914. IEEE, 2022.

[16] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to sensitivity in private data analysis. In TCC, 2006.

[6] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253–5270, 2023.

[17] European Parliament and Council of the European Union. Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (general data protection regulation). Official Journal of the European Union, L119, pp. 1– 88, 2016. URL: https://eur-lex.europa.eu/eli/ reg/2016/679/oj.

[7] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650, 2021.

[18] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pages 1322–1333, 2015.

[8] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 343–362, 2020.

[19] Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart. Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing. In 23rd USENIX Security Symposium, pages 17–32, 2014.

[9] Sheng-Yen Chou, Pin-Yu Chen, and Tsung-Yi Ho. How to backdoor diffusion models? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4015–4024, 2023. 14

[20] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.

In 2023 IEEE Security and Privacy Workshops (SPW), pages 77–83. IEEE, 2023. [32] Yan Pang and Tianhao Wang. Black-box membership inference attacks against fine-tuned diffusion models. arXiv preprint arXiv:2312.08207, 2023.

[21] Nils Homer, Szabolcs Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V Pearson, Dietrich A Stephan, Stanley F Nelson, and David W Craig. Resolving individuals contributing trace amounts of dna to highly complex mixtures using highdensity snp genotyping microarrays. PLoS genetics, 4(8):e1000167, 2008.

[33] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.

[22] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.

[34] Mingxing Rao, Bowen Qu, and Daniel Moyer. Scorebased membership inference on diffusion models. Transactions on Machine Learning Research, 2026. URL: https://openreview.net/forum?id=Ckvsu5xRmf.

[23] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural networks, 3(5):551–560, 1990.

[35] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389–5400. PMLR, 2019.

[24] Yuke Hu, Zheng Li, Zhihao Liu, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen. Membership inference attacks against {Vision-Language} models. In 34th USENIX security symposium (USENIX Security 25), pages 1589– 1608, 2025.

[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.

[25] Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18(3):1059–1076, 1989.

[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211–252, 2015.

[26] Robert W Keener. Theoretical statistics: Topics for a core course. Springer Science & Business Media, 2010. [27] Fei Kong, Jinhao Duan, Heng Tao Shen, Xiaoshuang Shi, Xiaofeng Zhu, Kaidi Xu, et al. An efficient membership inference attack for the diffusion model by proximal initialization. In International Conference on Learning Representations, volume 2024, pages 57876–57896, 2024.

[38] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 5558–5567, 2019.

[28] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

[39] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479–36494, 2022.

[29] Zhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, and Volkan Cevher. Membership inference attacks against large vision-language models. Advances in Neural Information Processing Systems, 37:98645–98674, 2024. [30] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015.

[40] Shawn Shan, Wenxin Ding, Josephine Passananti, Stanley Wu, Haitao Zheng, and Ben Y. Zhao. Nightshade: Prompt-specific poisoning attacks on text-to-image generative models. In IEEE Symposium on Security and Privacy (SP), pages 807–825, 2024.

[31] Tomoya Matsumoto, Takayuki Miura, and Naoto Yanai. Membership inference attacks against diffusion models. 15

[41] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In International Conference on Learning Representations, volume 2024, pages 51826–51843, 2024. [42] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017. [43] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6048– 6058, 2023. [44] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. [45] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Scorebased generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. [46] Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing Systems, volume 29, pages 4790–4798, 2016. [47] Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on Machine Learning, pages 1747–1756, 2016. [48] James Vincent. Ai art tools stable diffusion and midjourney targeted with copyright lawsuit. The Verge, 16:2023, 2023. [49] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, et al. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint arXiv:2109.12298, 2021. [50] Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 253–261, 2020.

16

Ethical Considerations

So outside of the zero-measure set {g : ∃ i ̸= k s.t. di (g) = dk (g)},

We describe a potential privacy adversary that could be deployed by malicious actors, although the attack can also be developed benevolently for privacy auditing. We believe the immediate risk of our own work is outweighed by the benefits from increasing public understanding of diffusion model behavior and privacy vulnerabilities. We also present a feasible and effective defense against our attack to counteract potential harms incurred.

( j)

it holds that ri (g) → 1Vi . It remains to show the aforementioned identity. For notational convenience, let σ = σ( j) , let√ ting a := 1 − σ2 and si (g) = exp(− 2σ1 2 ∥g−axi ∥2 ). We have   sk (g) 1 1 2 2 = exp − 2 ∥g − axk ∥ + 2 ∥g − axi ∥ si (g) 2σ 2σ   a2 a = exp 2 ⟨g, xk − xi ⟩ − 2 (∥xk ∥2 − ∥xi ∥2 ) σ 2σ

Open Science All code and instructions for recreating all results and figures can be found at the anonymous repository linked: https: //anonymous.4open.science/r/dime-mia-E1C5/.

A

= exp

Additional Theoretical Proofs and Details

ah 1 ⟨g, xk − xi ⟩ − (∥xk ∥2 − ∥xi ∥2 ) 2 σ 2 i 1−a 2 + (∥xk ∥ − ∥xi ∥2 ) 2



 a a(1 − a) 2 2 2 2 = exp (di (g) − dk (g) ) + (∥xk ∥ − ∥xi ∥ ) . 2σ2 2σ2

Lemma 4. Consider a sequence of functions ( j)

( j) si (g)

We can write

si (g)

( j)

ri (g) =

( j)

,

∑k sk (g)

a = σ2

  = exp − 2(σ1( j) )2 ∥g − a( j) xi ∥2 , a( j) =

where p 1 − σ( j)2 , and σ( j) → 0+ . Then

a(1 − a) a(1 − a) a 1 = = → as σ → 0+ . 2σ2 2(1 − a2 ) 2(1 + a) 4

( j)

So within the exponential, we have one term that blows up and one term that becomes irrelevant asymptotically. Subsequently, the convergence behavior depends on the sign of di (g)2 − dk (g)2 , i.e. whether g is closer to xi or xk , and we conclude as desired that ( 0 : di (g) < dk (g) sk (g) −→ as σ → 0+ . si (g) ∞ : di (g) > dk (g)

Proof. We have that ( j)

si (g)

( j)

( j) ∑k sk (g)

1

=

( j)

s

1 + ∑k̸=i k( j)

. (g)

si (g)

We want to show that ( j)

( 0: −→ ( j) ∞: s (g) sk (g) i

di (g) < dk (g) . di (g) > dk (g)

□ Corollary 1 (Ideal denoiser as a scaled bias vector). For any query point g and timestep t, we can write

Suppose the above identity holds, then considering any g such that di (g) ̸= dk (g) for any i ̸= k, then letting i∗ be the unique nearest index, i.e. di∗ (g) := mini di (g), we have two cases:

ε̂∗ (g,t) =

• If i = i∗ , then every k ̸= i has di (g) < dk (g), so 1

( j)

ri (g) =

s

( j)

1 + ∑k̸=i k( j)

→ (g)

• If i ̸= i∗ , then di∗ (g) < di (g), so (observing all ratios are positive) 1

( j)

( j) sk (g) k̸=i ( j) si (g)

1+∑

≤

1 s

( j)

1 + (i∗j)

1 (g − E[Y ]), σt

(9)

√ where Y = αt I ∼ r(g) is the (scaled) training point random variable denoted in (4). √ Proof. By definition, E[Y ] = αt ∑i ri (g)xi . Substituting into Lemma 1, √
g αt g E[Y ] 1 ∗ ε̂ (g,t) = − ri (g)xi = − = g − E[Y ] . ∑ σt σt i σt σt σt

1 = 1. 1+0

si (g)

ri (g) =

as σ → 0+ .

On the other hand, it holds that

ri (g) −→ 1Vi (g) a.e.

ri (g) =

√ 1 − σ2 →∞ σ2

→ 0. (g)

□

si (g)

17

Lemma 5 (Error of the squared-bias estimator). Fix a timestep t and a query point g, and write ⋆

By convexity of the square, 2

B̂ =

B := ∥ f (g)∥2 .

f (x) := ε̂ (x,t), i.i.d.

Let z1 , . . . , zq ∼ Rad({−1, +1}d ), and define B̂ :=

Taking expectations and expanding the squared norm,

1 q ∑ ∥ f (g + γσt zk )∥2 . q k=1

E[B̂2 ] ≤ E[X(z)2 ] = E∥a + ∆(z)∥22 = ∥a∥22 + 2⟨a, E[∆(z)]⟩ + E∥∆(z)∥22 .

Suppose that, throughout the perturbation neighborhood n o Nγ (g) := g + sγσt z : s ∈ [0, 1], z ∈ {−1, +1}d ,

Using Equations 10, 11, and ∥a∥2 = B,
E[B̂2 ] ≤ B2 + BHt + Lt2 γ2 σt2 d.

there exist constants Lt , Ht < ∞ such that

and

Moreover,

Ht ∥h∥22 , 2

J := ∇g f (g). E[B̂] = Ez ∥a + ∆(z)∥2 ≥ ∥a + E[∆(z)]∥2 .

Then
−BHt γ2 σt2 d ≤ E[B̂2 ] − B2 ≤ BHt + Lt2 γ2 σt2 d.

It follows that E[B̂2 ] ≥ ∥a + E[∆(z)]∥22

Consequently,

=B

E[B̂2 ] − B2 ≤ BHt + Lt2 γ2 σt2 d,

≥B

E[B̂2 ] = B2 + O(γ2 ).

∆(z) := f (g + h) − f (g).

∥h∥22 = γ2 σt2 d.

By the assumed second-order expansion, ∆(z) = Jh + R(h),

∥R(h)∥2 ≤

Ht ∥h∥22 . 2

Ht 2 2 γ σt d. 2

(10)

The Lipschitz assumption also gives √ ∥∆(z)∥2 ≤ Lt γσt d.

− BHt γ2 σt2 d.

(14) (15) (16) □

Lemma 6 (Error order of crowding√estimator V̂ ). Fix a query point x∗ , timestep t, and let g = ᾱt x∗ . Let I ∼ r(g), Y = √ ᾱt xI ∈ Rd , and V = tr(Cov(Y )). For step size γ > 0 and i.i.d. Rademacher vectors z1 , . . . , zq ∈ {−1, +1}d , define
∗ ∗ 1 q z⊤ k ε̂ (g + γσt zk , t) − ε̂ (g,t) ∗ V̂ (x ,t, γ; q) := ∑ . (17) q k=1 γσt

Therefore, ∥E[∆(z)]∥2 = ∥E[R(h)]∥2 ≤

2

(13)

Remark 2 (Why Lemma 5 concerns B̂2 ). The unsquared estimator B̂ does not, in general, satisfy E[B̂] = B + O(γ2 ). The norm is nondifferentiable at points where ε̂⋆ (g,t) = 0, and at such points the bias can be Θ(γ). For example, in one dimension, if f (u) = u/σt and g = 0, then B = 0 while B̂ = γ for every Rademacher perturbation. Nevertheless, B̂2 = γ2 , which is consistent with Lemma 5. This is also the relevant quantity because the exact reconstruction error decomposition contains B2 , not B.

Because z is Rademacher, E[h] = 0,

+ 2⟨a, E[∆(z)]⟩ + ∥E[∆(z)]∥22

Combining (12) and (16) proves the result.

Proof. Let h := γσt z,

2

≥ B2 − 2B∥E[∆(z)]∥2

and hence, for fixed t, g, and d,

a := f (g),

(12)

For the opposite direction, Jensen’s inequality gives
2 E[B̂2 ] ≥ E[B̂] .

∥ f (g + h) − f (g)∥2 ≤ Lt ∥h∥2

∥ f (g + h) − f (g) − Jh∥2 ≤

!2 1 q 1 q X(zk ) ≤ ∑ X(zk )2 . ∑ q k=1 q k=1

(11)

Then E[V̂ ] =

For one perturbation, define X(z) := ∥a + ∆(z)∥2 .

d 1 − V + O(γ), σt σt3

where d is the input dimension. 18

(18)

and for fixed zk , σt it holds that

Proof. Let m(g) := ∑ ri (g)xi ,

√ li (g) := −∥g − ᾱt xi ∥2 /(2σt2 ),

O(∥h∥2 ) = O(γ2 σt2 ∥zk ∥2 ) = O(γ2 ),

i

so we have that

so ri (g) = exp(li (g))/ ∑ j exp(l j (g)). Differentiating, ∇g li (g) = −(g −

p

∗ ∗ z⊤ k ε̂ (g + γσt zk ,t) − ε̂ (g,t) = z⊤ k Jzk + O(γ). γσt

αt xi )/σt2 ,

and the standard softmax-Jacobian identity gives
√ αt xi − m(g) ∇g ri (g) = ri (g) · . σt2

Averaging over q draws and taking expectation, E[V̂ ] = tr(J) + O(γ).

This O(γ) term is specific to our finite-step approximation of the exact matrix-vector product assumed by the standard estimator, and is the only respect in which V̂ deviates from it. Finally, substituting Equation 20 into Equation 21 gives Equation 18. □

Using ∇g m(g) = ∑ xi ∇g ri (g)⊤ i

and

∑ ri (g) · (xi − m(g))(xi − m(g))⊤ = Covi∼r(g) (xi ),

B

i

it holds that (using m := m(g) and ri := ri (g) as shorthand) # √ " αt ∇g m(g) = 2 ∑ ri (xi − m)(xi − m)⊤ + m ∑ ri (xi − m)⊤ , σt i i

!⊤

∑ ri (xi − m) = ∑ ri xi i

i

Synthetic Visualizations

This appendix supplements the main body with experiments conducted in a fully synthetic, idealized setting: rather than a trained neural network approximating ε̂∗ , we use the exact closed-form idealized denoiser (Lemma 1) directly, computed exactly from a small, explicitly constructed training set. This removes any confound from network approximation error, isolating whether the theoretical constructions and empirical trends described in the main body hold true of the underlying ideal object itself. These experiments are illustrative and diagnostic rather than a substitute for the real-network results in Section 5, and are not intended to be read as a continuous narrative with one another.

and ⊤

(21)

− m⊤ ∑ ri = m⊤ − m⊤ = 0, i

so it holds that √ αt ∇g m(g) = 2 Covi∼r(g) (xi ). σt

B.1

Since p C = Cov( αt xI ) = αt Covi∼r(g) (xi ),

Validating the bias-variance decomposition

Figure 4 directly validates the exact decomposition M = σt2 B2 + V (Equation 5) against the three-case construction motivating our approach (Section 4): a genuinely isolated member, an isolated non-member whose nearest training point lies elsewhere (biased), and a non-member embedded in a dense cluster of training points whose responsibility-weighted centroid happens to coincide with the candidate (crowded). The bias2 panel confirms the intended construction: the isolated member’s bias remains near zero across nearly the entire displayed range, while both non-member constructions show substantial bias, with the crowded case’s bias notably lower than the biased case’s by design. The variance panel shows the key, motivating asymmetry: the crowded non-member’s variance rises far earlier (before t ≈ 50) than either the member’s or the biased non-member’s, both of which remain near zero until much later in the range. This is precisely the signature predicted in Section 4: a candidate whose bias alone would appear small can still be correctly flagged once variance is

then we have p ∇g m(g) = C/( αt σt2 ). Writing J := ∇g ε̂∗ (g,t), where ε̂∗ (g,t) is defined as in Lemma 1, we have √ 1 αt C 1 1 = Id − 3 C, (19) J = Id − ·√ 2 σt σt σt αt σt σt d 1 tr(J) = − 3 tr(C). (20) σt σt By the Hutchinson trace estimator [25], Ez [z⊤ Jz] = tr(J) for any fixed matrix J and Rademacher draws z. We do not have access to Jz directly, only to forward evaluations of ε̂∗ ; by Taylor’s theorem, for h = γσt zk ,
∗ ∗ z⊤ O(∥h∥2 ) k ε̂ (g + γσt zk ,t) − ε̂ (g,t) = z⊤ , k Jzk + γσt γσt 19

Figure 4: Membership signal of Bias and Variance term from the decomposition in Equation 5 from idealized denoiser over simulated data. The variance term is more effective at capturing membership signal of held-out data in a dense region of member data. taken into account, and the earliest, clearest such signal in this construction comes specifically from the crowded case. We note the displayed range is deliberately restricted to where this asymmetry is visible; at sufficiently large t, even the isolated member’s estimation error rises as noise begins to overwhelm any fixed geometric separation, consistent with Lemma 4’s asymptotic result being specifically a small-t statement.

B.2

exact closed-form Jacobian trace rather than the finite-sample Hutchinson estimator used in our real-network experiments (Section 4); this synthetic comparison therefore reflects the idealized, zero-estimation-noise crowding signal, not the practical q + 1-query estimator’s behavior.

B.4

DIME versus SimA-MC across dataset size, at fixed query budget

Figure 7 visualizes ε̂∗ (x,t) directly on simple, onedimensional synthetic training sets of size n = 2, 3, 10, at several noise levels, with vertical dashed lines marking training point locations. Two properties predicted by Corollary 1 are directly visible: first, at low noise (dark curves), ε̂∗ crosses zero exactly at each training point and grows as the query moves away from it, consistent with ε̂∗ being proportional to the bias vector (x − m(x)); second, as noise increases (lighter curves), this structure is progressively smoothed away, consistent with responsibility spreading across the training set rather than concentrating on individual points. The n = 10 panel additionally illustrates richer, multi-modal behavior near the two deliberately-constructed clusters (points near x = −3 and x = 3), foreshadowing the crowding phenomenon central to Section 4.

Figure 5 compares DIME against SimA-MC in the idealized setting across three synthetic dataset sizes (n = 10, 100, 1000), at a fixed query budget, with query-averaged attack effectiveness plotted against the noise level αt used at query time. DIME outperforms SimA-MC across every metric (ASR, AUC, TPR@1%FPR), every dataset size, and nearly every noise level, with the gap most pronounced at higher noise levels, where SimA-MC’s performance collapses toward chance while DIME retains substantially more signal. This advantage persists as n grows across two orders of magnitude, suggesting DIME’s improvement over a level-only statistic is not an artifact of the small, low-dimensional training sets used in this synthetic construction.

B.3

Illustrating the idealized denoiser in one dimension

DIME versus SimA-MC across query budget, at fixed dataset size

Figure 6 complements the above by fixing dataset size and instead varying the query budget (q = 1, 10, 30). The same qualitative pattern holds: DIME’s advantage over SimA-MC is consistent across all three query budgets, indicating the gap is not primarily an artifact of how many queries either method is given, but reflects a persistent difference in the two statistics themselves. We note that both methods’ bias components share the same q perturbed queries in this construction, and DIME’s crowding term is computed via the 20

Figure 5: DIME vs. SimA-MC for the idealized denoiser on simulated data with q = 10, over various dataset sizes. The additional crowding signal captured in DIME improves membership signal decay with more diffusion noise.

21

Figure 6: DIME vs. SimA-MC for the idealized denoiser on simulated data with n = 100, over various query budgets. The additional crowding signal captured in DIME improves membership signal decay with more diffusion noise.

Figure 7: The idealized denoiser on synthetic 1D data. ε̂∗ (x,t), computed exactly from the closed-form optimal denoiser, on synthetic training sets of size n = 2, 3, 10 (vertical dashed lines mark training points), at four noise levels αt .

22

Related documents

• JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models#630080
• JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models#628218
• FERMI: Exploiting Relations for Membership Inference Against Tabular Diffusion Models#178813
• Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetry#627622
• Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles#267652