Picture the Epsilon: Pursuing Identity-Level Privacy Guarantees for Images

Picture the Epsilon: Pursuing Identity-Level Privacy Guarantees for Images Arman Zareian Jahromi∗ , Vishnu Bondalakunta∗ , Mohammad Akbar Bin Shah∗ , Naimul Haque† , Shuangqing Wei† , George T. Amariucai∗ ∗ Kansas State University, Manhattan, KS, USA

{zareian, vishnub, mohammadakbar, amariucai}@ksu.edu {naimul.haque, swei}@lsu.edu

arXiv:2608.17147v1 [cs.CR] 17 Aug 2026

† Louisiana State University, Baton Rouge, LA, USA

Abstract—Image-to-image face generators are widely used, and visual dissimilarity between their outputs and source images is sometimes treated as evidence of privacy. Auditing whether these systems satisfy formal identity-level (ε, δ)-differential privacy requires choosing among several distinct routes for converting embedding-space observations into estimates or bounds on the differential privacy parameter ε. We present a comparative study of four such audits applicable to pre-trained, blackbox face generators: a Gaussian-mechanism reading of peridentity sensitivity (G AUSS M ECH); a per-dimension kerneldensity log-ratio aggregated by basic composition (KDELR); an analytical population-level lower bound on pureDP ε derived from the maximum mean discrepancy via the total variation distance (MMD-TV); and a hypothesis-testing evaluation of a cross-validated classifier’s out-of-fold ROC (ROC-HT). For each method we make explicit its assumptions, hyperparameter dependence, finite-sample limitations, and the regime in which its ε estimate is informative. Applied to FaceFusion and InstantID across multiple identity encoders and reference datasets, the audits consistently reveal substantial identity distinguishability while reporting markedly different ε estimates that reflect each method’s distinct assumptions and finite-sample treatment. In this high-distinguishability regime, the experiments do not support a reliable ranking of the four methods. Their relative trade-offs should be evaluated on partially private mechanisms, which we identify as the natural next study. The resulting framework places these audits in a shared identity-level audit setting and clarifies how their assumptions and finite-sample treatments shape the resulting differential privacy estimates. Index Terms—Differential privacy, privacy auditing, face recognition, maximum mean discrepancy, kernel density estimation, hypothesis testing, image-to-image generation.

1. Introduction Image-to-image face generators are now part of consumer and prosumer pipelines: face-swap systems such as FaceFusion [1] replace an identity in a video with another, and diffusion-based zero-shot identity models such as InstantID [2] synthesize a target person in new poses or scenes from a few reference images. These systems are not advertised as privacy-preserving; they are explicitly identitytransferring or identity-preserving by design. Nevertheless,

visual dissimilarity between an input frame and the generated output is sometimes treated as informal evidence of privacy, and as identity-aware image generation gets used in adjacent pipelines (face anonymization, deepfake mitigation, privacy-aware data augmentation) the question of how much identity actually survives the generator becomes a measurement question. We use FaceFusion and InstantID as high-distinguishability stress tests. InstantID is designed to preserve its reference identity; in our FaceFusion setup, one donor is fixed while the target images vary, allowing us to test whether the supposedly replaced target identity remains detectable in the output. Differential privacy [3], [4] provides a precise ratio constraint for such a measurement: a randomized mechanism M is (ε, δ)-DP if its output distributions obey the required inequality for every adjacent pair of private inputs. Here the private input is an identity label rather than a database. We use complete identity adjacency: every two distinct identities must satisfy the DP inequalities, and each identity indexes a per-query output distribution. Mathematically, this is a local-DP condition on the identity-label domain because the channel acts directly on one private label. We call it distributional because sampling an image from that identity’s reference distribution is part of the channel’s randomness. This secret-indexed view is related to Pufferfish privacy [5]. Several routes exist for turning an embedding-space observation into an estimate or lower bound on (ε, δ), and the literature does not converge on a single one. A sensitivityand-Gaussian-mechanism reading treats the difference between per-identity means as the input change and reads off a closed-form score. This borrows the algebra used by calibrated feature- and image-space mechanisms [6], [7], but its retrospective application to arbitrary generated outputs is a diagnostic rather than a formal lower bound. A per-dimension kernel-density log-ratio similarly gives a marginal diagnostic whose coordinate sum is not a jointmechanism guarantee. The hypothesis-testing characterization of differential privacy [8], [9] translates the population operating point of a fixed distinguisher into a lower bound on ε and underlies empirical DP auditing [10]– [12]. Prior work also connects maximum mean discrepancy (MMD) [13] to DP auditing [14]–[16]. We study a pureDP chain through total variation [8], [17], [18], compare it with the tighter Hockey-Stick-based result of González et al. [16], and evaluate all four routes on a shared task and

data. This paper presents that comparison for pre-trained, black-box image-to-image face generators. The four methods we study are: •

•

•

•

G AUSS M ECH (Section 3.1): a Gaussian-mechanisminspired diagnostic from the L2 distance between per-identity means and a pooled residual scale; KDE-LR (Section 3.2): a basic-composition sum of per-dimension kernel-density log-ratios, with a smoothing weight α; MMD-TV (Section 3.3): an analytical lower bound on pure-DP ε derived from the maximum mean discrepancy through the chain MMD ≤ 2 dTV and the TV characterization of pure DP; ROC-HT (Section 3.4): a hypothesis-testing evaluation based on the out-of-fold receiver operating characteristic of a cross-validated classifier.

For each method we make explicit its assumptions, its hyperparameter dependence, and the regime in which its ε estimate or lower bound is informative. We then evaluate all four on the same audit task across two identity encoders (ArcFace, FaceNet), two reference datasets (VGGFace2, CelebA), and two generators (FaceFusion, InstantID), for a total of eight conditions. Contributions. •

•

•

•

We present a comparative framework that places G AUSS M ECH, KDE-LR, MMD-TV, and ROCHT side by side on the same identity-substitution audit task with a shared per-condition pair list and a common 100-pair summary, and clarifies for each method which quantities are formal ε lower bounds and which are finite-sample diagnostics. We document the pure-DP MMD-TV specialization obtained by composing MMD ≤ 2 dTV [17] with the total variation characterization of pure DP [8], [18], state its population and finite-sample limits, and compare it explicitly with the tighter HockeyStick-based MMD result used in prior auditing work [15], [16]. We add a finite-sample ROC-HT protocol that separates pair and threshold selection from evaluation: all adaptive choices use a development split, while fixed classifiers and thresholds for 40 selected, vertex-disjoint identity pairs are evaluated once on untouched data with simultaneous one-sided exact confidence bounds. We empirically evaluate the four methods on eight (encoder × dataset × generator) conditions of FaceFusion and InstantID auditing. The four audits consistently reveal substantial identity distinguishability; the reserved-test protocol certifies positive ε lower bounds for all 20 selected VGGFace2 pairs, with 15 exceeding ε = 1, while its loss of power on CelebA exposes the sample cost of finitesample certification. The methods’ different scales still do not support a reliable ranking in this highdistinguishability regime.

The honest finding is that the four methods cannot be ranked on these generators because the audited regime is already highly distinguishable: every method reports substantial identity signal, but the reported numbers are controlled by method-specific assumptions, endpoint behavior, or looseness rather than by a shared notion of tightness. None of the evaluated off-the-shelf generators combines a known identity-level guarantee under our threat model with tunable, intermediate distinguishability. We therefore outline in Section 5 a controlled calibrated-noise benchmark on which the four methods could produce non-endpoint, comparable ε values.

2. Background and Threat Model 2.1. Differential Privacy A randomized mechanism M : D → Y is (ε, δ)differentially private with respect to an adjacency relation ∼ if, for every adjacent pair D0 ∼ D1 and every measurable S ⊆ Y, Pr[M(D0 ) ∈ S] ≤ eε Pr[M(D1 ) ∈ S] + δ.

(1)

Smaller ε corresponds to stronger privacy, and δ = 0 gives pure ε-DP [3]. The adjacency relation is application-defined. In our audit, D is the set of identity labels and idA ∼ idB whenever idA ̸= idB (Section 2.5). Thus “adjacent” means that the pair is required to satisfy the privacy inequality; it does not mean that the two faces are visually or semantically similar. An equivalent characterization in the language of hypothesis testing will be central to two of the four audits we compare. A mechanism M is (ε, δ)-DP if and only if, for every adjacent pair D0 ∼ D1 (in our setting, every two distinct identity labels) and every test distinguishing M(D0 ) from M(D1 ) with false-positive rate FPR and false-negative rate 1 − TPR, FPR+eε (1−TPR) ≥ 1−δ,

(1−TPR)+eε FPR ≥ 1−δ. (2) This characterization is developed by Kairouz, Oh, and Viswanath [8], and the view is reframed as f -DP by Dong, Roth, and Su [9]. ROC-HT is a direct instantiation of Equation 2.

2.2. Total Variation and Differential Privacy The chain that MMD-TV uses passes through the total variation distance. The relationship between TV and DP is one-directional: pure DP implies a TV upper bound, but a TV bound alone does not imply DP. We state the direction we need below. For an adjacent pair D0 ∼ D1 , define the output laws Pb = Law(M(Db )) for b ∈ {0, 1}. For any measurable rejection region A, let the test predict input D1 when the output lies in A. Its two error rates are βI (A) = P0 (A) and βII (A) = 1 − P1 (A), so βI (A) + βII (A) = 1 −

(P1 (A) − P0 (A)). Minimizing over all measurable A gives the classical identity



 dTV (P0 , P1 ) = 1 − inf βI (A) + βII (A) . A

(3)

Ghazi and Issa [18] (Section II, Definition 4 and Theorem 1), building on Kairouz, Oh, and Viswanath [8], show that any (ε, δ)-DP mechanism satisfies βI (A) + βII (A) ≥ 2(1 − δ)/(1 + eε ) for every test, so dTV (M(D0 ), M(D1 )) ≤

eε − 1 + 2δ . eε + 1

(4)

For pure ε-DP (δ = 0) this collapses to dTV ≤ tanh(ε/2). Since t 7→ tanh(t/2) is a bijection from [0, ∞) onto [0, 1), the inequality rearranges to 

1 + dTV ε ≥ ln 1 − dTV

 .

(5)

This is the inequality that closes the analytical chain of MMD-TV (Theorem 1).

2.3. Face Identity Encoders and Generators A face identity encoder is a function ϕ : I → Rd mapping an image x ∈ I to an identity embedding. We use precomputed 512-dimensional ArcFace/InsightFace embeddings [19], [20] and precomputed 512-dimensional InceptionResnet-V1 embeddings produced with facenet-pytorch [21]. The repository does not preserve the exact encoder checkpoint identifiers, so we do not infer a training set or checkpoint variant from the arrays alone. Before every audit, each embedding is L2-normalized to the unit hypersphere. We treat ϕ as a fixed identityoriented representation, not as ground truth; encoder errors propagate into all four audits (Section 8). We audit two generators from distinct architectural families. FaceFusion [1] is an open-source face-swap system, and InstantID [2] is a diffusion-based zero-shot identity generator. In every FaceFusion condition, a single donor is held fixed and the images associated with each dataset identity are supplied as targets; the stored identity label therefore denotes the target identity. InstantID conditions group outputs by the reference identity used for generation. The official InstantID pipeline uses InsightFace’s antelopev2 identity model; if the precomputed outputs followed that default, the ArcFace/InsightFace audit partially overlaps the representation used to condition generation. The absent generation manifest prevents us from verifying that configuration, and the FaceNet conditions provide the cleaner cross-encoder check. We use both systems as highdistinguishability stress tests rather than privacy mechanisms.

2.4. Maximum Mean Discrepancy Let k : X × X → R be a positive-definite kernel with reproducing kernel Hilbert space Fk . For two distributions P0 , P1 on X , the Maximum Mean Discrepancy [13] is MMD(P0 , P1 ; Fk ) =

sup f ∈Fk ∥f ∥Fk ≤1

EP0 [f ] − EP1 [f ] .

(6)

When k is characteristic (the RBF kernel k(u, v) = exp(−∥u − v∥2 /(2σ 2 )) with bandwidth σ > 0 is), MMD(P0 , P1 ) = 0 ⇐⇒ P0 = P1 , so MMD is a genuine metric on probability distributions. The squared form admits a closed-form unbiased estimator: given samples {xi }ni=1 ∼ P0 and {yj }m j=1 ∼ P1 , X 2 1 \u = k(xi , xi′ ) MMD n(n − 1) ′ i̸=i X 1 + k(yj , yj ′ ) m(m − 1) j̸=j ′ 2 X − k(xi , yj ). (7) nm i,j This estimator is unbiased for the population MMD2 and admits a permutation test for p-values under H0 : P0 = P1 [13]. In what follows, a sample estimate means that an unknown population quantity is replaced by its empirical estimate (often called a plug-in estimate). Unless accompanied by a one-sided confidence argument, this substitution yields a point estimate, not a certified population lower bound.

2.5. Threat Model Distributional mechanism. Let Pid be the reference-dataset distribution of images labeled with identity id, and let R collect any internal randomness of the generator. For InstantID, X ∼ Pid is a reference image and the generated image is Y = gID (X; R). For FaceFusion, X ∼ Pid is a target image, one donor image d0 is fixed for every identity, and the generated image is Y = gFF (X; d0 , R). The preencoder image-output channel and the embedding channel actually audited are Mimg (id) = Law(Y | X ∼ Pid , R ∼ PR ) , Mϕ (id) = Law(ϕ(Y ) | X ∼ Pid , R ∼ PR ) .

(8)

Thus identity-conditioned image sampling is part of the channel being measured. Each CelebA identity contributes 26–35 usable generated embeddings, whereas each VGGFace2 identity contributes 90–100. Operationally, the channel models one query whose image is drawn from the stated identity-conditioned population. It does not guarantee privacy for an arbitrary fixed photograph selected by a user, because that deployment conditions on one particular image rather than averaging over X ∼ Pid . This secret-indexed family resembles the Pufferfish organization of secrets and data-generating distributions [5], while Definition 1 imposes the ordinary pairwise DP inequalities. For InstantID, the

private label is the reference identity. For FaceFusion, it is the target identity; the donor d0 is identical for both classes in every pairwise test. The classifier receives one generated 512-dimensional embedding and predicts which of two target identities produced it, so target separation cannot be caused by changing the donor. Adjacency and post-processing. We use complete adjacency over identity labels: any two distinct labels idA and idB are adjacent. This is the DP convention specifying which input pairs must satisfy the inequality, not a claim that the identities are similar. Because Mϕ = ϕ ◦ Mimg and the encoder is deterministic, any DP guarantee for the generated-image distribution Mimg would also hold after encoding. Conversely, a confidence-certified violation for Mϕ refutes the same proposed guarantee for Mimg . Passing an encoder audit does not prove privacy of the generated image. Definition 1 (Identity-level (ε, δ)-DP). The distributional channel Mϕ is identity-level (ε, δ)-DP if, for every ordered pair of distinct identities (idA , idB ) and every measurable S ⊆ Rd ,

Here σ bid is the root-mean-square within-identity residual per embedding coordinate. The textbook sufficient Gaussian p calibration σ ≥ ∆2 2 ln(1.25/δ)/ε can be algebraically inverted [4]. Substituting the empirical pairwise separation and residual scales motivates the diagnostic p b 2 (A, B) 2 ln(1.25/δ) ∆ . (9) εbG AUSS M ECH (A, B) := max(σ bA , σ bB ) Using the larger of the two residual scales produces a smaller score than using the smaller scale. This implementation convention does not establish the DP inequality for the pair in either ordering. We report the per-pair median. Pros. •

•

Cons. •

Pr[Mϕ (idA ) ∈ S] ≤ eε Pr[Mϕ (idB ) ∈ S] + δ.

This is equivalent to requiring that no test of a single mechanism output can distinguish Mϕ (idA ) from Mϕ (idB ) better than the bound in Equation 2. Adversary. The adversary observes one draw from Mϕ and tests which identity distribution produced it. The precomputed outputs provide repeated observations used to estimate this single-draw channel; the paper makes no group-privacy claim for releasing the full batch.

•

3. Four Audit Methods •

3.1. G AUSS M ECH: Gaussian Calibration Diagnostic G AUSS M ECH applies the familiar Gaussian-calibration algebra to the distance between per-identity means and an empirical residual scale. The textbook sufficient calibration assumes additive, input-independent isotropic Gaussian noise, global sensitivity, and is commonly stated for ε ∈ (0, 1) [4]. Those conditions do not hold here, and inverting the expression to obtain scores in the tens extrapolates beyond that stated regime. We therefore treat the result only as a Gaussian signal-to-noise diagnostic. Definition. For identity id, let zid,i ∈ Rd denote its ith normalized generated embedding, and define n

id 1 X zid,i , nid i=1 v u nid u 1 X σ bid = t ∥zid,i − µ bid ∥22 , nid d i=1

µ bid =

b 2 (A, B) = ∥µ ∆ bA − µ bB ∥2 .

Closed form and interpretable. The score reduces each pair to mean displacement divided by a residual scale. Cheap. Once per-identity means and residual scales are computed, each pair costs O(d).

•

Gaussian residual assumption fails empirically. After L2 normalization, the per-dimension residual distributions are concentrated but non-Gaussian (Figure 1). A Shapiro–Wilk test of Gaussianity [22] on the pooled within-identity residuals rejects at p < 10−3 on 509 of 512 dimensions for the canonical ArcFace+VGGFace2+InstantID condition, and on a majority of the 512 dimensions in all eight of the conditions we audit. Isotropic noise assumption collapses crossdimensional variation. The single σ averages the per-dimension residual variance into one number, discarding heteroscedasticity across dimensions. Finite-sample treatment is implicit. Empirical means and variances are substituted without a confidence correction. Single per-pair number. The per-pair ε does not separate mechanism noise from sampling noise; an identity with high within-identity variance receives a small ε, which can mask identity leakage in a different geometric direction.

Hyperparameters. We fix δ = 10−5 for this diagnostic and use the root-mean-square within-identity residual per coordinate, σ bid , as the scale. Because the method is not a valid Gaussian-mechanism inversion for these generators, its δ should not be compared directly with the pure-DP MMDTV and ROC-HT formulas.

3.2. KDE-LR: Per-Dimension KDE Log-Ratio (Basic-Composition Sum) KDE-LR fits one-dimensional KDEs to the coordinate marginals and sums finite-grid log-ratio maxima. The sum is inspired by basic composition, but coordinate marginals of one vector release are not separate DP mechanisms and do not satisfy the conditional guarantees required by

Figure 1. Quantile–quantile plots for the four least-Gaussian ArcFace dimensions under VGGFace2+InstantID. Each panel uses the same 5,000-row subsample drawn from approximately 40,000 pooled within-identity residuals; dashed lines are Gaussian references.

composition. The result is therefore a product-of-marginals diagnostic, not a bound on the joint mechanism. Definition. For identity id ∈ {A, B} and dimension d′ ∈ (d′ ) {1, . . . , d}, let p̂id denote the kernel density estimate of the marginal ϕ(g(X))d′ | X ∼ id, computed with a Gaussian kernel and bandwidth set by Silverman’s rule of thumb [23]. Given a smoothing weight α ∈ [0, 1], define the smoothed density (d′ )

(d′ )

′

p̂id,α (x) := (1 − α) p̂id (x) + α u(d ) (x),

(10)

′

where u(d ) is uniform over a finite interval containing both (d′ ) identities’ samples plus a three-bandwidth margin. Let GA,B be the 201-point grid on that interval, let τ = 10−12 , and write p̃ = max{p̂, τ }. The implemented diagnostic is (d′ )

′

(d′ )

) εb(d α (A, B) := max′ log p̃A,α (x) − log p̃B,α (x) , (d ) x∈GA,B

εbKDE-LR (A, B) :=

d X

′

) εb(d α (A, B).

(11)

d′ =1

Separate Silverman bandwidths are fitted for A and B ; their maximum is used only to set the grid margin. We report perpair medians at α = 0 and 10−2 . Pros. •

•

•

Cons.

Nonparametric. The KDE makes no Gaussian assumption on the per-dimension marginals. It accommodates multimodal and heavy-tailed shapes that G AUSS M ECH cannot model. Captures per-dimension shape. The pointwise logratio responds to local differences in the marginals, not only to mean displacement. Transparent sensitivity. The effect of smoothing, bandwidth, grid resolution, and the density floor can be inspected directly.

Per-dimension factorization loses joint structure. Marginals can either exaggerate coordinate-wise differences or miss leakage that exists only in dependence, so their sum can overstate or understate joint distinguishability. • No DP composition interpretation. Basic composition applies to mechanisms with appropriate conditional guarantees, not to fitted marginals of one release. • Operationally large. On our data the composed ε is in the thousands across all eight conditions (Table 1). This reflects the looseness of the per-dimension framing, not a tight bound on the mechanism’s joint ε. • Hyperparameter-dependent. The smoothing weight α has no principled choice; on our data, increasing α from 0 to 10−2 decreases the composed ε by a factor of approximately five (Table 1), and there is no internal criterion that selects one of these settings over the other. • Tail and discretization artifacts. The finite grid and density floor prevent numerical divergence but directly affect the large α = 0 values. Hyperparameters. KDE-LR depends on α, separate identity-specific Silverman bandwidths, the Gaussian KDE kernel, the 201-point grid and three-bandwidth margin, and the floor τ = 10−12 . •

3.3. MMD-TV: Analytical Lower Bound via Total Variation MMD-TV derives a population-level lower bound on ε by composing MMD ≤ 2dTV for bounded-diagonal kernels [17] with the total-variation characterization of pure DP [8], [18]. The constituent inequalities are known; our role is to expose this specialization and its empirical behavior inside the comparison.

Definition. Let PA and PB denote the distributions of Mϕ under identities A and B . For a positive-definite kernel k , the population MMD-TV bound is   1 + MMD(PA , PB )/2 . (12) εMMD-TV := ln 1 − MMD(PA , PB )/2 Its population justification follows from Lemma 1 and Theorem 1; empirical results later evaluate the expression using a sample MMD estimate without a one-sided confidence correction. Lemma 1 (MMD ≤ 2 dTV for bounded-diagonal kernels). Let k : X × X → R be a positive-definite kernel with k(x, x) ≤ 1 for all x ∈ X . For any two distributions P0 , P1 on X , MMD(P0 , P1 ; Fk ) ≤ 2 dTV (P0 , P1 ).

interpretable, but we do not claim it improves the prior bound. Mapping to ε. We use an RBF kernel with a per-pair median-heuristic q bandwidth. The implementation computes 2 \ u , 0}, substitutes it into Equa\ max{MMD MMD = tion 14, and reports the maximum sample estimate over 500
audited pairs. A condition with 399 identities has 399 = 79,401 unordered candidate pairs, while a condition 2
= 79,800 . The reported maximum is with 400 has 400 2 therefore neither the supremum over all candidate pairs nor a one-sided population certificate. Pros. •

(13) •

Proof. For any f ∈ Fk with ∥f ∥Fk ≤ 1, the reproducing property and p Cauchy–Schwarz give |f (x)| = |⟨f, k(·, x)⟩Fk | ≤ ∥f ∥Fk k(x, x) ≤ 1. Therefore the unit ball of Fk is contained in the unit ball of L∞ , and MMD(P0 , P1 ; Fk ) ≤

sup

•

EP0 f − EP1 f

∥f ∥∞ ≤1

= 2 dTV (P0 , P1 ).

This is a specialization of the MMD-vs.-IPM comparison in [17]. Theorem 1 (MMD-TV analytical lower bound on ε). Let M be any pure ε-DP mechanism on an adjacent pair D0 ∼ D1 . Let Pb = Law(M(Db )) for b ∈ {0, 1}, and let k be a positive-definite kernel with k(x, x) ≤ 1. Then   1 + MMD(P0 , P1 )/2 ε ≥ ln . (14) 1 − MMD(P0 , P1 )/2 The bound is valid at the population level for any pure εDP mechanism; it is generally loose because the inequality MMD ≤ 2 dTV is itself loose unless the MMD witness function attains the boundary of the L∞ ball. We restrict the MMD-TV statement to pure DP in this paper rather than using an approximate-DP slack term. Proof. The pure-DP TV bound [8], [18] states that any pure ε-DP mechanism satisfies dTV (P0 , P1 ) ≤ tanh(ε/2) = (eε − 1)/(eε + 1). Inverting the bijection t 7→ tanh(t/2) from [0, ∞) to [0, 1) gives ε ≥ 2 arctanh(dTV ) = ln((1 + dTV )/(1 − dTV )). By Lemma 1, dTV (P0 , P1 ) ≥ MMD(P0 , P1 )/2. The map t 7→ ln((1+t)/(1−t)) is monotone increasing on [0, 1), so substituting MMD(P0 , P1 )/2 for dTV (P0 , P1 ) can only decrease the right-hand side, preserving the lower bound. Comparison to the Hockey-Stick-based bound. González et al. [16], improving Kong et √ al. [15], show for 0 ≤
ε k(x, y) ≤ 1 that MMD(P0 , P1 ) ≤ 2 1−2(1−δ)/(1+e ) . √ At δ = 0, this inverts to ε ≥ 2 arctanh(MMD/ 2), which is tighter than Equation 14 for the nonnegative RBF kernel used here. Our TV chain requires only positive definiteness and k(x, x) ≤ 1 and makes the source of its slack directly

•

Distribution-free. The bound holds for arbitrary PA , PB , with no Gaussianity or independence assumption on the embedding marginals. Joint embedding. MMD with a characteristic kernel operates on the full d-dimensional embedding via its RKHS mean embedding, rather than on perdimension marginals. Analytical at the population level. Equation 14 is a chain of implications that gives a population-level lower bound. The audit reports the value of the same 2 \u expression evaluated at the unbiased sample MMD estimator; see the Cons block for the population-tosample distinction. Inspectably bandwidth-dependent. The sample estimate remains positive across the bandwidth sweep on the two canonical ArcFace+VGGFace2 conditions.

Cons. •

•

•

•

Loose. The classifier-restricted empirical TV diagnostic is 4.4–7.5 times larger than the MMD/2 sample estimate on average. Because the classifier itself need not attain total variation, this gap cannot be attributed exclusively to Lemma 1. Sub-unit sample estimates. Every reported value is below one; such an estimate would not by itself exclude a mechanism with ε = 1 even if it received a valid confidence correction. Population-to-sample gap. The bound is stated for the population MMD. The audit substitutes the un2 \ u estimator; this estimator is not a onebiased MMD sided lower bound on the population MMD2 and can overshoot, so the reported εMMD-TV is a point estimate rather than a high-probability lower bound. Kernel choice. The theorem requires positive definiteness and k(x, x) ≤ 1; characteristicness is needed only for MMD to identify equality of distributions.

Hyperparameters. MMD-TV has three: (i) the kernel family (we use RBF, fixed throughout); (ii) the bandwidth σ (we use the median heuristic on each per-pair pooled embedding); (iii) the audited pair sample size (we use 500 unordered pairs drawn uniformly without replacement from the candidate set). The first is a discrete choice; the second

has a data-driven default; the third trades off audit cost for tightness of the per-pair maximum.

3.4. ROC-HT: Hypothesis-Testing Bound from a Classifier’s ROC ROC-HT applies the hypothesis-testing characterization of differential privacy [8], [9] to a classifier trained to distinguish two identity distributions. At the population level, a fixed distinguisher gives a lower-bound route. We use pooled out-of-fold ROC curves for the broad comparative diagnostic, then run a separate development-selected, reserved-test protocol for finite-sample certification. Definition. For each identity pair (A, B), train a crossvalidated classifier hA,B : Rd → [0, 1] on the pooled embedding set {(ϕ(g(xk )), A)}k ∪ {(ϕ(g(yk )), B)}k . Concatenate the out-of-fold predicted probabilities across the cross-validation folds to obtain a per-pair receiver oper[ i , TPR [ i )}i . The Kairouz–Oh– ating characteristic {(FPR Viswanath bound applied at each operating point and maximized gives ) ( [i [ i TPR 1 − FPR . , εbROC-HT (A, B) := max ln max 1, i [i [ i FPR 1 − TPR (15) We use 5-fold stratified cross-validation with a fixed random seed and record the per-pair empirical value of Equation 15 over the observed out-of-fold operating points. Because the classifier and maximizing threshold are data-dependent, these values should be read with the finite-sample caveats below. Pros. •

• •

Distribution-free at the population level. Equation 15 follows the Kairouz–Oh–Viswanath characterization of pure DP at δ = 0. For a fixed population distinguisher, it requires no parametric model for the output distribution. Improves with the distinguisher. A stronger fixed population test can yield a larger lower bound. Cross-validation reduces training-set optimism. Outof-fold operating points are not evaluated on the same examples used to train each fold’s classifier. This reduces training-set optimism, but it is not a substitute for a finite-sample confidence certificate.

Cons. •

•

Diverges at ROC endpoints. The raw empirical OOF value is +∞ whenever the ROC contains an oper[ = 1 and FPR [ < 1, or with ating point with TPR [ [ FPR = 0 and TPR > 0. This endpoint divergence is algebraically correct for Equation 15, but it does not by itself imply exact zero-error separation. We therefore report AUC and separately track the stricter [ TPR) [ = (0, 1). exact corner (FPR, Sensitive to classifier choice. The bound is only as tight as the trained classifier. A random forest

•

•

in place of logistic regression returns a different operating point on the same data, which is why we report classifier sensitivity as a diagnostic rather than treating one classifier as canonical. The out-of-fold value is not a certificate. Five foldspecific classifiers produce the pooled scores, and the maximizing threshold is selected on those same outof-fold predictions. We therefore keep this quantity as a diagnostic and use a separate reserved-test protocol with exact simultaneous confidence treatment in Section 4.3. Sample-size dependence in the endpoint regime. The OOF value +∞ is an empirical finite-sample statement about the observed ROC. A larger audit could reveal overlap and move the relevant operating points away from the zero-denominator endpoints.

Hyperparameters. ROC-HT has three: (i) the classifier family (we use logistic regression and include a randomforest sensitivity check in Section 4.4); (ii) the number of cross-validation folds (we use 5); (iii) the per-pair sample size, which is fixed by the data (n ≈ 100 on VGGFace2, n ≈ 30 on CelebA). The first is the principal lever for tightening the bound; the second has the conventional choice; the third is determined by the reference dataset.

4. Comparative Experimental Evaluation 4.1. Setup Datasets. We use selected identity subsets from VGGFace2 [24] and CelebA [25]. Seven conditions retain 400 identities; FaceNet+CelebA+FaceFusion retains 399 after filtering. Each VGGFace2 identity has 90–100 usable generated embeddings, while each CelebA identity has 26–35. These provide contrasting sample-size regimes. Generators and provenance. The audit input consists of eight CSV files of precomputed original/generated embedding pairs for all encoder × dataset × generator combinations. Each row stores an identity label and its generated embedding. For FaceFusion [1], one donor is fixed across all targets, so the label denotes the target identity; for InstantID [2], it denotes the reference identity. The files do not include a versioned generation manifest with exact revisions, prompts, seeds, checkpoint hashes, or the donor asset identifier. We therefore report only provenance preserved by the files and repository. Throughout the tables and figure legends, Gen. denotes the generator; FF and ID denote FaceFusion and InstantID, respectively; and AF, FN, CB, and VGG denote ArcFace, FaceNet, CelebA, and VGGFace2, respectively. Identity encoders. The files contain embeddings produced by ArcFace/InsightFace and facenet-pytorch; exact checkpoint identifiers are unavailable. We remove rows whose generated embedding is missing or has norm at most 10−6 , then L2-normalize every retained vector. Because the two encoder files can lose different rows during this filtering, their results test whether the qualitative conclusion persists

across representations; they are not comparisons on exactly the same retained images. The official InstantID configuration uses InsightFace antelopev2, so FaceNet+InstantID is also the cleaner check against possible representationfamily overlap. Pair sampling. The 399- and 400-identity conditions contain 79,401 and 79,800 possible unordered identity pairs, respectively. For each condition, a fixed seed shuffles this full list once. Every method uses the first 100 pairs; G AUSS M ECH and MMD-TV additionally use pairs 101–500, and ROC-HT additionally uses pairs 101–200. Thus Table 1 compares medians on exactly the same 100 pairs, while method-specific sensitivity analyses use the stated larger prefixes of the same ordering. G AUSS M ECH configuration. We set δ = 10−5 , compute the distance between the two sample mean embeddings, and divide by the larger of their two root-mean-square residual scales as specified in Equation 9. KDE-LR configuration. Identity-specific Silverman bandwidths are used to evaluate Gaussian KDEs on 201-point grids. Each grid spans the joint sample range plus three times the larger bandwidth; densities are floored at 10−12 . We report α ∈ {0, 10−2 }. MMD-TV configuration. We use an RBF kernel, a per-pair median-heuristic bandwidth, and the unbiased estimator in Equation 7. Before taking a square root, negative estimates are clipped to zero. Permutation tests use 104 permutations on the first 100 pairs per condition with the Phipson–Smyth correction [26]. ROC-HT configuration. Per identity pair, we train scikitlearn’s LogisticRegression [27] with C = 1.0, the lbfgs solver, and max_iter = 1000 on the L2-normalized 512-dimensional embeddings, under 5-fold StratifiedKFold cross-validation with a fixed random seed. ROC curves are built from the concatenation of outof-fold predicted probabilities across all 5 folds. Compute and reproducibility. The unified audit script reads the precomputed embedding CSVs; it does not rerun image generation or encoder inference. The artifact records SHA-256 hashes, row and identity counts for all eight files, the software environment, and the shared pair lists, while explicitly listing the generation metadata that are unavailable.

4.2. Comparative Audit Across Eight Conditions Table 1 uses the same first 100 identity pairs per condition. Figure 2 summarizes the four routes and the formal status of their outputs. The first three finite ε-scale columns have different formal status, while ROC-HT reports both its raw OOF infinity rate and AUC. Table 2 separately reports confidence-certified ROC lower bounds. On the common pairs, median AUC is 0.921–1.000 and median MMD-TV is 0.217–0.475. Every raw OOF ROC estimate is +∞ because each empirical ROC has a zerodenominator endpoint; this also holds over all 200 ROCtested pairs per condition. The exact (0, 1) corner count depends on the condition and ranges from 0 to 130 of 200

pairs. The infinity is therefore an endpoint-sensitive empirical result, not a population certificate. Method-specific maximum MMD-TV sample estimates over 500 pairs are 0.406– 0.719. Figure 3 shows the pooled same-identity versus inter2 \ u distributions and one representative out-ofidentity MMD fold ROC curve per condition.

4.3. Finite-Sample ROC Certificates on Reserved Test Data The out-of-fold ROC summaries above are broad diagnostics, not confidence certificates. We therefore run a separate protocol that makes all adaptive choices before touching the evaluation data. Within each condition, every identity’s samples are split once into 70% development and 30% test subsets. On development data only, we score the common first 100 pairs with five-fold out-of-fold predictions, greedily select the five highest-ranked pairs subject to no identity appearing twice within a condition, and choose one threshold for each pair. We then fit one logistic-regression classifier per selected pair on its full development data. Each fixed classifier and threshold is evaluated once on the corresponding identities’ reserved test subsets. The disjointness constraint prevents a few identities from supplying most of the condition-level evidence; it is not required for the unionbound guarantee. Let UFPR and UFNR be one-sided Clopper–Pearson upper confidence bounds [28] for the two test error rates. We allocate α = 0.05/(8 · 5 · 2) = 0.000625 to each rate: two rates for each of five pairs in each of eight conditions. A union bound therefore gives simultaneous coverage of at least 95% for all 40 pairwise certificates, without assuming independence between pairs or conditions. Substituting the upper error bounds into Equation 2 yields   1 − δ − UFPR 1 − δ − UFNR , ln , εLB (δ) = max 0, ln UFPR UFNR (16) where a term is omitted if its numerator is nonpositive. Pair and threshold selection do not consume test-set coverage because they use only the development split. Under the iid interpretation of samples from each Pid , the bounds in Table 2 hold simultaneously with at least 95% confidence for all 40 development-selected pairs. Every one of the 20 VGGFace2 pairs certifies a positive pure-DP lower bound; 15 certify ε > 1. The FaceNet+VGGFace2+InstantID condition is weaker but still positive on all five pairs, with bounds from 0.685 to 0.925. The CelebA result exposes a finite-sample limit rather than a contradiction with its high AUC. Eleven of its 20 selected pairs make no test error, yet the globally corrected one-sided upper error bound is at least 0.522 with only 10 test samples per class, so Equation 16 cannot certify a positive ε. Under the same 40-pair correction, a balanced zero-error test gives εLB = 0 at n = 10 per class, 0.454 at n = 15, and 1.277 at n = 30. Thus AUC can show strong empirical ordering while exact DP certification remains sample-limited. Because identity-level DP must hold for

Figure 2. Four-audit framework, formal status, and principal evidence. Shared identity-pair lists feed two modeling diagnostics (G AUSS M ECH, KDE-LR), an uncertified sample estimate for the population MMD-TV route, and broad OOF ROC-HT diagnostics. A separate development-selected, reserved-test ROC-HT branch supplies 40 simultaneous pairwise certificates. TABLE 1. C OMMON - PAIR COMPARISON ON L2- NORMALIZED EMBEDDINGS . F INITE COLUMNS ARE MEDIANS OVER 100 SHARED PAIRS . T HE ROC-HT COLUMN COUNTS RAW OOF EMPIRICAL VALUES EQUAL TO +∞ DUE TO OBSERVED ZERO - DENOMINATOR ENDPOINTS ; IT IS NOT AN INFINITE - POPULATION CLAIM . G AUSS M ECH KDE-LR KDE-LR MMD-TV ROC-HT OOF diagnostic α=0 α=10−2 sample est. ε b = +∞

ROC AUC

Encoder Dataset

Gen.

ArcFace ArcFace ArcFace ArcFace

VGGFace2 VGGFace2 CelebA CelebA

FF ID FF ID

93.1 46.1 81.6 58.7

10,447 8,560 8,271 7,249

1,657 1,334 1,689 1,523

0.475 0.236 0.388 0.257

100/100 100/100 100/100 100/100

1.000 0.982 1.000 0.978

FaceNet FaceNet FaceNet FaceNet

VGGFace2 VGGFace2 CelebA CelebA

FF ID FF ID

91.8 44.2 80.3 54.0

9,976 7,872 8,659 7,190

1,652 1,278 1,721 1,502

0.459 0.217 0.381 0.243

100/100 100/100 100/100 100/100

0.999 0.937 0.992 0.921

every pair, each certified pair rules out a smaller mechanismwide ε for the defined channel. The five pairs are adaptively chosen stress tests, not an estimate of leakage prevalence or of the supremum over all identity pairs, and they do not make the OOF or MMD-TV sample estimates confidencecertified.

4.4. Per-Method Sensitivity G AUSS M ECH: Gaussianity of the per-dimension residual. G AUSS M ECH’s interpretation is only meaningful when the generator behaves like an isotropic Gaussian mechanism in embedding space. We pool the within-identity residuals ϕ(g(x))d′ − µ̂id,d′ across all retained identities (399 in one condition and 400 in the other seven) and use Q-Q plots together with Shapiro–Wilk tests of Gaussianity [22] as

diagnostics. Because the pooled residual sample is large and not a perfect iid sample from a single identity, the pvalues should not be read as exact model-selection tests. They are nevertheless consistent with the visual diagnostics: rejection at p < 10−3 across the 512 dimensions ranges from 296/512 on ArcFace+CelebA+InstantID to 512/512 on ArcFace+VGGFace2+FaceFusion, and exceeds 300/512 on seven of the eight conditions. The Gaussian residual assumption that G AUSS M ECH relies on is therefore not supported in most audited conditions. KDE-LR: sensitivity to the smoothing weight α. Table 1 reports the per-pair summed diagnostic at α = 0 and α = 10−2 . The ratio between the two settings is between 4.8 and 6.4 across the eight conditions, with α = 10−2 giving the smaller value. The per-dimension log-ratio at α = 0 is

TABLE 2. R ESERVED - TEST ROC-HT CERTIFICATES FOR FIVE DEVELOPMENT- SELECTED , VERTEX - DISJOINT PAIRS PER CONDITION . AUC AND εLB COLUMNS SHOW RANGES ACROSS THE FIVE RESERVED - TEST EVALUATIONS . C OUNTS AND BOUNDS ARE SIMULTANEOUS AT 95% OVER ALL 40 PAIRS ; ZERO MEANS THAT THE FINITE TEST DOES NOT CERTIFY A POSITIVE BOUND .

Encoder

Dataset

Gen. Test n per class

ArcFace ArcFace ArcFace ArcFace

VGGFace2 VGGFace2 CelebA CelebA

FF ID FF ID

30 1.000–1.000 29–30 0.986–1.000 9–10 1.000–1.000 9 1.000–1.000

5/5 5/5 0/5 0/5

5/5 1.192–1.277 5/5 1.151–1.277 0/5 0 0/5 0

FaceNet FaceNet FaceNet FaceNet

VGGFace2 VGGFace2 CelebA CelebA

FF ID FF ID

30 0.997–1.000 29–30 0.970–0.999 9–10 1.000–1.000 9 0.914–1.000

5/5 5/5 0/5 0/5

5/5 1.192–1.277 0/5 0.685–0.925 0/5 0 0/5 0

dominated by the tails where the KDE approaches zero; the mixture with the per-dimension uniform pulls the log-ratio out of those tails. Neither setting has a principled choice in the underlying construction. MMD-TV: bandwidth. The sample estimate depends on the RBF bandwidth σ , which we set by the median heuristic (σ ⋆ ). We sweep σ/σ ⋆ ∈ {0.5, 0.75, 1.0, 1.5, 2.0} on the canonical ArcFace+VGGFace2 condition using the same L2-normalized pair list as Table 1. The resulting maximum audited-pair estimates are {0.677, 0.724, 0.642, 0.480, 0.374} for FaceFusion and {0.413, 0.448, 0.406, 0.310, 0.244} for InstantID. The value is largest at 0.75 σ ⋆ for both generators and remains positive across the swept range; this sensitivity is one reason not to read the sample maximum as a certificate. MMD-TV: comparison with a classifier witness. For any fixed score threshold, TPR − FPR is a lower bound on total variation. We therefore compute the classifier-restricted diagnostic dbTV,h = maxt [TPR(t) − FPR(t)] from the outof-fold scores. Here a witness function is the function f achieving (or approximating) the supremum that defines an integral probability metric: for the population MMD, the witness f ⋆ lies in the unit ball of the RKHS Fk and attains |EP f ⋆ − EQ f ⋆ | in Equation 6; for the classifier-restricted dbTV,h the witness is the threshold indicator 1[h(x) > t⋆ ] at the operating point that maximizes TPR − FPR. Across the eight conditions, the mean ratio dbTV,h /(MMD/2) ranges from 4.39× to 7.52× on the 200 ROC-tested pairs, and mean dbTV,h ranges from 0.735 to 0.992. This gap shows that the RKHS witness scaled by 1/2 (the quantity Lemma 1 bounds) is much smaller than the classifier-threshold witness on these samples; because neither sample quantity is the population TV, the gap does not isolate the slack of Lemma 1. ROC-HT: classifier sensitivity. To check whether the ROC signal is an artifact of the linear scorer, we rerun ROC-HT with a random forest [29] (100 trees, otherwise default scikit-learn settings) on the same identity pairs and identical 5-fold splits. The per-pair AUC difference (logistic minus random forest), averaged over the 200 ROC-tested pairs per condition, ranges from −0.002 (FaceNet+VGGFace2+FaceFusion, where random forest is marginally better on average) to +0.013 (Arc-

Test AUC εLB > 0 εLB > 1

εLB (0)

Face+CelebA+InstantID); the two classifiers’ mean AUC agrees to within 0.013 in every condition. The concern this addresses is the converse direction: if logistic regression under-fit the available signal, a strictly more flexible classifier should expose distinguishability we missed and raise the AUC. Random forest does not: it matches logistic to within 0.013. The high-AUC reading therefore reflects the distinguishability that either scorer can extract, not a ceiling imposed by an overly restrictive linear model. We do not use this sensitivity check to make a new finite-sample confidence claim about ε; it is a robustness diagnostic for the classifier choice. Sanity checks across the four audits. Three support experiments probe spurious explanations for the main signal. (i) The inter-identity MMD permutation tests reject equality on 799/800 tested pairs, whereas same-identity split-half MMD2 values are near zero and the corresponding tests reject 16/400 controls at α = 0.05, consistent with nominal calibration. (ii) Same-identity split-half ROC-HT controls have mean AUC near chance across all conditions and no exact (0, 1) corners. (iii) All 20 selected VGGFace2 pairs receive positive simultaneous reserved-test certificates, whereas the smaller CelebA test sets cannot certify positivity even for zero-error pairs under the global 40-pair correction. These checks reduce, but do not eliminate, the possibility that dataset or generation artifacts contribute to the interidentity separation.

4.5. Honest Comparison The methods cannot be ranked from these experiments. The classifier and MMD tests provide direct empirical evidence of identity distinguishability, the reserved-test ROC protocol certifies all 20 selected VGGFace2 pairs under one simultaneous confidence family, and the two modeling diagnostics are consistent with the same qualitative conclusion. Their numerical scales differ for method-specific reasons and do not reveal relative tightness at intermediate privacy levels. A controlled benchmark with a known privacy guarantee and non-endpoint behavior is needed for that comparison.

Figure 3. Left: distribution of the unbiased MMD2 statistic for H0 (same identity, split-half pairs) versus H1 across the eight encoder × dataset × generator conditions, computed on L2-normalized embeddings. The panel pools real split-half H0 values and real inter-identity H1 values; no synthetic null samples are used. Right: actual out-of-fold ROC curves from the cross-validated logistic-regression classifier used by ROC-HT, showing one representative medianAUC pair per condition. The curves show high empirical distinguishability without implying that every tested pair contains the exact (0, 1) corner; the observed count ranges from 0 to 130 of 200 pairs across conditions.

5. Discussion 5.1. Why the Four Methods Cannot Be Ranked Here Not every number in Table 1 has the same formal status. MMD-TV is a population pure-DP lower-bound route evaluated here with an uncertified sample estimate; ROC-HT is a hypothesis-testing route whose broad OOF estimate diverges at ROC endpoints, while Table 2 gives separately certified selected-pair bounds; G AUSS M ECH and KDE-LR are diagnostics tied to modeling and aggregation assumptions. A natural question is still which audit is most informative. On the eight conditions of Table 1, that question is not well-posed. The generators are already highly distinguishable, and the reported values are dominated by each method’s assumptions, confidence treatment, or endpoint behavior: KDE-LR reports values in the thousands, G AUSS M ECH in the tens, MMD-TV below one, and ROCHT has high AUC with raw endpoint-divergent OOF values. The four methods therefore do not separate cleanly on these generators.

5.2. A Benchmark on Which the Four Methods Could Differ The cleanest comparison would use a per-query benchmark with a known privacy guarantee and a controllable range of distinguishability. One construction is to add cal-

ibrated isotropic Gaussian noise to the normalized encoder output: fν (id) = ϕ(g(X)) + ξ, M X ∼ Pid , ξ ∼ N (0, ν 2 Id ). where Pid is the identity’s source-image distribution and ν is varied. Because L2 normalization places every output on the unit sphere Sd−1 , the L2 distance between any two outputs is at most 2 (the diameter of Sd−1 ). The per-pair L2 sensitivity of the mean-shift mechanism this benchmark is calibrated against therefore satisfies ∆2 ≤ 2, so the analytic Gaussian-mechanism calibration of Balle and Wang [30] at a fixed nonzero δ supplies a reference upper guarantee. The audits could then be compared against that reference as ν increases and the ROC leaves its endpoint regime. This experiment would evaluate calibration and sensitivity, not turn the original black-box generators into certified mechanisms. Such a controlled benchmark is needed because none of the evaluated off-the-shelf generators provides both a known guarantee for our identity channel and tunable intermediate distinguishability.

5.3. Which Method to Use When The four methods are not interchangeable. G AUSS M ECH is a cheap diagnostic for mechanisms plausibly dominated by additive Gaussian noise. KDE-LR exposes coordinatelevel marginal differences but its sum is not a joint DP guarantee. MMD-TV supplies a classifier-free population lower-bound route, provided its sample estimate receives a one-sided finite-sample treatment. ROC-HT can exploit

a strong distinguisher; our reserved-test protocol shows how a fixed, independently evaluated test and simultaneous confidence correction turn that route into a finite-sample certificate, at the cost of reduced power on small test sets. These requirements, rather than the magnitudes in Table 1, should determine the audit choice.

6. Related Work DP auditing. Jagielski, Ullman, and Oprea [10] audit DPSGD using membership-inference attacks; Nasr et al. [11] instantiate strong canary adversaries; and Steinke, Nasr, and Jagielski [12] develop single-run auditing. These works use hypothesis tests to lower-bound effective privacy parameters. Their neighborhoods and mechanisms differ from ours: they audit training algorithms under record-level changes, whereas we study an inference-time distribution indexed by identity labels. The hypothesis-testing conversion is not our contribution; our focus is its comparison with three other embedding-space routes in this identity-level setting. Training-data privacy in generative models. Carlini et al. [31] demonstrate extraction of memorized diffusionmodel training examples, while Hu and Pang [32] study membership inference against denoising diffusion models. Those works ask whether training records are memorized or identifiable. We instead ask whether a generated-output distribution is distinguishable across two input identity labels. The threat models are complementary and neither result implies the other. DP for image and generative mechanisms. DP-SGD protects a training algorithm under a specified dataset neighborhood [33], whereas feature-space and local-DP image mechanisms add calibrated noise to an image or representation before subsequent processing [6], [7]. These guarantees answer different questions from our distributional inference-time audit. Our G AUSS M ECH diagnostic borrows the algebra of Gaussian calibration, but neither it nor KDELR inherits a guarantee from those prior mechanisms when applied retrospectively to arbitrary black-box outputs. MMD, total variation, and the hypothesis-testing characterization of DP. The kernel two-sample formulation used here follows Gretton et al. [13]. Sriperumbudur et al. [17] studied its relationship to other integral probability metrics, including the comparison underlying Lemma 1. The TV and hypothesis-testing characterizations of DP follow Kairouz, Oh, and Viswanath [8], Ghazi and Issa [18], and the f DP formulation of Dong, Roth, and Su [9]. Kernel-based DP auditors include Domingo-Enrich and Mroueh [14], DPAuditorium [15], and the sequential MMD audit of González et al. [16]. The mathematical pieces of MMD-TV are therefore pre-existing; our contribution is their explicit pureDP specialization and their assumption-matched empirical comparison beside three other identity-level audit routes.

7. Ethics and Dual Use This work measures identity leakage in face-generation systems and therefore involves sensitive biometric data and

dual-use risk. Its intended use is defensive evaluation of privacy claims. The same methods could support identity recognition, so we report aggregate embedding-level results and do not release generated faces or identity-specific attack artifacts. The study uses selected subsets of public face datasets and precomputed outputs; it collects no new participant data, conducts no user study, and does not attempt to re-identify unnamed individuals. Public availability does not remove concerns about consent, demographic bias, or downstream misuse. Encoder performance may vary across demographic groups, but the available metadata do not support a subgroup analysis; this limits the fairness conclusions that can be drawn. Artifact releases should exclude face images, minimize identity-linked metadata, document dataset licenses and model terms, and state the intended defensive use.

8. Limitations and Future Work Generators, data, and provenance. The comparison is limited to precomputed FaceFusion and InstantID outputs built from selected VGGFace2 and CelebA identities. The loader removes missing and zero-norm vectors; after filtering, the ArcFace+CelebA+FaceFusion and FaceNet+CelebA+FaceFusion conditions each retain one duplicated vector. The FaceFusion protocol fixes one donor and varies target images by dataset identity, so the stored labels represent target identities. However, the repository lacks a versioned generation manifest containing exact revisions, prompts, seeds, checkpoints, and the donor asset identifier. We therefore cannot reproduce image generation from this repository alone or attribute differences to a specific generator revision. Other generators, data distributions, donors, or preprocessing choices may produce different results. Encoder as the identity oracle. The four audits treat the identity encoder ϕ as a fixed proxy for identity. A generator that leaks identity as either encoder defines it but not as a human defines it would still register strong distinguishability under our audits. We probe encoder dependence with ArcFace/InsightFace and FaceNet representations (Section 2.3); the qualitative verdict is preserved across both, although the numerical values shift. Exact checkpoint identifiers for the precomputed embeddings are unavailable. In addition, InstantID’s official pipeline uses the ArcFacefamily antelopev2 representation for conditioning, so the ArcFace+InstantID result may benefit from representation overlap; FaceNet+InstantID is the cleaner cross-encoder check. High-distinguishability regime. The principal limitation of this paper is that the audited generators are already highly distinguishable (Section 4.5). The comparison is honest about this: it does not allow us to claim that one method is tighter than another at intermediate privacy levels. The natural follow-up is the calibrated-noise mechanism sketched in Section 5, on which the four methods would produce nonendpoint, comparable ε values across a sweep of the noise scale.

Per-query versus multi-query privacy. Definition 1 is a per-query identity-level notion: the mechanism output is one generated embedding, and the N images per identity are used to estimate the corresponding per-identity output distribution. We do not claim a group-privacy guarantee for releasing the full generated batch from an identity. An adversary who observes many independent outputs from the same identity can compose evidence across queries, and translating the per-query audit numbers to a multiquery release requires specifying the composition model and the number of queries. We therefore avoid dividing or multiplying the reported ε values by N in the main comparison. Finite-sample certification. The population MMD and hypothesis-testing inequalities are valid, but the reported MMD-TV and broad OOF ROC sample estimates are not confidence-certified lower bounds. Section 4.3 adds a separate reserved-test ROC evaluation with simultaneous exact confidence treatment for five development-selected, vertexdisjoint pairs per condition. Its guarantees are conditional on iid sampling from the defined Pid , concern adaptively selected stress-test pairs rather than a representative pair sample, and lose all power on the smaller CelebA test sets under the global 40-pair correction. A one-sided MMD concentration or sequential treatment remains future work. Neighborhood and deployment choices. Definition 1 treats identity-conditioned image sampling from Pid as mechanism randomness. It does not establish DP for a fixed userselected photograph or for releasing multiple outputs. In the FaceFusion audit the donor is fixed, so the reported values do not characterize privacy under a changing-donor neighborhood. Per-image, prompt, variable-donor, and datasetlevel neighborhoods require new mechanism definitions and, in some cases, different sampling and sensitivity analyses; the current numerical results do not transfer automatically. Representation dependence. Every conclusion is conditional on the selected encoders and preprocessing. Using two encoders is a robustness check, not proof that all identity information in the pixels is captured. Extension to another modality would require a new threat model, representation validation, and finite-sample study rather than a direct reuse of the reported numbers.

Across the eight encoder × dataset × generator conditions, the classifier and two-sample tests reveal substantial identity-distinguishing signal. Our finite-sample ROC protocol (Section 4.3) sets aside an untouched test split per identity: pair selection, classifier fitting, and threshold selection use only the development split, and the test split is reserved for one fixed evaluation. Evaluated on this reserved test split, the protocol certifies positive selected-pair ε lower bounds for all 20 VGGFace2 stress-test pairs under one simultaneous 95% confidence family; 15 exceed ε = 1. Its inability to certify the much smaller CelebA test sets despite high AUC makes the sample cost of finite-sample DP auditing explicit. The modeling diagnostics are consistent with the distinguishability conclusion, but their numerical values differ because of assumptions, aggregation, finitesample treatment, and ROC endpoint behavior, not because the experiments identify a tightest audit. G AUSS M ECH and KDE-LR are not formal joint-mechanism lower bounds here; the MMD-TV and broad out-of-fold ROC-HT values remain uncertified sample estimates, even though the ROCHT values produced by the separate reserved-test protocol of Section 4.3 are certified. A benchmark with a known privacy guarantee whose audited ROC keeps every operating point strictly inside the unit square, so that no point has [ = 0 or TPR [ = 1 and the hypothesis-testing plug-in FPR of Equation 15 returns finite values rather than the endpoint divergences observed here, is therefore the necessary next step for comparing calibration and tightness.

References [1]

H. Ruhs et al., “FaceFusion: Industry-leading face manipulation platform,” GitHub repository. [Online]. Available: https://github.com/ facefusion/facefusion. Accessed: Jun. 2026.

[2]

Q. Wang, X. Bai, H. Wang, Z. Qin, A. Chen, H. Li, X. Tang, and Y. Hu, “InstantID: Zero-shot identity-preserving generation in seconds,” arXiv:2401.07519, 2024.

[3]

C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of Cryptography Conference (TCC), 2006, pp. 265–284.

[4]

C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,” Foundations and Trends in Theoretical Computer Science, vol. 9, nos. 3–4, pp. 211–407, 2014.

[5]

D. Kifer and A. Machanavajjhala, “Pufferfish: A framework for mathematical privacy definitions,” ACM Trans. Database Syst., vol. 39, no. 1, Art. 3, 2014.

[6]

H. Xue, B. Liu, M. Ding, T. Zhu, D. Ye, L. Song, and W. Zhou, “DP-Image: Differential privacy for image data in feature space,” arXiv:2103.07073, 2021.

[7]

H. Shibata et al., “Practical medical image generation with provable privacy protection based on denoising diffusion probabilistic models for high-resolution volumetric images,” Applied Sciences, vol. 14, no. 8, Art. 3489, 2024.

[8]

P. Kairouz, S. Oh, and P. Viswanath, “The composition theorem for differential privacy,” in Int. Conf. Machine Learning (ICML), 2015, pp. 1376–1385.

[9]

J. Dong, A. Roth, and W. J. Su, “Gaussian differential privacy,” J. Royal Statistical Society, Series B, vol. 84, no. 1, pp. 3–37, 2022.

9. Conclusion We presented a comparative study of four empirical differential privacy audits applicable to pre-trained, blackbox image-to-image face generators: G AUSS M ECH, KDELR, MMD-TV, and ROC-HT. For each audit we made explicit its definition, assumptions, hyperparameter dependence, finite-sample status, and route from an embeddingspace measurement to an ε diagnostic or population lower bound. We documented the MMD-TV pure-DP specialization for positive-definite kernels satisfying k(x, x) ≤ 1, aligned the empirical comparison on common identity pairs, and added a development-selected, reserved-test ROC-HT certificate with simultaneous exact confidence control.

[10] M. Jagielski, J. Ullman, and A. Oprea, “Auditing differentially private machine learning: How private is private SGD?” in NeurIPS, 2020.

[11] M. Nasr, S. Song, A. Thakurta, N. Papernot, and N. Carlini, “Adversary instantiation: Lower bounds for differentially private machine learning,” in IEEE S&P, 2021, pp. 866–882. [12] T. Steinke, M. Nasr, and M. Jagielski, “Privacy auditing with one (1) training run,” in NeurIPS, 2023. [13] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A kernel two-sample test,” J. Machine Learning Research, vol. 13, pp. 723–773, 2012. [14] C. Domingo-Enrich and Y. Mroueh, “Auditing differential privacy in high dimensions with the kernel quantum Rényi divergence,” arXiv:2205.13941, 2022. [15] W. Kong, A. Muñoz Medina, M. Ribero, and U. Syed, “DPAuditorium: A large scale library for auditing differential privacy,” in IEEE Symp. Security and Privacy, 2024, pp. 1211–1226. [16] T. González Lara, M. Dulce Rubio, A. Ramdas, and M. Ribero, “Sequentially auditing differential privacy,” in NeurIPS, 2025. [17] B. K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, and G. R. G. Lanckriet, “Hilbert space embeddings and metrics on probability measures,” J. Machine Learning Research, vol. 11, pp. 1517–1561, 2010. [18] E. Ghazi and I. Issa, “Total variation meets differential privacy,” IEEE J. Sel. Areas Inf. Theory, vol. 5, pp. 207-220, Jun. 2024. [19] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “ArcFace: Additive angular margin loss for deep face recognition,” in CVPR, 2019, pp. 4685– 4694. [20] J. Deng et al., “InsightFace: 2D and 3D face analysis project,” GitHub repository. [Online]. Available: https://github.com/deepinsight/insightface. Accessed: Jun. 2026. [21] T. Esler, “facenet-pytorch: Pretrained PyTorch face recognition models,” GitHub repository. [Online]. Available: https://github.com/ timesler/facenet-pytorch. Accessed: Jun. 2026. [22] S. S. Shapiro and M. B. Wilk, “An analysis of variance test for normality (complete samples),” Biometrika, vol. 52, nos. 3–4, pp. 591– 611, 1965. [23] B. W. Silverman, Density Estimation for Statistics and Data Analysis. London: Chapman & Hall, 1986. [24] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “VGGFace2: A dataset for recognising faces across pose and age,” in IEEE FG, 2018, pp. 67–74. [25] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in IEEE ICCV, 2015, pp. 3730–3738. [26] B. Phipson and G. K. Smyth, “Permutation P -values should never be zero: calculating exact P -values when permutations are randomly drawn,” Statistical Applications in Genetics and Molecular Biology, vol. 9, no. 1, 2010. [27] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Machine Learning Research, vol. 12, pp. 2825–2830, 2011. [28] C. J. Clopper and E. S. Pearson, “The use of confidence or fiducial limits illustrated in the case of the binomial,” Biometrika, vol. 26, no. 4, pp. 404–413, 1934. [29] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001. [30] B. Balle and Y.-X. Wang, “Improving the Gaussian mechanism for differential privacy: Analytical calibration and optimal denoising,” in Proc. 35th Int. Conf. Machine Learning, PMLR, vol. 80, 2018, pp. 394–403. [31] N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramèr, B. Balle, D. Ippolito, and E. Wallace, “Extracting training data from diffusion models,” in USENIX Security Symposium, 2023. [32] H. Hu and J. Pang, “Membership inference of diffusion models,” arXiv:2301.09956, 2023. [33] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in ACM CCS, 2016, pp. 308–318.

Related documents

• Onyx: Cost-Efficient Disk-Oblivious ANN Search#123957
• Better Privacy Guarantees for Larger Groups#373340
• Privacy Leakage from a Thousand Words: Millipixel Location Recovery from Dot Maps#667926
• Bit-ViP: Leveraging Bit-planes to Preserve Visual Privacy in Images through Obfuscation#321765
• Privacy Icons:#400411