arXiv:2605.27292v1 [cs.LG] 26 May 2026
Detectability in Diversity: Improved Canary Crafting for Privacy Auditing in One Run
Mathieu Dagréou PreMeDICaL team, Inria Idesp, Inserm, Univ. de Montpellier [email protected]
Aurélien Bellet PreMeDICaL team, Inria Idesp, Inserm, Univ. de Montpellier [email protected]
Abstract Privacy auditing aims to empirically assess privacy leakage in machine learning models using membership inference attacks (MIAs), and to derive lower bounds on differential privacy (DP) parameters. Recent one-run auditing methods address the high cost of standard approaches by relying on a single training run with multiple "canary" points whose inclusion or exclusion must be detected by the auditor. In this work, we study the problem of efficiently crafting canaries for one-run privacy auditing. Motivated by recent theoretical insights suggesting that interference between canaries contributes to weaker leakage estimates compared to multi-run methods, we propose to optimize canaries to be both highly detectable and minimally interfering. Our approach combines a greedy initialization based on influence functions with a bilevel optimization procedure that maximizes distinguishability while promoting diversity in embedding space, enabling the use of computationally efficient bilevel algorithms. Experiments show that our method achieves stronger privacy leakage estimates at a lower computational cost than existing canary crafting approaches.
1
Introduction
Machine learning models are known to leak information about their training data [4, 8, 9, 28, 44], motivating the need for privacy auditing. Membership Inference Attacks (MIAs) [9, 30, 52, 57, 60] probe how much a machine learning model has memorized individual training points: given a data point z and access to a trained model, an adversary attempts to infer whether z was a member of the training set, typically by constructing a confidence score derived from the model’s outputs (e.g., based on the loss value at z). MIAs serve as an empirical measure of privacy leakage, as they quantify the minimal information a model can memorize about a training sample.MIAs are also used to audit formal Differential Privacy (DP) [20] guarantees. DP has become the standard framework for training models with provable privacy guarantees, as it bounds the success probability of any MIA [43]. For large-scale, non-convex optimization, Differentially Private Stochastic Gradient Descent (DP-SGD) [1, 5] is the de facto approach, which enforces privacy by clipping per-example gradients and adding Gaussian noise during training to limit the influence of any single data point. However, the theoretical privacy analysis of DP-SGD provides only worst-case upper bounds on the privacy parameters, which can be overly pessimistic in practice. DP auditing [12, 15, 31, 43, 45] complements this analysis by using MIAs to derive an empirical lower bound on the privacy parameters: the better an adversary distinguishes members from non-members, the stronger the lower bound on the actual privacy loss. A standard component of MIA evaluation and DP auditing consists of inserting "canary" data points into the training set, intentionally designed to maximize detectable leakage, and measuring how accurately an adversary can infer whether a given canary was included in or excluded from the training data. The usual approach randomizes the inclusion of a single canary point at a time and repeats independent model training thousands of times to obtain statistically reliable estimates. To reduce this cost, Preprint.
Steinke et al. [54] proposed the one-run auditing paradigm for DP, where a single model is trained on a dataset in which m canaries are independently included with probability 1/2. The adversary then tries to infer, from the trained model, which canary were included. This framework was recently extended to the broader MIA setting [21], where the model need not satisfy DP and performance is measured using standard MIA metrics, such as the True Positive Rate (TPR) at low False Positive Rate (FPR) [10] or the Area Under the ROC Curve (AUC). In the black-box setting—where the adversary inserts sample canaries and observes only the final model’s outputs—there is still a gap between the upper bound provided by accounting and the lower bound estimated by auditing [43, 46]. This looseness is amplified in the one-run paradigm which underperforms its multi-run counterparts [33, 55]. Several works [21, 33, 55] attribute this gap to interference between canaries: the inclusion or exclusion of a canary z ′ can affect the MIA confidence score of another canary z, thereby degrading the adversary’s ability to infer the membership of z. This interference is particularly strong when canaries are similar, since nearby points in representation space tend to produce correlated effects on the model; in such cases, including z ′ can reduce the model’s sensitivity to z and make its membership harder to detect. Bridging the gap with multi-run auditing therefore requires canaries that are both detectable–i.e., strongly memorized by the model in isolation–and diverse in representation space, so that their MIA scores remain approximately independent. Existing canary crafting methods [6, 31, 46, 56] do not explicitly enforce diversity. Contributions. In this work, we build upon the recent observations of [21, 33, 55] to craft better canaries for one-run auditing by leveraging influence functions and bilevel optimization. Specifically: 1. We design a greedy selection algorithm based on influence functions that identifies training points with high self-influence (strong memorability) and low pairwise cross-influence (minimal mutual interference), providing a strong initialization at low computational cost. 2. We introduce a bilevel optimization formulation with an explicit diversity regularizer promoting orthogonality in representation space. Our formulation requires maintaining only a single model across canary updates, enabling an efficient bilevel algorithm that updates the model incrementally rather than retraining from scratch at each outer step. 3. We empirically validate our approach with WideResNet and CNN architectures, showing competitive or stronger privacy leakage estimates compared to existing canary crafting methods, while requiring substantially lower computational cost. Related work on canary crafting. Heuristic canary crafting methods are routinely used, such as flipping the label of a sample or using a blank image [46]. The seminal work of Jagielski et al. [31] introduced the ClipBKD technique, which optimizes the canary for multi-run auditing by aligning it with the direction of lowest singular value of the samples’ covariance matrix. Several follow-up works also operate primarily in the multi-run setting. Lu et al. [36] use influence functions to design canaries that maximize their effect on model parameters, but restrict their analysis to logistic regression. More recently, Yaghini et al. [56] leverage bilevel optimization for canary design, but it is unclear how to extend their framework to one-run auditing. Recent work by Boglioni et al. [6] is the closest to our setting as it explicitly targets the one-run regime. However, their approach relies on costly retrainingbased optimization (see the discussion in Section 4.2) and does not include an explicit mechanism to enforce canary diversity, limiting scalability and effectiveness. A slightly orthogonal line of work considers a different threat model where the auditor injects gradients rather than data canaries. In this setting, Cebere et al. [11] show that carefully crafted gradients can achieve tight auditing in hidden state models, and Maddock et al. [38] study gradient-based canaries in federated learning.
2
Background
2.1
Membership inference attacks
Membership Inference Attacks (MIAs) [9, 52] are privacy attacks in which an adversary attempts to infer whether a given data point z was part of the training set of a model A(D) trained on dataset D. The adversary computes a score s(A(D), z), for instance based on predicted probabilities or loss values, and thresholds it to produce a binary guess. MIAs are classically formulated as a hypothesis test where the null hypothesis is that z ∈ / D, yielding standard performance metrics such as the Area 2
Under the Curve (AUC) or the True Positive Rate (TPR) at low False Positive Rate (FPR) [9]: the higher these metrics, the stronger the MIA. 2.2
Differential private machine learning
Differential Privacy (DP) is the standard framework to quantify the privacy guarantees of randomized algorithms [19, 20]. It ensures that the algorithm’s output distribution does not change significantly when a single input data point is added or removed. Definition 2.1. A randomized algorithm A : D → O satisfy (ϵ, δ)-DP if for any datasets D, D′ ∈ D such that D′ = D ∪ {z}, and for any set O ⊂ O, we have P[A(D) ∈ O] ≤ eϵ P[A(D′ ) ∈ O] + δ. The parameter ϵ controls the strength of the guarantee, with smaller values providing stronger privacy, while δ should be a small constant that allows a negligible probability of failure. DP enjoys several desirable properties: it is composable (privacy guarantees degrade gracefully under multiple analyses) and robust to post-processing. Importantly, DP also provably limits the success of MIAs, ensuring that an observer cannot reliably determine whether any individual’s data is included in the input from the algorithm’s output [43]. This fundamental link between DP and MIAs forms the basis of DP auditing (see Section 2.3). The standard approach for training machine learning models with differential privacy at scale is DP-SGD [1, 5, 53]. It extends standard SGD by first clipping individual gradients to bound their sensitivity, then averaging them and adding Gaussian noise to the update direction. Starting from an initialization θ0 ∈ Rp , DP-SGD iteratively applies the following update rule: h 1 X i θt+1 = θt − η ClipC [∇θ ℓ(θt , z)] + Z , |Bt | z∈Bt
C )v is the clipping where Z ∼ N (0, C 2 σ 2 Ip ), Bt ⊂ D is a batch of samples, ClipC [v] = min(1, ∥v∥ operator with C > 0 the clipping threshold, and η > 0 is the step size. The privacy guarantees of DP-SGD result from the sequential composition of subsampled Gaussian mechanisms applied at each iteration. To accurately track the cumulative privacy loss over multiple iterations, one relies on DP variants with tight composition properties, such as Rényi Differential Privacy [42], or on numerical composition methods [18, 25, 35]. However, these analyses only provide upper bounds for the privacy guarantees of DP-SGD, which can be loose in practice, motivating the need for empirical evaluation of the privacy parameters.
2.3
Privacy auditing in one run
In one-run auditing [54], the auditor has a set Algorithm 1 Auditing in one run [54] C = {z1 , . . . , zm } of canaries. A selection 1: Input: Dataset D, audited algorithm A, score S = {zi : Ai = 1} is built by drawing function s, canaries C = {z1 , . . . , zm } Ai ∼ Bernoulli(1/2) independently for each 2: Draw A ∼ Bernoulli(1/2)⊗m . canary. The model is then trained on D ∪ S, 3: Compute θ = A(D ∪ {zi : Ai = 1}) yielding model parameters θ. A MIA score 4: for i = 1, . . . , m do s(θ, z) is computed for each canary z ∈ C; this 5: Compute the score ŝi = s(θ, zi ) function should be such that s(θ, z) is high when 6: end for z ∈ S and low otherwise.In our experiments, we use the loss-based score s(θ, z) = −ℓ(θ, z), 7: if MIA evaluation then which is a simple and commonly used choice. 8: metric = TPR@α-FPR (ŝ, A) The procedure is summarized in Algorithm 1. 9: else if DP auditing then These scores alongside the ground-truth mem- 10: metric = ϵ̂(ŝ, A) berships can then be used for both MIA evalu- 11: end if ation and DP auditing. For MIA evaluation, we 12: Output: metric compute standard MIA metrics, such as the TPR at low FPR [10], which is given by P P ≥ τ ] Ai ≥ τ ] (1 − Ai ) i 1[ŝ i 1[ŝ Pi Pi TPR@α (ŝ, A) = max : ≤α . τ i Ai i (1 − Ai ) For DP auditing, following Steinke et al. [54], we can construct a lower bound of the privacy parameter ϵ. To do so, the k+ highest scores are guessed to have been included in the training set, while the k− lowest scores are guessed to have been excluded. From these statistics, the auditor can either directly estimate a lower bound on ϵ [54], or estimate the Gaussian Differential Privacy (GDP [17]) parameter for a tighter bound [39, 46]. 3
2.4
Influence functions
Influence functions aim at measuring the effect of upweighting a data sample on a model. They were first introduced in robust statistics [29] and then applied in machine learning for various purposes such as explainability [34], or to study the memorization phenomenon in neural networks [22, 41, 61] thanks to the notion of cross and self-influence presented in Section 3.1. Recently, the computational cost of influence functions has been reduced so that they can be used with large language models [3, 27]. Consider a dataset D = {z1 , . . . , zn }, a loss function ℓ and the solution of the Empirical Risk Pn Minimization (ERM) problem θ∗ (D) = arg minθ L(θ, D), where L(θ, D) ≜ n1 i=1 ℓ(θ, zi ). For Pn α > 0 and a data point z, we denote θz∗ (α; D) the model obtained by solving minθ n1 i=1 ℓ(θ, zi ) + αℓ(θ, z). Since θ∗ (D) = θz∗ (0; D), the difference θz∗ (α; D) − θ∗ (D) can be approximated by the Jacobian α∂α θz∗ (0; D). Under strong convexity of the loss function ℓ, the implicit function theorem −1 yields the influence function of z: ∂α θz∗ (0; D) = − ∇2 L(θ∗ (D), D) ∇ℓ(θ∗ (D), z). Then, for a measurement function f , substituting the influence function into a first-order Taylor expansion yields f (θz∗ (α; D)) ≈ f (θ∗ (D)) + α∇f (θ∗ (D))⊤ ∂α θz∗ (0; D).
3
Canary selection by influence optimization
The work of Yaghini et al. [56] in the multi-run setting highlights the importance of initialization in gradient-based canary crafting. Inspired by their work, we propose to leverage influence functions to find points in the dataset that are at the same time memorizable and have low mutual interference. 3.1
Cross- and self-influence on MIA scores
Consider a MIA score function s such that s(θ∗ (D), z) is higher when z ∈ D. To quantify memorizability and interference, we track how the score of z changes when a point z ′ is removed from D. Applying the first-order approximation from Section 2.4 with α = −1/|D| gives 1 s(θ∗ (D), z) − s(θ∗ (D \ {z ′ }), z) ≈ |D| Is (z, z ′ ), with Is (z, z ′ ) defined in Definition 3.1. Definition 3.1 (Self- and cross-influence). The influence of z ′ on the score of z is Is (z, z ′ ) ≜ ∇θ s(θ∗ (D), z)⊤ ∂α θz∗′ (0; D). When z ′ = z, Is (z, z) is the self-influence of z (memorizability proxy); when z ′ ̸= z, Is (z, z ′ ) is the cross-influence of z ′ on z (interference proxy). Remark. Under strongly-convex ERM and the loss-based score s(θ, z) = −ℓ(θ, z), both reduce to Is (z, z ′ ) = ∇ℓ(θ∗ (D), z)⊤ [∇2 L(θ∗ (D), D)]−1 ∇ℓ(θ∗ (D), z ′ ). Self-influence Is (z, z) ≥ 0 recovers the intuition that removing z increases its loss; a positive cross-influence Is (z, z ′ ) > 0 means removing z ′ also raises the score of z, creating interference. In the one-run auditing setting (Algorithm 1), a large Is (z, z ′ ) means that the inclusion or exclusion of z ′ shifts the score of z, making it harder to correctly classify z. This motivates selecting canaries with low pairwise cross-influence. 3.2
Greedy search of canary set
In this section, we introduce the first step of our canary-crafting approach: selecting strong initial canary candidates from the training set D. These candidates are subsequently used to initialize a gradient-based refinement phase (Section 4). Our selection strategy leverages both self-influence and cross-influence to identify a set C = {z1 , . . . , zm } ⊂ D whose elements exhibit high memorization while minimizing mutual interference. Specifically, we require that the inclusion of each canary z ∈ C can be reliably inferred from its MIA score s(θ, z), and that this score remains largely insensitive to the presence or absence of any other canary z ′ ∈ C, with z ′ ̸= z. To achieve this, we propose to construct a set C where each element has high self-influence and low cross-influence with all other selected canaries. Formally, for each z ∈ C, we want the ratio max ′ Is (z,z)Is (z′ ,z) z ∈C\{z} to be as large as possible. This leads to the following combinatorial optimization problem: max f (C) =
C⊂D |C|=m
X z∈C
Is (z, z) . maxz′ ∈C\{z} Is (z ′ , z)
4
(1)
Algorithm 2 Greedy canary selection Solving (1) exactly requires evaluating |D| m 1: Input: Dataset D, number of canaries m, number candidate subsets, which is computationally of preselected canaries p infeasible. Moreover, since f is not submodu2: Let D ⊂ D be the p samples with highest selflar, no efficient algorithm with approximation influence. guarantees is known. We therefore adopt a greedy heuristic [47], summarized in Algo- 3: Set C1 = {arg maxz∈D Is (z, z)} rithm 2. The procedure begins with a prese- 4: for k = 1, . . . , m − 1 do lection step: we construct a subset D ⊂ D 5: Set zk+1 = arg maxz∈D\Ck max ′Is (z,z) ′ z ∈Ck Is (z ,z) containing the p samples with highest selfinfluence. This reduces the number of re- 6: Update Ck+1 = Ck ∪ {zk+1 } quired cross-influence evaluations. We then 7: end for build C iteratively. Given a partial set Ck of 8: Output: Cm size k, the next canary is chosen by maximizing, over z ∈ D \ Ck , the ratio max ′Is (z,z) Is (z ′ ,z) . This requires computing and storing all pairwise z ∈Ck
self- and cross-influences within D, resulting in a p×p score matrix. The computation of the influence functions which involves inverse Hessian-vector products can be made more efficient in large scale setting by leveraging K-FAC approximation of the Hessian [3, 40]. As we will see experimentally, this procedure yields canary sets within the training data that exhibit significantly higher privacy leakage than randomly selected samples. Such in-distribution canaries are particularly useful when the goal is to estimate the empirical privacy leakage of observed data points. However, when the goal is to assess worst-case leakage (e.g., in DP auditing), one must instead consider worst-case canaries, which may lie outside the training set. This motivates the gradient-based refinement step introduced next.
4
Gradient-based refinement of canaries
In this section, we design a bilevel objective function to craft canaries that are both highly memorizable and exhibit low mutual interference. The set of canaries selected using the influence-based procedure described in the previous section serves as an initialization for this optimization. 4.1
Warm-up: canary interference in least squares regression
To build intuition, we first consider the simplest setting: least squares regression with two canaries. Least squares regression has the advantage of admitting a closed-form solution. Let D = {zi = (xi , yi )}1≤i≤n denote the training set, where xi ∈ Rd represents input features and yi ∈ R the corresponding label. Denoting X = (x1 , . . . , xn )⊤ ∈ Rn×d the design matrix and y = (y1 , . . . , yn )⊤ ∈ Rn the target vector, least squares regression seeks θ∗ (X, y) ∈ arg minθ∈Rd 21 ∥Xθ − y∥2 . When n ≥ d and X has full rank, the solution is unique and given by θ∗ (X, y) = (X ⊤X)−1X ⊤ y. Consider now two canaries zc,1 = (xc,1 , yc,1 ) and zc,2 = (xc,2 , yc,2 ). X y We define X̃i = , and ỹi = . Let ℓ(θ, zi ) = 12 (yi − θ⊤ xi )2 denote the least square loss. xc,i yc,i We are interested in quantifying how the inclusion of zc,1 in the training set affects the loss evaluated at zc,2 , i.e., how ℓ(θ∗ (X̃1 , ỹ1 ), zc,2 ) differs from ℓ(θ∗ (X, y), zc,2 ). The following proposition characterizes this effect. Proposition 4.1. Assume that n ≥ d and that the matrices X and X̃1 have rank d. Then we have " #2 ⊤ −1 x K x c,2 c,1 ℓ(θ∗ (X̃1 ,ỹ1 ), zc,2 ) − ℓ(θ∗ (X, y), zc,2 ) = ℓ(θ∗ (X, y), xc,1 , yc,1 ) −1 x 1 + x⊤ c,1 c,1 K − (⟨θ∗ (X, y), xc,1 ⟩ − yc,1 )(⟨θ∗ (X, y), xc,2 ⟩ − yc,2 )
−1 x⊤ xc,2 c,1 K , ⊤ −1 1 + xc,1 K xc,1
(2)
where K = X ⊤ X is the empirical covariance matrix. We provide the proof in Section A.1. Equation (2) shows that a sufficient condition for the two canaries not to interfere is orthogonality with respect to the inverse empirical covariance matrix. This observation motivates incorporating an orthogonality constraint into our bilevel objective. While this derivation is specific to the least squares setting in input space, it suggests a more general principle: 5
we will promote orthogonality in the representation space induced by the model, rather than in raw input space. 4.2
Canary optimization as a bilevel optimization problem
Unlike Section 3, where canaries are selected from a fixed dataset D, we now treat canary samples as continuous variables–keeping labels fixed–and show how to optimize them via gradient-based methods. To clarify the differences with prior work on canary crafting for one-run auditing [6], we first revisit their formulation. They propose to learn canary sets by updating the canaries using gradient steps computed by differentiating through SGD updates in a non-private training loop. This approach can be interpreted as approximately solving a bilevel optimization problem via iterative differentiation [7, 16, 37]. Throughout this section, we slightly abuse notation and write θ∗ ≜ θ∗ (D) ∗ and θS∗ ≜ θ∗ (D ∪ S) for any set S, with the special case θC ≜ θ∗ (D ∪ C). The objective of Boglioni et al. [6] can then be written as: X X min Ψ(C) ≜ ES s(θS∗ , z) − s(θS∗ , z) s.t. θS∗ ∈ arg min L(θ, D ∪ S), ∀S ⊂ C, (3) C
θ∈Θ
z∈S
z∈C\S
where the expectation is taken with respect to the random subset S, constructed by including each canary independently with probability 1/2. Intuitively, minimizing Ψ encourages canary sets such that, on average over subsets S, included canaries yield high scores under the trained model while excluded canaries yield low scores. However, this formulation is computationally expensive, as it requires retraining a model for each sampled subset S, since each subset corresponds to a different training set defined by a particular configuration of included canaries. This makes the approach impractical for efficient auditing. To address this limitation, we introduce a more tractable bilevel formulation. The canaries obtained via Algorithm 2 already exhibit low pairwise crossinfluence by construction, implying that for any canary z ∈ C and subset S ⊂ C, the score s(θS∗ , z) is approximately determined only by whether z ∈ S. Under this approximation, it suffices to consider P the full inclusion∗case S = C, yielding a simplified distinguishability objective of the form z∈C s(θ∗ , z) − s(θC , z). To preserve the low-interference structure during gradient updates, we further introduce a diversity-promoting regularizer. R(C). This leads to the following bilevel problem ∗ X θ ∈ arg minθ∈Θ L(θ, D), ∗ ∗ min Φ(C) ≜ s(θ , z) − s(θC , z) + Rθ∗ (C) s.t. (4) ∗ C θC ∈ arg minθ∈Θ L(θ, D ∪ C). z∈C
Building on the intuition from Section 4.1, we define P the regularizer in terms of representation-space orthogonality. Specifically, we set Rθ∗ (C) = λ z,z′ ∈C ⟨φθ∗ (z), φθ∗ (z ′ )⟩2 , where λ > 0 controls z̸=z ′ d′
the regularization strength and φ : R → R maps each sample to its embedding under θ∗ . Using ∗ θ∗ rather than θC decouples the regularizer from the inner optimization problem, avoiding additional complexity while yielding satisfying empirical results. In the bilevel problem (4), θ∗ is independent ∗ of C and can therefore be computed once in advance. A key advantage over (3) is that only θC ∗ depends on the canary set, enabling amortized bilevel schemes [2, 32], where the inner solution θC is updated incrementally as the canary set C evolves rather than recomputed from scratch between successive updates. In contrast, an amortized treatment of (3) would require maintaining θS∗ for all subsets S ⊂ C, which is computationally intractable and requires prohibitive memory usage. θ∗
4.3
d
Efficient gradient-based canary crafting
We now introduce an efficient algorithm to solve the bilevel optimization problem defined in the previous section. The optimization is initialized using the canary set obtained from Section 3, and then refined via gradient-based updates. As labels are kept fixed throughout training, only the input samples are optimized. For a canary set C = {(x1 , y1 ), . . . , (xm , ym )}, we therefore denote Cx = {x1 , . . . , xm } the set of optimized canary variables. To perform gradient-based optimization of Φ, we require its gradient with respect to Cx , denoted by ∇Cx Φ, which is referred to as the hypergradient in the bilevel optimization literature. Proposition 4.2 provides an explicit expression for ∇Cx Φ for the case where the inner problem is strongly convex. 6
Proposition 4.2. Assume that the score function s is differentiable w.r.t. both θ and the input sample x, and that the loss function ℓ is twice differentiable, strongly convex in θ and differentiable in x. Then Φ is differentiable and, letting n ≜ |D| + |C|, its gradient is given by X ∗ ∗ , x, y)v ∗ (C) + ∇Cx Rθ∗ (C) ∇Cx Φ(C) = ∇x s(θ∗ , x, y) − ∇x s(θC , x, y) + n1 ∇2x,θ ℓ(θC z∈C z=(x,y) ∗ where v ∗ (C) is the solution of the linear system ∇2θ,θ L(θC , D ∪ C)v =
∗ z∈C ∇θ s(θC , z).
P
This result is the standard implicit differentiation result; the proof is provided in Section A.2. While Proposition 4.2 provides an exact expression, directly using it to perform gradient descent ∗ is intractable in large-scale scenarios. Indeed, it requires repeatedly recomputing θC as the canary set evolves, effectively retraining the model at each update, as well as solving a high-dimensional linear system to obtain v ∗ (C). To address this, we rely on Approximate Implicit Differentiation ∗ (AID) [24, 50], which replaces θC and v ∗ (C) with estimates updated during optimization. Although AID is theoretically grounded in settings with strongly convex inner problems, it has also been successfully applied in non-convex regimes [26].We instantiate this approach via an adaptation of the SOBA algorithm [13]. SOBA maintains three coupled variables θ, v and Cx . At iteration t, the iterates θt and vt approximate θ∗ (D ∪ Ct ) and v ∗ (Ct ) respectively. They are updated using stochastic gradient steps to track their respective equilibria as the canary set evolves, while Cx is updated via an approximate hypergradient step. The resulting recursion is: θt+1 = θt − η∇θ L(θt , Bt ) h i X vt+1 = vt − η ∇2θ,θ L(θt , Bt )vt + ∇θ s(θt , z)
(5) (6)
z∈Ct
Cx,t+1 = Cx,t − ρ
hX z∈Ct
i 1 2 ∇x,θ ℓ(θt , z)vt + ∇x s(θ∗ , z) − ∇x s(θt , z) + ∇Cx Rθ∗ (Ct ) n
(7)
where Bt ⊂ D ∪ Ct is a mini-batch of samples. Note that (7) coincides with the exact hypergradient ∗ update when θt = θC and vt = v ∗ (Ct ). Moreover, although the recursion involves Hessian-vector t products, these can be computed efficiently without explicitly forming the Hessian using the Pearlmutter trick [14, 49]. The main advantage of (5)-(7) is that it avoids full retraining at each canary update. Instead, the model parameters θ evolve jointly with the canary optimization. In practice, ∗ we initialize θ0 and v 0 by approximately computing θC and v ∗ (C0 ), ensuring that for small enough 0 ∗ ∗ step sizes, we keep θt ≈ θC and v ≈ v (C ) throughout optimization. The convergence of SOBA t t t towards a stationary point of Φ is proved in [13, Theorem 1 and 2] in the case where the inner function is strongly convex. Notably, it enjoys similar convergence guarantees as non-convex SGD [23].
7
4.4
IBIS — Influence-Bilevel canary crafting with Interference Suppression
Our full procedure, which we call IBIS, is summarized in Algorithm 3. It combines the Algorithm 3 IBIS: Influence-Bilevel canary crafting two contributions of this paper into a unified with Interference Suppression 1: Input: Dataset D, score s, m, p, λ, η, ρ, T pipeline: the influence-based greedy selection 2: Get C0 via Algorithm 2; set D ← D \ C0 . (Algorithm 2) provides an initialization 3: Compute θ ∗ ≈ arg minθ L(θ, D); initialize with strong detectability and low pairwise θ0 ≈ θ∗ (D ∪ C0 ), v0 ≈ v ∗ (C0 ). interference, and the bilevel updates (5)–(7) 4: for t = 0, . . . , T − 1 do further refine the canaries while preserving their diversity through the regularizer Rθ∗ . 5: Sample Bt ⊂ D ∪ Ct ; 6: Update θt+1 via (5). In terms of global computational cost, this requires training the model three times in total: 7: Update vt+1 via (6). 8: Update Cx,t+1 via (7). once for the influence-based selection, and ∗ ∗ 9: end for twice to initialize θ and θ0 ≈ θ (D ∪ C0 ) before the bilevel loop, which is significantly 10: Output: CT . cheaper than the approach of Boglioni et al. [6]. This reduced cost gives the auditor more flexibility in the choice of crafting architecture, and as we show in Section 5.2, using the audited model’s own architecture to generate canaries significantly improves performance.
5
Experiments
5.1
Ablation study
5.2
Evaluation
TPR at 0.05 FPR
Random Influence-based We first conduct an ablation study on the 1.00 2.0 WRN16-4 [59] architecture with the CI0.75 1.5 FAR10 dataset to evaluate the improvement 0.50 1.0 brought by our influence-based preselec0.5 0.25 tion and our orthogonality regularization. 0.0 0.00 We select a set of m = 1000 canaries Init. No reg. With reg. Init. No reg. With reg. from the dataset either randomly or with the influence-based approach introduced in Figure 1: Ablation study on WRN16-4/CIFAR10 (6 Section 3. These canaries are then used as runs per boxplot). Left: TPR @ 0.05 FPR, non-private initialization for the bilevel algorithm, with model. Right: Estimated ϵ for DP-SGD with ϵ = 10. or without regularization. We compute the MIA scores after private and non-private training. The results are displayed in Figure 1. In both the private and non-private settings, influence-based selection yields better performance than random selection even before running the bilevel algorithm. Moreover, canaries found by IBIS are systematically better when initialized with the influence-based selection, demonstrating the effectiveness of the preselection step. We also observe, in the non-private case, that regularization helps IBIS find better canaries. In the private case, the unregularized version has slightly better results than the regularized one.
With our pipeline, it is possible to use a model that is different from the one that is audited. This is also the setup of [6], who train their canaries using a ResNet9 architecture to audit a WRN16-4 model. In Section C, we also evaluate our pipeline on a CNN architecture. To assess the transferability of the canaries between models and provide a fair comparison with [6], we generate canaries using either ResNet9 or WRN16-4 as the crafting model, and use them to audit WRN16-4. We compare our method against three baselines: canaries from Boglioni et al. [6], canaries randomly selected in the dataset and canaries randomly selected in the dataset with flipped labels.In Table 1, we report the [email protected] scores. When canaries are trained on ResNet9, regularized IBIS achieves comparable performance to [6], while unregularized IBIS yields weaker canaries. When canaries are trained on WRN16-4, regularized IBIS achieves near-perfect scores. For regularized IBIS, using the audited model architecture for canary generation therefore improves performance; unregularized IBIS remains weak regardless of the crafting model. Although Random + Flip and regularized IBIS trained on ResNet9 achieve high mean TPR scores, their standard deviations (0.34 and 0.31, respectively) indicate highly variable performance across runs, in contrast to regularized IBIS trained 8
on WRN16-4, which achieves perfect and consistent scores. In Table 2 we report the estimated ϵ̂ Table 1: [email protected] scores of one-run auditing on WRN16-4 on CIFAR10. For architecturedependent methods, canaries are optimized using either ResNet9 or WRN16-4 as the crafting model. We report mean, standard deviation, and median over 6 runs. IBIS no reg.
IBIS reg.
Boglioni et al. [6]
Random
Random + Flip
Craft. arch.
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
ResNet9 WRN16-4
0.34 ± 0.19 0.32 ± 0.35
0.29 0.16
0.84 ± 0.31 1.00 ± 0.00
1.00 1.00
0.82 ± 0.24 –
0.95 –
0.05 ± 0.01
0.05
0.86 ± 0.34
1.00
when performing one-run auditing. In each case, ϵ̂ is computed using the GDP in one-run auditing technique [39] since it is known to provide tighter results than the original one-run technique [54]. The results exhibit high variance across runs, reflecting the difficulty of the DP auditing setting. In the next section, we show that IBIS offers a clear advantage in terms of computational cost. Table 2: Auditing score estimate ϵ̂ of one-run auditing on WRN16-4 on CIFAR10. For architecturedependent methods, canaries are optimized using either ResNet9 or WRN16-4. We report mean, standard deviation, and median over 6 runs. IBIS no reg. ϵ 10 4 2 1
5.3
ResNet9 WRN16-4 ResNet9 WRN16-4 ResNet9 WRN16-4 ResNet9 WRN16-4
IBIS reg.
Boglioni et al. [6]
Random
Random + Flip
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
0.90 ± 0.26 0.99 ± 0.62 0.41 ± 0.43 0.28 ± 0.20 0.12 ± 0.09 0.18 ± 0.22 0.11 ± 0.11 0.00 ± 0.08
0.85 0.90 0.28 0.34 0.17 0.18 0.07 0.00
0.60 ± 0.22 0.38 ± 0.24 0.40 ± 0.18 0.32 ± 0.16 0.36 ± 0.20 0.14 ± 0.14 0.19 ± 0.21 0.18 ± 0.17
0.58 0.34 0.47 0.34 0.37 0.14 0.16 0.18
0.74 ± 0.48 – 0.41 ± 0.46 – 0.26 ± 0.14 – 0.24 ± 0.16 –
0.64 – 0.22 – 0.25 – 0.25 –
0.12 ±0.09
0.11
0.20 ±0.17
0.19
0.22 ±0.34
0.09
0.27 ±0.29
0.17
0.03 ±0.04
0.01
0.28 ±0.25
0.22
0.08 ±0.12
0.01
0.18 ±0.35
0.01
Computational cost of the canary generation
We report computational cost in Table 3. Using AID instead of backpropagating through full training trajectories reduces memory overhead substantially. Overall, generating 1000 canaries on ResNet9 with our approach requires less than three GPU hours on an NVIDIA A100, compared to 90–120 hours reported in [6]. This improvement stems from updating the model and canaries incrementally, rather than retraining from scratch at each optimization step. Table 3: Computational cost of each stage of our canary optimization pipeline on a NVIDIA A100. Influence selection provides the initialization and IBIS performs the bilevel optimization. We report GPU time (in hours) and peak memory (in GB) for each stage. ResNet9 WRN16-4
6
Time [h]
Memory [GB]
Time [h]
Memory [GB]
Influence selection Bilevel optim. step
0.33 2.17
10.20 25.28
0.21 1.33
15.65 27.25
IBIS
2.50
25.28
1.54
27.25
Conclusion
In this work, we focused on the canary crafting stage of one-run privacy auditing. Motivated by the fact that a key limitation of existing one-run approaches is interference between canaries, we proposed an influence-based method to select canaries that are both highly detectable and diverse. These canaries initialize a bilevel optimization procedure that further refines them to improve auditing strength. Overall, our results show clear improvements in one-run membership inference auditing. For DP privacy auditing, our method matches prior work while being significantly more computationally efficient. 9
Acknowledgments and Disclosure of Funding The authors thank Arielle Zhang, Mohammad Yaghini, and Nicolas Papernot for insightful discussion. This work is partially supported by grant ANR-22-PESN-0014 project of the Digital Health PEPR under the France 2030 program. Part of this work was performed using HPC resources from GENCI–IDRIS (Grant 2025-AD011016328). Part of the experiments presented were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www. grid5000.fr).
References [1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308–318, 2016. pages 1, 3 [2] M. Arbel and J. Mairal. Amortized Implicit Differentiation for Stochastic Bilevel Optimization. In International Conference on Learning Representations (ICLR), 2022. page 6 [3] J. Bae, N. Ng, A. Lo, M. Ghassemi, and R. Grosse. If Influence Functions are the Answer, Then What is the Question? In Advances in Neural Information Processing Systems (NeurIPS), 2022. pages 4, 5 [4] F. Barbero, X. Gu, C. A. Choquette-Choo, C. Sitawarin, M. Jagielski, I. Yona, P. Veličković, I. Shumailov, and J. Hayes. Extracting alignment data in open models. arXiv preprint arXiv:2510.18554, 2025. page 1 [5] R. Bassily, A. Smith, and A. Thakurta. Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 464–473, Philadelphia, PA, USA, 2014. IEEE. pages 1, 3 [6] M. Boglioni, T. Liu, A. Ilyas, and Z. S. Wu. Optimizing Canaries for Privacy Auditing with Metagradient Descent. In International Conference on Learning Representations (ICLR), 2026. pages 2, 6, 8, 9, 15 [7] J. Bolte, E. Pauwels, and S. Vaiter. One-step differentiation of iterative algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2023. page 6 [8] N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pages 267–284, 2019. page 1 [9] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel. Extracting Training Data from Large Language Models, 2021. pages 1, 2, 3 [10] N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramèr. Membership Inference Attacks From First Principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914, San Francisco, CA, USA, 2022. IEEE. pages 2, 3 [11] T. Cebere, A. Bellet, and N. Papernot. Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model. In International Conference on Learning Representations (ICLR), 2025. page 2 [12] T. Cebere, A. Bellet, and N. Papernot. Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model. In ICLR, 2025. page 1 [13] M. Dagréou, P. Ablin, S. Vaiter, and T. Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2022. page 7 [14] M. Dagréou, P. Ablin, S. Vaiter, and T. Moreau. How to compute Hessian-vector products? In ICLR Blogposts, 2024. page 7 10
[15] Z. Ding, Y. Wang, G. Wang, D. Zhang, and D. Kifer. Detecting violations of differential privacy. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 475–489, 2018. page 1 [16] J. Domke. Generic methods for optimization-based modeling. In Conference on Artificial Intelligence and Statistics (AISTATS), 2012. page 6 [17] J. Dong, A. Roth, and W. J. Su. Gaussian Differential Privacy. arXiv preprint arXiv:1905.02383, 2019. page 3 [18] V. Doroshenko, B. Ghazi, P. Kamath, R. Kumar, and P. Manurangsi. Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions, 2022. page 3 [19] C. Dwork and A. Roth. The Algorithmic Foundations of Differential Privacy. Foundations and Trends® in Theoretical Computer Science, 9(3-4):211–407, 2014. ISSN 1551-305X, 1551-3068. page 3 [20] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Theory of Cryptography, volume 3876, pages 265–284, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. pages 1, 3 [21] M. Even, C. Berenfeld, L. Bleistein, T. Cebere, J. Josse, and A. Bellet. Membership Inference Attacks from Causal Principles. arXiv preprint arXiv:2602.02819, 2026. page 2 [22] V. Feldman and C. Zhang. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2020–8202009. page 4 [23] S. Ghadimi and G. Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341–2368, 2013. page 7 [24] S. Ghadimi and M. Wang. Approximation Methods for Bilevel Programming. arXiv preprint arXiv:1802.02246, 2018. page 7 [25] S. Gopi, Y. T. Lee, and L. Wutschitz. Numerical Composition of Differential Privacy. In Journal of Privacy and Confidentiality, 2024. page 3 [26] D. Grangier, P. Ablin, and A. Hannun. Adaptive Training Distributions with Scalable Online Bilevel Optimization. Transactions on Machine Learning Research (TMLR), 2024. page 7 [27] R. Grosse, J. Bae, C. Anil, N. Elhage, A. Tamkin, A. Tajdini, B. Steiner, D. Li, E. Durmus, E. Perez, E. Hubinger, K. Lukošiūtė, K. Nguyen, N. Joseph, S. McCandlish, J. Kaplan, and S. R. Bowman. Studying Large Language Model Generalization with Influence Functions. arXiv preprint arXiv:2308.03296, 2023. pages 4, 16 [28] N. Haim, G. Vardi, G. Yehudai, O. Shamir, and M. Irani. Reconstructing Training Data from Trained Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), 2022. page 1 [29] F. R. Hampel. The Influence Curve and its Role in Robust Estimation. Journal of the American Statistical Association, 69(346):383–393, 1974. ISSN 0162-1459, 1537-274X. page 4 [30] J. Hayes, I. Shumailov, C. A. Choquette-Choo, M. Jagielski, G. Kaissis, K. Lee, M. Nasr, S. Ghalebikesabi, N. Mireshghallah, M. Sundaram Mutu Selva Annamalai, et al. Strong membership inference attacks on massive datasets and (moderately) large language models. arXiv e-prints, pages arXiv–2505, 2025. page 1 [31] M. Jagielski, J. Ullman, and A. Oprea. Auditing Differentially Private Machine Learning: How Private is Private SGD? In Advances in Neural Information Processing (NeurIPS), 2020. pages 1, 2 [32] K. Ji, J. Yang, and Y. Liang. Bilevel Optimization: Convergence Analysis and Enhanced Design. In International Conference on Machine Learning (ICML), 2021. page 6 11
[33] A. Keinan, M. Shenfeld, and K. Ligett. How Well Can Differential Privacy Be Audited in One Run? In Advances in Neural Information Processing Systems (NeurIPS), 2025. page 2 [34] P. W. Koh and P. Liang. Understanding Black-box Predictions via Influence Functions. In International Conference on Machine Learning (ICML), 2017. page 4 [35] A. Koskela, J. Jälkö, and A. Honkela. Computing Tight Differential Privacy Guarantees Using FFT. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020. page 3 [36] F. Lu, J. Munoz, M. Fuchs, T. LeBlond, E. Zaresky-Williams, E. Raff, F. Ferraro, and B. Testa. A General Framework for Auditing Differentially Private Machine Learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. page 2 [37] D. Maclaurin, D. Duvenaud, and R. P. Adams. Gradient-based Hyperparameter Optimization through Reversible Learning. In International Conference on Machine Learning (ICML), 2015. page 6 [38] S. Maddock, A. Sablayrolles, and P. Stock. CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learning. In International Conference on Learning Representations (ICLR), 2023. page 2 [39] S. Mahloujifar, L. Melis, and K. Chaudhuri. Auditing $f$-Differential Privacy in One Run. In International Conference on Machine Learning (ICML), 2025. pages 3, 9 [40] J. Martens and R. Grosse. Optimizing Neural Networks with Kronecker-factored Approximate Curvature. arXiv preprint arXiv:1503.05671, 2015. page 5 [41] M. Meeus, I. Shilov, G. Kaissis, and Y.-A. de Montjoye. Counterfactual Influence as a Distributional Quantity. In ICML Workshop MemFM, 2025. page 4 [42] I. Mironov. Rényi Differential Privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 263–275, Santa Barbara, CA, 2017. IEEE. page 3 [43] M. Nasr, S. Song, A. Thakurta, N. Papernot, and N. Carlini. Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning. In IEEE Symposium on Security and Privacy, 2021. pages 1, 2, 3 [44] M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035, 2023. page 1 [45] M. Nasr, J. Hayes, T. Steinke, B. Balle, F. Tramèr, M. Jagielski, N. Carlini, and A. Terzis. Tight auditing of differentially private machine learning. In 32nd USENIX Security Symposium (USENIX Security 23), pages 1631–1648, 2023. page 1 [46] M. Nasr, J. Hayes, T. Steinke, B. Balle, F. Tramèr, M. Jagielski, N. Carlini, and A. Terzis. Tight Auditing of Differentially Private Machine Learning. arXiv preprint arXiv:2302.07956, 2023. pages 2, 3 [47] C. H. Papadimitriou and K. Steiglitz. Combinatorial Optimization: Algorithms and Complexity. Prentice-Hall, Inc., USA, 1982. page 5 [48] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems (NeurIPS), 2019. page 16 [49] B. A. Pearlmutter. Fast Exact Multiplication by the Hessian. Neural Computation, 6(1):147–160, 1994. ISSN 0899-7667, 1530-888X. page 7 [50] F. Pedregosa. Hyperparameter optimization with approximate gradient. In International Conference on Machine Learning (ICML), 2016. page 7 12
[51] J. Sherman and W. J. Morrison. Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix. The Annals of Mathematical Statistics, 21(1):124–127, 1950. ISSN 0003-4851. page 14 [52] R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership Inference Attacks against Machine Learning Models. In IEEE Symposium on Security and Privacy (S&P)., 2017. pages 1, 2 [53] S. Song, K. Chaudhuri, and A. D. Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pages 245–248, Austin, TX, USA, 2013. IEEE. page 3 [54] T. Steinke, M. Nasr, and M. Jagielski. Privacy Auditing with One (1) Training Run. In Advances in Neural Information Processing Systems (NeurIPS), 2023. pages 2, 3, 9 [55] Z. Xiang, T. Wang, and D. Wang. Privacy Audit as Bits Transmission: (Im)possibilities for Audit by One Run. arXiv preprint arXiv:2501.17750, 2025. page 2 [56] M. Yaghini, M. Aerni, J. Zhang, F. Tramèr, and N. Papernot. OptiFluence: Scalable and Principled Design of Privacy Canaries. 2025. pages 2, 4 [57] S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE, 2018. page 1 [58] A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao, G. Cormode, and I. Mironov. Opacus: User-Friendly Differential Privacy Library in PyTorch. In Advances in Neural Information Processing Systems (NeurIPS), 2021. page 16 [59] S. Zagoruyko and N. Komodakis. Wide Residual Networks. arXiv preprint arXiv:1605.07146, 2016. page 8 [60] S. Zarifzadeh, P. Liu, and R. Shokri. Low-cost high-power membership inference attacks. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 2127, 2024. OpenReview.net, 2024. URL https://openreview.net/forum?id=sT7UJh5CTc. page 1 [61] C. Zhang, D. Ippolito, K. Lee, M. Jagielski, F. Tramèr, and N. Carlini. Counterfactual Memorization in Neural Language Models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. page 4
A
Proofs
A.1
Proposition 4.1
For convenience, we denote θ∗ ≜ θ∗ (X, y) and θ∗ (X̃1 , ỹ1 ) = θ1∗ . We have n n hX i−1 h X i −1 ⊤ ⊤ θ1∗ = X̃1⊤ X̃1 X̃1 ỹ1 = xi x⊤ + x x yi xi + yc,1 xc,1 c,1 c,1 i i=1
i=1
13
Let us denote K = X ⊤ X. By the Sherman-Morrison formula [51], we have −1 ⊤ θ1∗ = X̃1⊤ X̃1 X̃1 ỹ1 −1 K −1 xc,1 x⊤ c,1 K X̃1⊤ ỹ1 −1 x 1 + x⊤ c,1 c,1 K K −1 xc,1 x⊤ c,1 = I− K −1 X̃1⊤ ỹ1 −1 x 1 + x⊤ c,1 c,1 K K −1 xc,1 x⊤ c,1 (θ∗ + yc,1 K −1 xc,1 ) = I− −1 x 1 + x⊤ c,1 c,1 K
= K −1 −
K −1 xc,1 x⊤ c,1 (K −1 yc,1 xc,1 + θ∗ ) −1 x K 1 + x⊤ c,1 c,1 ! ⊤ −1 ∗ yc,1 xc K xc,1 + ⟨θ , xc,1 ⟩ K −1 xc,1 yc,1 − −1 x 1 + x⊤ c,1 c,1 K
= θ∗ + yc,1 K −1 xc,1 − = θ∗ + = θ∗ −
K −1 −1 x 1 + x⊤ c,1 c,1 K
(⟨θ∗ , xc,1 ⟩ − yc,1 )xc,1 .
Therefore we have ℓ(θ1∗ , z2 ) =
1 2 (⟨θ1∗ , x2 ⟩ − y2 ) 2
= ℓ(θ∗ , x2 , y2 ) + ℓ(θ∗ , x1 , y1 )
−1 x⊤ x2 1K ⊤ −1 1 + x 1 K x1
− (⟨θ∗ , x1 ⟩ − y1 )(⟨θ∗ , x2 ⟩ − y2 )
2
−1 x⊤ x2 1K ⊤ 1 + x1 K −1 x1
which yields the result. A.2
Proposition 4.2
Recall that the function Φ is defined as X ∗ Φ(C) = s(θ∗ , z) − s(θC , z) + Rθ∗ (C) z∈C ∗
∗ ∈ arg minθ∈Θ L(θ, D ∪ C). By the chain rule, we get: with θ ∈ arg minθ∈Θ L(θ, D) and θC X ∗ ∗ ∗ ∇Cx Φ(C) = ∇x s(θ∗ , z) − ∇x s(θC , z) + ∇Cx Rθ∗ (C) + (dθC )⊤∇θ s(θC , z) z∈C ∗ ∗ where dθC is the Jacobian of (Cx 7→ θC ). Since L( · , D ∪ C) and L( · , D ∪ C) are strongly convex, ∗ the value function Φ is well-defined. Moreover, for any canary sample C, θC verifies the following first order condition ∗ ∇θ L(θC , D ∪ C) = 0. (8) 2 ∗ The Hessian matrix ∇θ,θ L(θC , D ∪ C) is positive definite by strong convexity and since ℓ is differentiable with respect to the input samples Cx , by the Implicit Function Theorem, we can differentiate Equation (8) with respect to Cx to get ∗ ∗ ∗ ∇2θ,θ L(θC , D ∪ C)dθC + ∇2Cx ,θ L(θC , D ∪ C) = 0.
Rearranging yields the result.
B
Visualization of the canaries
While the images we get after the influence-based selection step are ensured to be in-distribution since they are picked from the dataset, it is interesting to ask whether the images still look like natural 14
images after the bilevel optimization algorithm. In Figure 2, we display some randomly chosen canaries crafted from the CIFAR10 dataset, using the WRN16-4 architecture. For the first three rows, for both the unregularized and the regularized cases the canaries look like degraded versions of the initialization. Moreover, in the regularized case, the images are better conserved. Nevertheless, in the last row, in both cases the images crafted by the bilevel algorithm have lost their semantic information. Therefore, we often get canaries that are close to the dataset distribution, but not systematically.
Initialization No reg.
Reg.
Figure 2: Example of canaries generated by Algorithm 3. Left: column is the initialization of the bilevel algorithm. Middle: output of IBIS without regularization. Right: output of IBIS with the regularization
C
Additional experiments on CNN
We present additional results on CNN with CIFAR10. Table 4 display the MIA scores for nonprivately trained model while Table 5 present GDP estimates of models trained with DP-SGD.
Table 4: [email protected] scores of one-run auditing on CNN on CIFAR10. For architecture-dependent methods, canaries are optimized using either ResNet9 or CNN as the crafting model. We report mean, standard deviation, and median over 6 runs. IBIS no reg.
IBIS reg.
Boglioni et al. [6]
Random
Random + Flip
Craft. arch.
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
ResNet9 CNN
0.09 ± 0.03 0.13 ± 0.07
0.08 0.13
0.14 ± 0.05 0.25 ± 0.05
0.12 0.23
0.21 ± 0.05 –
0.21 –
0.05 ± 0.02
0.06
0.18 ± 0.04
0.18
Table 5: Auditing score estimate ϵ̂ of one-run auditing on CNN on CIFAR10. For architecturedependent methods, canaries are optimized using either ResNet9 or CNN. We report mean, standard deviation, and median over 6 runs. IBIS no reg. ϵ 10 4 2 1
ResNet9 CNN ResNet9 CNN ResNet9 CNN ResNet9 CNN
IBIS reg.
Boglioni et al. [6]
Random
Random + Flip
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
Mean (± std)
Median
0.23 ±0.18 0.25 ±0.21 0.34 ±0.16 0.38 ±0.30 0.21 ±0.15 0.27 ±0.30 0.27 ±0.23 0.08 ±0.18
0.22 0.30 0.40 0.37 0.21 0.18 0.25 0.00
0.45 ±0.27 0.50 ±0.15 0.40 ±0.37 0.59 ±0.28 0.25 ±0.34 0.19 ±0.34 0.21 ±0.17 0.12 ±0.24
0.38 0.49 0.36 0.58 0.12 0.06 0.21 0.03
0.60 ±0.39 – 0.55 ±0.21 – 0.40 ±0.28 – 0.11 ±0.17 –
0.51 – 0.55 – 0.32 – 0.06 –
0.22 ±0.33
0.13
0.24 ±0.32
0.09
0.19 ±0.22
0.12
0.24 ±0.34
0.08
0.12 ±0.14
0.07
0.05 ±0.07
0.00
0.03 ±0.06
0.00
0.18 ±0.16
0.17
15
D
Experimental details
D.1
Setting
The experiments are run in the PyTorch framework [48]. For DP-SGD, we use the opacus implementation [58]. For the computation of the influence scores, we use the kronfluence package [27]. D.2
Hyperparameters
We report in Table 6 the hyperparameters used for model training and in Table 7 those used for the canary optimization pipeline. Table 6: Training hyperparameters for each architecture and dataset. For DP-SGD runs. CIFAR-10 Hyperparameter
ResNet9
WRN16-4
CNN
Optimizer Learning rate Momentum Weight decay Batch size Epochs
SGD 0.2 0 0 250 18
SGD 0.25 0 0 1024 75
SGD 0.25 0 0 2048 50
DP-SGD only (same epochs as non-private; δ = 10−5 ) Clipping norm C – 1 1 Learning rate (ϵ = 10) – 2 2 Learning rate (ϵ = 4) – 1 2 Learning rate (ϵ = 2) – 1 1 Learning rate (ϵ = 1) – 1 1
Table 7: Hyperparameters for IBIS (Algorithm 3). CIFAR-10 Hyperparameter
ResNet9
WRN16-4
CNN
Influence-based selection Num. canaries m 1000 Preselection size p 10000
1000 10000
1000 10000
SOBA Inner learning rate η Outer learning rate ρ Regularization λ Epochs Batch size
0.05 100 10 100 1024
0.05 100 10 100 2048
0.05 1000 0.1 100 250
16
NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The method yields canary with better MIAs scores in the non-private regime (see Section 5.2) and lower computational cost that competitors (see Section 5.3). Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: As noticed several times in the paper, the performance in terms of DP auditing are on-par with competitors. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 17
Answer: [Yes] Justification: The proof of Proposition 4.1 is in Section A.1. The proof of Proposition 4.2 is in Section A.2. Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: The code is provided and details are given in Section D. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code 18
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code is attached to the submissions and documented. The repository will be made public upon acceptance. Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //neurips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: Details are available in Section D and in the code readme as well. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Figure 1 is a boxplot so statistical errors appear naturally. The tables include standard deviation. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). 19
• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: A computational assessment is provided in Section 5.3 Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The paper does not involve human subject and the datasets used is a classical public dataset. Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Our pipeline make easier the crafting of canary for one-run privacy auditing, facilitating the assessment of the privacy of a machine learning model. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact. 20
• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [N/A] Justification: Experiments are based on common public datasets and do not pose such risks Guidelines: • The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Papers related to datasets and used packages are cited. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 21
• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The code of the experiment is provided. Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: The paper does not involves crowdsourcing nor research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: The paper does not involves crowdsourcing nor research with human subjects. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 22
16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [N/A] Justification: This research does not involve LLMs as any important, original, or nonstandard components. Guidelines: • The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.
23