Discard the Dross and Select the Essential: Pre-query Sample Selection for Black-box Membership Inference Attacks
Dongdong Zhao1 , Jinrong Hu1 , Changtian Song1 , Jian Chen1 , Hongmin Wang2 , and Baogang Song1,*
arXiv:2606.30081v1 [cs.CR] 29 Jun 2026
1
Wuhan University of Technology Email: [email protected], [email protected], [email protected], [email protected], [email protected] 2 Bohai University Email: [email protected] Abstract—Black-box membership inference attacks (MIAs) rely on target-model queries to infer whether candidate samples were used for training. However, membership signals are highly nonuniform across samples: some candidate samples support strong member/non-member separability, whereas many others provide little useful signal. Consequently, indiscriminate querying can incur substantial query cost and increase query-induced exposure, with limited marginal benefit for inference. This raises a key question: which candidate samples are worth querying for black-box MIAs? To address this question, we propose PSSMIA, a pre-query sample selection framework which can be embedded with any existing MIA methods. PSS-MIA proceeds in two stages: it first ranks candidate samples and selects a subset expected to support stronger membership inference, then queries the selected samples and uses the returned outputs for an existing black-box MIA, thereby reducing query cost and query-induced exposure. In the first stage, we propose Loss-Gap Ranking (LGR), which ranks candidate samples by estimating the strength of their membership signal using loss gaps computed from reference models. Experiments on CIFAR-10, CIFAR-100, and CINIC-10 with five representative black-box MIA methods demonstrate that PSS-MIA with LGR consistently outperforms all other compared methods. Moreover, under a 0.1% FPR constraint, PSS-MIA can save at least 83.1%, 60.6%, and 80.4% of the query budget for the three datasets, respectively.
1. Introduction Membership inference attacks (MIAs) aim to determine whether a given candidate sample was used to train a target model [1], [2]. These attacks exploit the fact that trained models often behave differently on training samples and non-training samples, for example, prediction confidence, predicted labels and true-label losses may exhibit different distributions between the two groups [2], [3], [4]. In the blackbox setting, the adversary cannot access the training data or model internals and can only infer whether a candidate sample is a member by querying the target model and * Corresponding author: Baogang Song.
Target model
Candidate set
Log
Monitor threshold
7 many queries (higher cost)
(a) Query all candidate samples Target model
Candidate set
Log
Monitor threshold
few queries (lower cost)
(b) Query partial candidate samples
Figure 1. Motivation of pre-query sample selection for black-box MIAs.
analyzing the returned outputs [1], [5]. Because this access model matches many deployed prediction services, black-box MIA has become a common tool for assessing membership leakage in machine learning models. However, existing black-box MIAs typically assume that the adversary first queries candidate samples and then uses the model outputs to infer membership. Early blackbox MIA methods train multiple shadow models to imitate the behavior of the target model and use their prediction vectors to train an attack model [1], [5]. Later methods such as LiRA [4] use shadow models to estimate IN and OUT distributions and infer membership by comparing the target model output with these distributions [4], [6]. These methods have substantially advanced black-box membership inference, but they mainly focus on how to infer membership after target model outputs have become available. This query-all-then-infer practice has clear limitations in practical scenarios. Target model queries are often costly, especially for commercial APIs or online services, where frequent queries can consume a substantial query budget. Moreover, each query leaves an interaction trace on the server side, such as service-side logs or abnormal-access monitoring signals, thereby increasing query-induced exposure. Figure 1 shows the contrast between many and fewer queries in terms of
query cost and query-induced exposure. In particular, not all candidate samples are equally suitable for membership inference: some exhibit clear member/non-member contrast and provide strong membership signals, whereas many others yield only weak membership signals even after being queried [4], [7]. Therefore, indiscriminately querying all candidate samples may waste a large fraction of the query budget on samples with limited benefit. Together, these considerations raise a key question: which candidate samples are worth querying for black-box MIAs? To address this question, we propose PSS-MIA, a prequery sample selection framework for black-box MIAs that can be integrated with existing MIAs. PSS-MIA proceeds in two stages. The first stage ranks candidate samples without querying the target model and selects a subset expected to support stronger membership inference. The second stage applies a black-box MIA only to the selected samples. By limiting membership inference to the selected subset, PSSMIA reduces query cost and query-induced exposure. In the ranking stage, we propose Loss-Gap Ranking (LGR). LGR compares the losses of out-reference models (i.e., reference models trained without the candidate sample) and in-reference models (i.e., reference models trained with the candidate sample) to estimate membership signals, which serve as the basis for pre-query sample selection. For well-generalized target models, black-box MIAs may exhibit weak attack performance over the full candidate set and a low full-set AUC may therefore underestimate their membership leakage risk. Nevertheless, the same attack may achieve stronger performance on a selected subset of candidate samples. To characterize the risk that is not fully reflected by the full-set AUC, we introduce a new metric called Target Attack-Performance Coverage (TAPC). Given an attack goal, such as a specified AUC or ACC threshold, TAPC reports the size of the largest subset of candidate samples on which the evaluated MIA method achieves the attack goal. A positive TAPC indicates that at least one subset satisfies the prescribed attack goal, revealing membership leakage risk within part of the candidate set. A larger TAPC indicates that the goal is satisfied on a larger subset, reflecting more severe membership leakage. In some cases, AUC or ACC measured over the full candidate set may be low, but TAPC can expose substantial membership leakage risk on subsets. In summary, our main contributions are as follows: • We propose PSS-MIA, a pre-query sample selection framework that can be used with existing black-box MIAs. It ranks candidate samples, selects a subset expected to support stronger membership inference and only queries the target model with the selected subset, thereby reducing query cost and query-induced exposure. • We propose LGR, which ranks candidate samples by estimating the strength of their membership signal using loss gaps computed from reference models. This ranking enables PSS-MIA to select samples on which the chosen black-box MIA achieves stronger member/non-member separability.
• We define a new metric called TAPC, which can evaluate
the risk of models more reasonably. Instead of evaluating the privacy risk by an average metric over the full candidate set, TAPC focuses on the subsets that contain high-risk samples. • Extensive experiments across multiple datasets, target architectures and representative black-box MIAs show that PSS-MIA with LGR consistently improves attack performance on selected subsets over baseline methods. Under low-FPR constraints, it also substantially reduces the target-query budget required to find fixed true positive samples. The results further identify a stable tail on which black-box MIAs show weak membership inference performance.
2. Related Work 2.1. Membership Inference Attacks Black-box MIAs aim to determine whether a candidate sample was a part of target model’s training set, using only its accessible prediction outputs. Shokri et al. [1] pioneered the shadow-model paradigm, where auxiliary models mimic the target model and generate prediction outputs for training an attack classifier. Salem et al. [5] demonstrated that such attacks remain effective even with fewer shadow models and less knowledge of the target model and its training data distribution. Truex et al. [8] further studied membership inference in machine learning as a service (MLaaS) and analyzed how target model and data properties affect attack performance. Collectively, these studies established the shadow-model-based paradigm for inferring membership from black-box prediction outputs. Subsequent research showed that black-box MIAs can infer membership directly from prediction statistics, without training an attack classifier. Yeom et al. [2] proposed using the true-label loss as a membership score, while Sablayrolles et al. [9] analyzed loss-based membership inference from a Bayes-optimal perspective. Song et al. [10] introduced modified prediction entropy and examined membership leakage in adversarially robust models. Song and Mittal [7] later evaluated attacks based on confidence, entropy, modified entropy, and loss. Hui et al. [11] proposed a blind MIA based on differential comparisons. These studies showed that simple statistics derived from prediction outputs can provide effective membership signals. Another line of work considers more restrictive blackbox settings where the target model returns only limited output information. Choquette-Choo et al. [3] showed that membership can be inferred in a label-only setting using prediction stability under input perturbations. Li and Zhang [12] further proposed two decision-based attacks within this labelonly setting: a transfer attack and a boundary attack. More recently, Li et al. [13] proposed DHAttack, a label-only MIA that reduces the per-sample query cost for boundary-distance estimation. These studies show that black-box membership inference remains possible even when the target model returns only predicted labels.
A parallel line of work investigates how additional black-box observations or query variants affect membership inference. Liu et al. [14] examined membership leakage when model explanations are exposed alongside prediction outputs. Wen et al. [15] improved the performance of MIAs by querying adversarially optimized variants of the target sample. These studies focus on extracting richer membership signals from queried samples. More recent work has used statistical testing and calibration to improve black-box MIAs. EnhancedMIA [16] provides a hypothesis-testing framework that unifies several existing attacks. LiRA [4] trains shadow models with and without each candidate sample, estimates the corresponding score distributions, and then applies a likelihood-ratio test. Watson et al. [17] adjust membership scores according to sample classification difficulty and show that this calibration can reduce false positives. Rezaei and Liu [18] further show that many existing attacks struggle to achieve strong performance at low false positive rates. Quantile MIA [19] uses quantile regression to estimate a sample-specific threshold on the confidence score. RMIA [6] estimates membership using a likelihood-ratio test built from reference models and population samples, and remains effective with only a small number of reference models. RAPID [20] combines the original membership score with the calibrated score to reduce errors introduced by difficulty calibration. Other attacks exploit training-process information. TrajectoryMIA [21] uses the loss trajectory of each sample, and SeqMIA [22] evaluates multiple metrics on distilled models in chronological order and uses the resulting metric sequences for membership inference. These methods enrich the information used to compute membership scores. Different from all prior work, our work studies sample selection for black-box membership inference. We rank candidate samples and query only a selected subset, where existing black-box MIAs are expected to achieve stronger performance.
2.2. Non-Uniform Membership Signals Membership-inference risk is not determined solely by a model’s overall generalization behavior [23]. Early analyses linked overfitting to membership leakage [2]. Data augmentation and regularization can change MIA performance in ways that are not fully explained by the generalization gap [24], [25]. Consequently, models with similar overall generalization performance may still exhibit substantially different membership signals across individual samples. These studies mainly examine the relationship between model-level generalization behavior and membership leakage. To explain this variation, prior work has linked this phenomenon to sample difficulty, learning dynamics, and memorization. Difficulty-calibrated attacks adjust the membership score according to the classification difficulty of the target sample. Studies of example forgetting show that individual training examples undergo forgetting events at different frequencies during learning [26]. Work on longtailed data distributions shows that memorization of rare
and atypical examples can be necessary for achieving closeto-optimal generalization error [27], [28], while subsequent work shows that memorization can be localized to a small subset of neurons distributed across layers [29]. These studies help explain why membership signals can vary considerably across samples, while their focus remains exclusively on the underlying causes and mechanisms. Prior work has also examined membership leakage from other perspectives. Leino and Fredrikson [30] show that a model’s idiosyncratic use of features can provide evidence of membership, and Kulynych et al. [31] studied differential vulnerability to MIAs across population subgroups. The privacy onion effect further shows that membership vulnerability can depend on the surrounding data distribution [32]. These studies provide additional evidence that membership leakage is shaped by factors beyond global model accuracy. Taken together, these studies show that membership signals can differ across individual samples due to generalization behavior, sample difficulty, memorization, feature use, subgroup disparity, and data composition. Our work builds on this observation to study a different problem: before target-model outputs are available, how an adversary can select candidate samples that are expected to enable stronger black-box membership inference. We address this problem by proposing PSS-MIA, which uses LGR to rank candidate samples before querying the target model. We further introduce TAPC to characterize membership leakage within candidate subsets.
3. PSS-MIA 3.1. Problem Description and Threat Model We study a pre-query sample selection problem for blackbox MIAs: before querying the target model, the adversary ranks candidate samples and then selects a subset expected to provide stronger membership signals. Let f denote the target model trained on a dataset Dtrain . Let D = {zi = (xi , yi )}N i=1 denote a candidate set for auditing that contains the member samples in Dtrain together with additional non-member samples. For each zi ∈ D, the black-box MIA aims to infer whether zi ∈ Dtrain . Our focus is the preceding sampleselection stage. Specifically, we seek to construct a ranking π over the candidate samples, zπ(1) , zπ(2) , . . . , zπ(N ) ,
such that samples expected to exhibit stronger membership signals are ranked higher. Based on this ranking, the adversary can select the top samples for querying the target model, thereby reducing the query cost spent on candidates with weak membership signals and improving attack efficiency. The threat model follows a label-aware black-box setting. The adversary knows the ground-truth label y for each candidate sample z = (x, y) ∈ D, but does not know the exact composition of the target training set Dtrain or which
1 Pre-query ranking with LGR
Ranked candidate set
1
In-reference models
Candidate set D
Fin z = { g ∈ F : z ∈ Sg }
...
...
3
...
For each z∈D
2
μin (z) LGR score
r z = μout z − μin (z)
Out-reference models
Fout z = { g ∈ F : z ∉ Sg }
...
μout z
D −2 D −1 D
2 Selected-subset querying and black-box MIA MIAs (LiRA, Seq, RMIA, ...)
𝑘
Qk ={ zπ 1 , …, zπ(k) }
...
Top-k AUC TPR
query
inference
...
Target model
FPR
Figure 2. Overview of PSS-MIA. LGR ranks candidate samples using in-reference and out-reference losses before target model querying, after which an existing black-box MIA is applied to the selected samples.
candidate samples are members. The adversary can access auxiliary data drawn from the target training distribution or from a closely matched distribution and can train reference models under protocols matched to the target setting. The pre-query selection stage is completed without querying the target model.
3.2. Overview of the PSS-MIA PSS-MIA is a pre-query sample selection framework that can be integrated with existing black-box MIA methods. Figure 2 shows the overall workflow of PSS-MIA. Given a candidate set D, PSS-MIA first computes a score for each candidate sample without querying the target model. This score is not a membership prediction; rather, it estimates the expected strength of the sample’s membership signal. Candidate samples are then ranked according to this score and a top subset is selected for the subsequent MIA. Let r(z) denote the score for each candidate sample z ∈ D. For LGR, the score r(z) is defined in Section 3.4. Sorting candidate samples in descending order of r(z) yields a ranking π such that r zπ(1) ≥ r zπ(2) ≥ · · · ≥ r zπ(N ) . Given a selection size k , PSS-MIA selects the top subset Qk = {zπ(1) , zπ(2) , . . . , zπ(k) }.
The selection size k determines the number of top samples selected for querying the target model and can be set according to the available query budget or a specified attack goal. These samples are expected to exhibit stronger membership signals. After selecting Qk , the adversary queries the target model only with the samples in Qk and obtains the corresponding target model outputs. These outputs are then used by an existing black-box MIA method A to infer membership. Let sA (z) denote the membership score assigned by A to each z ∈ Qk . PSS-MIA therefore changes only which candidate samples are queried, while leaving the chosen black-box MIA unchanged.
3.3. Out-Minus-In Loss Shift as the LGR Ranking Signal Before defining LGR, we provide an intuition for its ranking signal. For a candidate sample z , a model trained with z is expected to incur a lower loss on z than a model trained without z . Accordingly, a larger out-minus-in loss shift reflects a greater loss difference between the two cases, which may indicate a stronger membership signal. In the following analysis, we consider an ideal out-minus-in loss shift between models trained with and without z and examine its relation to the Bayes error of in/out discrimination. This
ideal quantity is used only to motivate the ranking signal; the practical LGR score is defined in Section 3.4. For a candidate sample z = (x, y) and a trained model f , we define the true-label loss as ℓ(f ; z) = − log pf (y | x),
(1)
where pf (y | x) denotes the predicted probability assigned by f to the true label y . Since this loss can be unbounded when pf (y | x) approaches zero, we introduce a clipped version: ℓB (f ; z) = min{ℓ(f ; z), B},
B > 0.
(2)
The clipping is used only to ensure boundedness in the analysis in this subsection. The practical LGR score in Section 3.4 uses the unclipped loss in Eq. (1). For this analysis, we consider the distributions of ℓB (f ; z) under the two training conditions z ∈ Dtrain and z ∈ / Dtrain . Let Pzin = L(ℓB (f ; z) | z ∈ Dtrain ) , (3) Pzout = L(ℓB (f ; z) | z ∈ / Dtrain ) . denote the corresponding loss distributions. The randomness comes from the training process in each case. We define the target-side out-minus-in mean loss shift as ∆B (z) = E[ℓB (f ; z) | z ∈ / Dtrain ] − E[ℓB (f ; z) | z ∈ Dtrain ].
(4)
Under this definition, a positive ∆B (z) indicates that the expected clipped loss of z is higher under the out condition than under the in condition. in out Let Lin ∼ Pzout denote the clipped-loss z ∼ Pz and Lz variables under the two training conditions and define Lin Lout z , Uzout = z . B B We use TV(P, Q) to denote the total variation distance between two distributions P and Q. Then, Uzin =
∆B (z) = E Uzout − E Uzin B Z 1 = Pr Uzout > t − Pr Uzin > t dt Z0 1 ≤ Pr Uzout > t − Pr Uzin > t dt 0 ≤ TV Pzin , Pzout .
(5) (6) (7) (8)
It follows that ∆B (z) ≤ B · TV(Pzin , Pzout ).
(9)
Under equal priors, the Bayes error for discriminating between Pzin and Pzout based only on the clipped loss is 1 e⋆z = 1 − TV(Pzin , Pzout ) , (10) 2 which, together with Eq. (9), implies e⋆z ≤
1 ∆B (z) − . 2 2B
(11)
This bound relates the signed out-minus-in mean loss shift to the Bayes error for distinguishing between the clipped-loss distributions under the in and out conditions. In particular, a larger positive ∆B (z) gives a smaller upper bound on this Bayes error. A smaller Bayes error indicates greater separability between the clipped-loss distributions under the in and out conditions. This observation motivates using the out-minus-in loss shift as a ranking signal; it does not establish the optimality of the induced ranking. The shift ∆B (z) compares two target-side training cases for the same candidate sample, corresponding to whether z is used for training. This shift is not available in practice and the pre-query ranking stage cannot rely on target model queries. To obtain a practical ranking signal, LGR uses matched reference models instead. It compares the losses of reference models trained with z and those trained without z , using this reference-model loss gap to approximate the ideal shift ∆B (z). The LGR score is defined in Section 3.4.
3.4. Loss-Gap Ranking Motivated by the analysis in Section 3.3, LGR uses reference models to construct a practical ranking signal. For each candidate sample, LGR compares its losses from inreference and out-reference models and uses the resulting loss gap to rank candidate samples. For a reference model g and a candidate sample z = (x, y), LGR uses the unclipped true-label loss ℓ(g; z) = − log pg (y | x),
where pg (y | x) denotes the probability assigned by g to the true label y . Let F denote the collection of reference models. For each candidate sample z , we define Fin (z) = {g ∈ F : z ∈ Sg }, Fout (z) = {g ∈ F : z ∈ / Sg }.
(12)
These sets contain the reference models trained with and without z , respectively, where Sg is the training set of reference model g . Let Min (z) = |Fin (z)|,
Mout (z) = |Fout (z)|.
We assume that Min (z) > 0 and Mout (z) > 0 for every candidate sample z . The mean in-reference and out-reference losses are then computed as X 1 µin (z) = ℓ(g; z), Min (z) g∈Fin (z) (13) X 1 µout (z) = ℓ(g; z). Mout (z) g∈Fout (z)
The LGR score is then defined as the out-minus-in reference loss gap: r(z) = µout (z) − µin (z).
(14)
A larger r(z) indicates a stronger loss contrast between the out-reference and in-reference models. LGR ranks candidate samples in descending order of r(z) and selects the top k samples for subsequent black-box membership inference. LGR is a pre-query ranking method rather than a standalone MIA. Because r(z) is computed entirely from reference-model losses, the ranking is obtained before the target model is queried on the candidate samples. After a top subset is selected, the target model is queried only on that subset and the returned outputs are used by the chosen black-box MIA.
3.5. Target Attack-Performance Coverage Conventional MIA evaluation often reports AUC, ACC, or TPR at a fixed FPR over the entire candidate set, but they do not reveal how membership signals vary across samples. Therefore, weak full-set attack performance does not imply uniformly weak membership signals; a subset of samples may still provide sufficiently strong membership signals for effective membership inference. To characterize the risk potentially obscured by full-set attack performance, we introduce the metric TAPC. Given the ranked set of candidates QN = {zπ(1) , zπ(2) , . . . , zπ(N ) } and an attack goal (e.g., AUC > 0.90), TAPC reports the largest subset size k for which the chosen black-box MIA satisfies the attack goal on Qk . Let Keval = {k1 , k2 , . . . , km } ,
k1 < k2 < · · · < km ,
denote the ordered set of evaluated subset sizes. Let M be an MIA evaluation metric for which larger values indicate stronger attack performance. Given a prescribed performance level τ , we define TAPCM (τ ) = max {k ∈ Keval : M(Qk ) ≥ τ } .
(15)
If no evaluated subset satisfies the prescribed performance level, we set TAPCM (τ ) = 0. In our experiments, we use AUC and ACC as M. Under a fixed attack and evaluation protocol, TAPCM (τ ) > 0 indicates that at least one evaluated subset satisfies the prescribed performance level, revealing membership leakage risk that is concentrated in a subset of the candidate samples. A larger TAPC indicates that the chosen black-box MIA satisfies the prescribed attack goal on a larger selected subset, revealing membership leakage over more candidate samples. TAPC thus complements full-set attack performance by revealing membership-inference risk that may be concentrated among a subset of candidate samples.
4. Experiments We evaluate PSS-MIA with LGR across datasets, model architectures and MIAs. We measure attack performance
on the selected subsets and target-query efficiency under low-FPR constraints. We also study the effect of subset size, the sensitivity of LGR to the number of reference model pairs and the tail produced by LGR. Finally, we use TAPC to measure the largest subset that satisfies a prescribed attack goal. Together, these experiments evaluate the effectiveness of PSS-MIA and LGR.
4.1. Experimental Setup Since there is no directly comparable framework for pre-query sample selection in black-box MIAs, we evaluate PSS-MIA by comparing different sample-ranking methods (random ranking, loss-based ranking, LT-IQR-based ranking and LGR) within the same framework proposed in this paper. We keep the overall PSS-MIA procedure fixed and vary only the ranking method. For each black-box MIA, all ranking methods use the same target model, candidate set and selection size. Random ranking uses a random permutation of the candidate set. Loss-based ranking sorts candidate samples by their true-label loss from shadow models. LT-IQR-based ranking adapts the loss-trace interquartile-range method [33] to our setting: we train shadow models using the candidate set, record the true-label loss trajectory of each candidate sample during shadow model training and rank samples by the interquartile range of the loss trajectory. We employ LiRA [4], RMIA [6], Seq [22], Loss attack [2] and Attack R [16] as black-box MIAs. Given a size k , we first perform a global ranking over the candidate set without using target model outputs. We then select the highest-ranked k/2 members and k/2 nonmembers according to this ranking, ensuring that differences in AUC and TPR@FPR are not caused by class imbalance. Membership labels are used only to construct balanced subsets for controlled evaluation. They are not used by PSSMIA during ranking, target model querying, or membership inference. We report results from three perspectives: attack performance on selected subsets, TAPC and target-query efficiency. Top-k AUC and TPR at low FPR measure attack performance on balanced evaluation subsets. TAPC measures the largest subset that satisfies a prescribed attack-performance level. Minimum Query Budget (MQB) is the minimum query budget required to attain a prescribed number of true positives under a fixed low-FPR constraint. For MQB, we use the same balanced subset protocol described above and vary the selection size k . For a selected subset Qk and a low-FPR constraint α, let TPα (Qk ) =
max θ:FPR(Qk ,θ)≤α
TP(Qk , θ),
where θ is the attack decision threshold and TP(Qk , θ) denotes the number of member samples in Qk correctly classified as members. Given a target number of true positives m, MQB is defined as MQBα (m) = min{k : TPα (Qk ) ≥ m}.
TABLE 1. ATTACK PERFORMANCE (AUC, [email protected]%FPR) UNDER DIFFERENT RANKING METHODS AT 20% SELECTION RATIO .
Dataset
Ranking method
LiRA
RMIA
Seq
Loss
Attack R
AUC
TPR
AUC
TPR
AUC
TPR
AUC
TPR
AUC
TPR
CIFAR-10
Random Loss LT-IQR LGR
74.20% 85.80% 96.69% 97.64%
0.10 0.18 0.29 0.30
69.89% 79.69% 88.43% 89.14%
0.03 0.08 0.09 0.09
76.30% 84.91% 95.50% 96.17%
0.03 0.06 0.11 0.11
60.59% 71.10% 92.03% 95.02%
0.00 0.00 0.01 0.02
69.62% 81.29% 95.43% 96.88%
0.04 0.11 0.24 0.25
CIFAR-100
Random Loss LT-IQR LGR
94.78% 98.98% 99.96% 99.99%
0.30 0.54 0.89 0.96
89.54% 96.62% 99.66% 99.88%
0.24 0.40 0.60 0.68
96.38% 99.11% 99.82% 99.98%
0.21 0.22 0.60 0.97
82.61% 94.38% 99.70% 99.97%
0.00 0.00 0.32 0.87
91.16% 97.68% 99.85% 99.97%
0.18 0.30 0.55 0.97
CINIC-10
Random Loss LT-IQR LGR
85.87% 93.22% 99.36% 99.65%
0.15 0.33 0.46 0.52
80.10% 87.96% 97.14% 98.15%
0.10 0.22 0.26 0.26
96.95% 99.36% 99.34% 99.48%
0.09 0.33 0.30 0.61
71.27% 81.41% 98.01% 99.24%
0.00 0.00 0.01 0.13
81.40% 90.17% 98.91% 99.50%
0.12 0.23 0.31 0.36
If no evaluated subset reaches m true positives, MQB is reported as not reached. A smaller MQB indicates that fewer target model queries are required to identify the same number of true-positive samples under the same low-FPR constraint and balanced evaluation protocol. We evaluate PSS-MIA with LGR as its ranking method on CIFAR-10 and CIFAR-100 [34] and CINIC-10 [35] using three model architectures: ResNet-18 [36], VGG16 [37] and WRN-28-2 [38]. For CIFAR-10 and CIFAR-100, we use the standard 50K images as the candidate set. For CINIC-10, we uniformly sample a fixed subset of 50K images and use it throughout the experiments. For each dataset, the candidate set is randomly divided into two halves. One half is used to train the target model and constitutes the member set, while the other half forms a disjoint non-member set. The resulting evaluation set therefore contains equal numbers of members and non-members. All target and reference models are trained using the same training configuration. LGR uses 128 complementary reference-model pairs by default to compute the ranking scores. We also evaluate the sensitivity of LGR to the number of reference-model pairs.
4.2. Attack Performance on Selected Subsets This subsection evaluates whether PSS-MIA with LGR achieves higher attack performance than using other ranking methods (random ranking, loss-based ranking and LT-IQRbased ranking). For each evaluated MIA, we keep the attack unchanged and vary only the ranking method and report AUC and [email protected]%FPR on the resulting subsets. Table 1 compares the attack performance of PSS-MIA under different ranking methods at a fixed selection ratio of 20% across the three datasets, with ResNet-18 as the target architecture. Loss-based ranking, LT-IQR and LGR
TABLE 2. ATTACK PERFORMANCE (AUC/[email protected]%FPR) AT DIFFERENT SELECTION SIZES . size k
LiRA
Loss
Seq
5% 10% 20% 50% 100%
99.77% / 0.59 99.42% / 0.51 97.64% / 0.30 87.62% / 0.17 74.08% / 0.09
99.42% / 0.20 98.56% / 0.10 95.02% / 0.02 78.22% / 0.00 60.28% / 0.00
99.39% / 0.26 98.71% / 0.18 96.17% / 0.11 87.62% / 0.06 76.13% / 0.03
all outperform the random ranking, demonstrating the effectiveness of sample selection. Among these ranking methods, LGR achieves the best results across all reported settings. On CIFAR-10, for example, under LiRA, replacing random ranking with LGR increases AUC from 74.20% to 97.64% and [email protected]%FPR from 0.10 to 0.30. Notably, although LT-IQR achieves AUC values comparable to those of LGR in several settings, LGR yields substantially higher [email protected]%FPR. For example, on CIFAR-100, LGR increases the TPR from 0.60 to 0.97 for Seq and from 0.32 to 0.87 for the Loss attack. We further evaluate PSS-MIA across different target architectures. For each dataset, Figure 3 reports the ROC curves of LiRA on ResNet-18, VGG16 and WRN-28-2, comparing PSS-MIA using LGR with PSS-MIA using random ranking. Across all the datasets and target architectures, PSS-MIA with LGR yields better ROC curves than PSS-MIA with random ranking.
4.3. Sensitivity to Selection Size and ReferenceModel Pairs We first examine how the attack performance of PSSMIA changes as the selected subset size increases. Table 2
True Positive Rate
10
CIFAR-10
0
10
−1
10
−2
10
−3
CIFAR-100
ResNet18, selected (AUC=0.977) ResNet18, random (AUC=0.742) VGG16, selected (AUC=0.963) VGG16, random (AUC=0.728) WRN-28-2, selected (AUC=0.914) WRN-28-2, random (AUC=0.676)
10
−3
10
−2
10
−1
CINIC-10
ResNet18, selected (AUC=1.000) ResNet18, random (AUC=0.948) VGG16, selected (AUC=0.944) VGG16, random (AUC=0.816) WRN-28-2, selected (AUC=0.964) WRN-28-2, random (AUC=0.830)
10
0
10
−3
10
False Positive Rate
−2
10
−1
False Positive Rate
10
ResNet18, selected (AUC=0.997) ResNet18, random (AUC=0.859) VGG16, selected (AUC=0.977) VGG16, random (AUC=0.814) WRN-28-2, selected (AUC=0.950) WRN-28-2, random (AUC=0.767) 0
10
−3
10
−2
10
−1
10
0
False Positive Rate
Figure 3. ROC comparison of LiRA across datasets and target architectures (LGR-selected vs. random).
from 0.961 to 0.976 and raises the AUC on the selected 40% subset from 0.896 to 0.910. These results suggest that LGR already provides an effective ranking signal with a small number of reference-model pairs, while using more pairs brings further gains.
0.970
Attack AUC
0.960 0.950
Top-20% Top-40%
0.940 0.930 0.920
4.4. Target-Query Cost and Query-Induced Exposure
0.910 0.900 1
2
4
8
16
32
64
128
Number of reference pairs Figure 4. AUC on selected 20% and 40% candidate subsets with different numbers of reference-model pairs.
reports top-k AUC and [email protected]%FPR for representative attacks on CIFAR-10 with ResNet-18. Across the reported attacks, both metrics decrease as k increases. For LiRA, AUC decreases from 99.77% at k = 5% to 74.08% at k = 100%, while [email protected]%FPR decreases from 0.59 to 0.09. Seq and the loss-based attack show the same trend. These results demonstrate the effectiveness of LGR. When the selected subset is small, the queried samples retain stronger membership signals and support substantially higher attack performance. As the selection size increases, the subset includes more samples with weaker membership signals and the measured attack performance gradually decreases toward the full-set level. We also examine the sensitivity of LGR to the number of reference model pairs used for score aggregation. We compute LGR scores with different numbers of complementary reference model pairs and evaluate the resulting rankings under the same selected-subset protocol. As shown in Figure 4, using more reference-model pairs improves attack performance on the selected subsets, but the gain gradually diminishes. Increasing the number of complementary pairs from 1 to 128 raises the AUC on the selected 20% subset
Table 3 reports MQB of PSS-MIA across different lowFPR constraints and true positive targets. Unlike the fixedsubset evaluation in Section 4.2, MQB fixes a target number of true positive samples and measures the minimum subset size needed to find such a number of true positives. A smaller MQB indicates that PSS-MIA achieves this attack goal with a smaller target-query budget, providing a measure of targetquery efficiency. The results show that PSS-MIA with LGR substantially reduces MQB compared with the random-ranking baseline. For example, on CIFAR-10 under a 10−3 FPR constraint, random ranking requires 10,612, 20,976 and 41,798 target queries to find 500, 1,000 and 2,000 true-positive samples by LiRA, respectively, whereas PSS-MIA with LGR requires only 1,030, 2,140 and 5,296 queries. This yields reductions of 90.3%, 89.8% and 87.3% in the target-query budget. These results indicate that PSS-MIA with LGR improves query efficiency: true positives under the low-FPR constraint appear earlier in the rank, so the same targets can be reached with fewer target queries. This reduction is relevant beyond computational efficiency. In deployed black-box services, each target query is an observable interaction and may contribute to service-side logging, rate-limit pressure, or abnormal-access monitoring signals. Reducing unnecessary target queries can therefore lower target-query cost and may also reduce query-induced exposure.
TABLE 3. MQB OF PSS-MIA AT DIFFERENT LOW-FPR CONSTRAINTS AND TRUE POSITIVE TARGETS . CIFAR-10 FPR
Rank.
TP target m 10
10−1
CIFAR-100
50
100
CINIC-10
TP target m
500
1000
2000
10
50
100
500
TP target m 1000
2000
10
50
100
500
1000
2000
Rand. 52 238 510 2638 5406 10914 24 138 266 1268 2524 5014 38 184 350 1808 3642 7234 LGR 20 100 200 1006 2016 4090 20 100 200 1000 2000 4000 20 100 200 1004 2000 4016 Red. (↓) 61.5% 58.0% 60.8% 61.9% 62.7% 62.5% 16.7% 27.5% 24.8% 21.1% 20.8% 20.2% 47.4% 45.7% 42.9% 44.5% 45.1% 44.5%
Rand. 72 572 1240 6014 11830 23820 42 202 374 1898 3858 7718 74 330 662 3344 6794 13254 20 100 200 1012 2050 4298 20 100 200 1000 2000 4002 20 100 200 1004 2000 4046 10−2 LGR Red. (↓) 72.2% 82.5% 83.9% 83.2% 82.7% 82.0% 52.4% 50.5% 46.5% 47.3% 48.2% 48.1% 73.0% 69.7% 69.8% 70.0% 70.6% 69.5% Rand. 118 1084 2098 10612 20976 41798 52 270 508 2834 5640 11496 102 624 1176 6004 11882 23866 20 100 200 1030 2140 5296 20 100 200 1000 2002 4016 20 100 200 1004 2040 4280 10−3 LGR Red. (↓) 83.1% 90.8% 90.5% 90.3% 89.8% 87.3% 61.5% 63.0% 60.6% 64.7% 64.5% 65.1% 80.4% 84.0% 83.0% 83.3% 82.8% 82.1%
20,672
2
19,204 (96.0%)
3
17,642 (88.2%)
4 0
5k
10k
15k
20k
True Positive Rate
At least occurrences
10
22,482
1
0
10
−1
10
−2
25k
bottom (AUC=0.596) random (AUC=0.768)
Number of samples Figure 5. Occurrence frequency of samples in the 20,000-sample tails across different LGR ranking trials.
10
−3
10
−3
10
−2
10
−1
10
0
False Positive Rate
4.5. Stability and Weak Membership Signals in the Tail
Figure 6. ROC curves of LiRA on target models retrained from the lowerranked 30,000-sample subset and from a random 30,000-sample subset.
The results in Section 4.3 show that stronger attack performance is concentrated in the selected subset. We next examine the tail induced by LGR. Specifically, we test whether samples in this tail consistently show weak membership signals under the evaluated attacks and whether the tail remains stable across different LGR ranking trials. We consider three analyses in this subsection: attack performance on tails, stability across different LGR ranking trials and retraining-based validation on the lower-ranked subset. Table 4 compares the attack performance on top-ranked subsets and tails induced by LGR. The top-ranked subsets exhibit strong member/non-member separability, with LiRA reaching 97.64% AUC on the top-20% subset and 91.04% AUC on the top-40% subset. In contrast, the tails show much weaker membership signals: the AUC of LiRA drops to 51.15% and 51.25% on the bottom-20% and bottom40% tails; the loss-based attack stays near chance and the [email protected]%FPR is close to zero across attacks. These results indicate that the LGR-induced tails contain limited membership signals. We next examine whether the tail remains stable across different LGR ranking trials, rather than being an artifact of a single run. To this end, we conduct four ranking trials,
each of which trains 32 reference-model pairs independently. Figure 5 reports how many samples appear in at least 1, 2, 3, or 4 of the four tails: 19,204 samples appear in at least three tails, and 17,642 samples appear in all four tails, corresponding to 96.0% and 88.2% of a 20,000-sample tail, respectively. These results indicate that the tail induced by LGR is stable across different ranking trials. To further characterize this phenomenon, we examine whether a lower-ranked subset also yields weaker membership signals when the target model is retrained. We take the lower-ranked 30,000-sample subset directly induced by LGR and randomly split it into two halves (using one half for training and the other half as a non-member set). As shown in Figure 6, LiRA achieves an AUC of only 0.596 on the target model retrained from this lower-ranked subset, whereas it achieves a much higher AUC of 0.768 when the same retraining protocol is applied to a random 30,000-sample subset. These results show that membership signals differ significantly between top-ranked subsets and the tails induced by LGR. This rank-wise variation can be obscured by fullset attack metrics, making it important to propose a new evaluation metric. We next use TAPC to measure the largest
TABLE 4. ATTACK PERFORMANCE (AUC, [email protected]%FPR) ON TOP - RANKED SUBSETS AND TAILS INDUCED BY LGR.
LiRA
Ranked region top-20% top-40% bottom-20% bottom-40%
True Positive Rate
10
Loss
−1
10
−2
Seq
Attack R
AUC
TPR
AUC
TPR
AUC
TPR
AUC
TPR
AUC
TPR
97.64% 91.04% 51.15% 51.25%
0.30 0.19 0.00 0.00
89.14% 82.59% 48.92% 50.61%
0.09 0.07 0.00 0.00
95.02% 83.70% 50.14% 51.32%
0.02 0.00 0.00 0.00
96.17% 90.48% 51.63% 54.30%
0.11 0.07 0.00 0.00
96.88% 89.23% 49.49% 50.60%
0.25 0.23 0.00 0.00
0
10
10
RMIA
TABLE 5. TAPC COMPARISON UNDER DIFFERENT ATTACK GOALS .
full (AUC=0.582) top-5000 (AUC=0.808)
−3
10
−3
10
−2
10
−1
10
0
False Positive Rate Figure 7. ROC curves of LiRA on the full candidate set and on the LGRselected 5,000-sample subset for the well-generalized target model.
subset that satisfies a prescribed attack goal.
4.6. Attack-Performance Coverage Beyond Full-Set Evaluation For well-generalized target models, stronger generalization can substantially weaken full-set membership inference performance [39]. However, weak full-set attack performance does not necessarily imply that membership signals are uniformly weak across the candidate set. To make this distinction explicit, we evaluate a well-generalized target model and use TAPC to characterize membership leakage risk that may be obscured by weak full-set attack performance. We construct this target model by strengthening data augmentation and regularization, while keeping the remaining training and evaluation protocol unchanged. Under this setting, the target model has a generalization gap of about 3%. With the MIA method kept unchanged, we rank candidate samples with LGR and compare LiRA performance on the full candidate set and on the subset selected by LGR. As shown in Figure 7, LiRA achieves an AUC of 0.582 on the full candidate set, but the AUC increases to 0.808 on the selected subset. This gap shows that weak full-set attack performance can obscure stronger membership signals within selected candidate subsets. Table 5 further quantifies this difference using TAPC. Compared with the standard target model, the wellgeneralized target model has smaller TAPC values under
Attack goal
Well-generalized model
Standard model
AUC = 0.8 AUC = 0.7 ACC = 0.8 ACC = 0.7
5666 17180 40 9344
28882 50000 17354 37324
the same AUC or ACC threshold, indicating that stronger generalization can reduce the membership leakage risk. Nevertheless, the TAPC values remain positive under several AUC and ACC thresholds. This shows that weak fullset attack performance alone does not rule out localized membership leakage within the candidate set.
4.7. Cross-Architecture Generalization We further evaluate whether the LGR ranking signal transfers across model architectures. Although the default setting uses reference models matched to the target setting, the adversary may not know the exact target architecture in practice. We train reference models using ResNet-18, VGG16, and WRN-28-2 to compute LGR scores. We then evaluate LiRA on the selected 20% subsets for target models using each of the three architectures, covering both matched and mismatched architecture settings. The candidate set and selection ratio are kept fixed throughout. Table 6 reports AUC/[email protected]%FPR of PSS-MIA under different reference and target model architectures. Table 6 shows that LGR generally remains effective under different reference and target model architectures. The matched-architecture setting often achieves the best results, but mismatched reference architectures still produce strong results on the selected subsets. These results indicate that the out-minus-in reference loss gap provides a transferable ranking signal across model architectures.
4.8. Ablation Study We further examine whether LGR benefits from contrasting in-reference and out-reference losses, rather than relying on one-sided reference-loss signals alone. To this end, we conduct an ablation study with three scoring variants: (1) in-reference loss; (2) out-reference loss; (3) the outminus-in loss gap used by LGR. The in-reference-loss score ranks candidate samples by their mean loss from in-reference
TABLE 6. C ROSS - ARCHITECTURE EVALUATION OF LGR (AUC/[email protected]%FPR).
ResNet-18
Target model architecture VGG16
WRN-28-2
CIFAR-10
ResNet-18 VGG16 WRN-28-2
97.64% / 0.30 96.79% / 0.29 97.07% / 0.30
94.86% / 0.19 96.25% / 0.19 94.86% / 0.18
91.26% / 0.13 90.51% / 0.13 91.44% / 0.13
CIFAR-100
ResNet-18 VGG16 WRN-28-2
99.99% / 0.96 99.79% / 0.76 99.98% / 0.95
90.45% / 0.11 94.40% / 0.16 91.62% / 0.12
95.86% / 0.25 94.58% / 0.20 96.35% / 0.24
CINIC-10
ResNet-18 VGG16 WRN-28-2
99.65% / 0.52 99.31% / 0.53 99.48% / 0.54
96.57% / 0.23 97.68% / 0.24 96.54% / 0.23
94.90% / 0.17 94.31% / 0.17 95.02% / 0.17
Dataset
Reference architecture
TABLE 7. R EFERENCE - LOSS SIGNAL ABLATION (AUC, [email protected]%FPR).
LiRA
Ranking score
AUC
TPR
Seq AUC
Loss TPR
AUC
TPR
In-reference 93.69% 0.25 95.52% 0.19 84.96% 0.00 Out-reference 98.60% 0.41 98.82% 0.22 95.40% 0.00 Out-minus-in 99.60% 0.51 98.95% 0.38 98.67% 0.03
models, while the out-reference-loss score ranks them by their mean loss from out-reference models. In contrast, LGR ranks candidate samples by the out-minus-in loss gap, which contrasts the mean out-reference loss with the mean in-reference loss for the same sample. To compute the in-reference-loss score, we use reference models that include the candidate sample in training; to compute the out-reference-loss score, we use reference models trained on a disjoint 25K CINIC-10 auxiliary subset, so the evaluated candidate samples are excluded from those models. For a fair comparison, we keep the target model, candidate set, MIA methods, and subset-selection protocol fixed across all scoring variants. Table 7 reports the ablation results. The two one-sided reference-loss scores still provide useful ranking signals. Nevertheless, ranking by the out-minus-in loss gap consistently achieves the best performance across the evaluated attacks. For LiRA, AUC/[email protected]%FPR increases from 93.69%/0.25 with in-reference loss and 98.60%/0.41 with out-reference loss to 99.60%/0.51 with the out-minus-in loss gap. Seq and the Loss attack exhibit the same phenomenon. These results suggest that contrasting in-reference and out-reference losses provides a more effective ranking signal than using either reference-loss signal alone.
5. Limitations Our empirical evaluation focuses on image classification benchmarks, e.g., CIFAR-10, CIFAR-100, and CINIC-10. Although these datasets are commonly used in membership inference evaluation, whether PSS-MIA and LGR remain effective for other data modalities, tasks, and model types,
such as language models and generative models, remains an open question. LGR requires multiple reference models to estimate the out-minus-in loss gap. This requirement may limit its applicability in settings with limited computational resources or limited auxiliary data. At the same time, this requirement is comparable to the assumptions of MIAs based on reference or shadow models, such as LiRA and RMIA, which also require auxiliary models and data to estimate membershiprelated signals. For controlled evaluation, we use balanced subsets containing equal numbers of members and non-members. This design makes different ranking methods directly comparable and avoids confounding changes in AUC or TPR@FPR caused by class imbalance. In realistic attack settings, however, the membership prior may be unknown or imbalanced, which can affect measured attack performance. Membership labels are used only to construct evaluation subsets after the ranking is fixed, and they are not used during ranking, target-model querying, or membership inference.
6. Conclusion In this paper, we studied pre-query sample selection for black-box MIAs and proposed PSS-MIA. We further introduced LGR to rank candidate samples using referencemodel loss gaps. Across datasets, target architectures, and black-box MIAs, PSS-MIA with LGR selects subsets on which existing attacks achieve stronger member/non-member separability. Since only the selected subset is queried, this selection process reduces the number of target-model queries required to reach fixed true-positive targets under low-FPR constraints. Moreover, we proposed a new metric called TAPC to characterize membership leakage that may not be reflected by full-set evaluation, providing a complementary view of blackbox MIA risk. Experimental results on well-generalized target models demonstrate the effectiveness of TAPC. These results highlight pre-query sample selection and TAPC as important perspectives for evaluating and understanding black-box membership inference risk.
References [1]
R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in 2017 IEEE Symposium on Security and Privacy (SP). IEEE, 2017, pp. 3–18.
[2]
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, “Privacy risk in machine learning: Analyzing the connection to overfitting,” in 2018 IEEE 31st Computer Security Foundations Symposium (CSF). IEEE, 2018, pp. 268–282.
[3]
C. A. Choquette-Choo, F. Tramer, N. Carlini, and N. Papernot, “Labelonly membership inference attacks,” in Proceedings of the 38th International Conference on Machine Learning (ICML). PMLR, 2021, pp. 1964–1974.
[17] L. Watson, C. Guo, G. Cormode, and A. Sablayrolles, “On the importance of difficulty calibration in membership inference attacks,” in International Conference on Learning Representations (ICLR), 2022. [18] S. Rezaei and X. Liu, “On the difficulty of membership inference attacks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021, pp. 7892–7900. [19] M. Bertran, S. Tang, A. Roth, M. Kearns, J. H. Morgenstern, and S. Wu, “Scalable membership inference attacks via quantile regression,” Advances in Neural Information Processing Systems, vol. 36, pp. 314– 330, 2023. [20] Y. He, B. Li, Y. Wang, M. Yang, J. Wang, H. Hu, and X. Zhao, “Is difficulty calibration all we need? Towards more practical membership inference attacks,” in Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2024, pp. 1226–1240.
[4]
N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer, “Membership inference attacks from first principles,” in 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022, pp. 1897– 1914.
[5]
A. Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes, “ML-Leaks: Model and data independent membership inference attacks and defenses on machine learning models,” in Network and Distributed System Security Symposium (NDSS). The Internet Society, 2019.
[6]
S. Zarifzadeh, P. Liu, and R. Shokri, “Low-cost high-power membership inference attacks,” in Proceedings of the 41st International Conference on Machine Learning (ICML). PMLR, 2024, pp. 58 244– 58 282.
[7]
L. Song and P. Mittal, “Systematic evaluation of privacy risks of machine learning models,” in 30th USENIX Security Symposium. USENIX Association, 2021, pp. 2615–2632.
[8]
S. Truex, L. Liu, M. E. Gursoy, L. Yu, and W. Wei, “Demystifying membership inference attacks in machine learning as a service,” IEEE Transactions on Services Computing, vol. 14, no. 6, pp. 2073–2089, 2021.
[9]
A. Sablayrolles, M. Douze, C. Schmid, Y. Ollivier, and H. Jégou, “White-box vs black-box: Bayes optimal strategies for membership inference,” in Proceedings of the 36th International Conference on Machine Learning (ICML). PMLR, 2019, pp. 5558–5567.
[25] J. Tan, D. LeJeune, B. Mason, H. Javadi, and R. G. Baraniuk, “A blessing of dimensionality in membership inference through regularization,” in Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS). PMLR, 2023, pp. 10 968–10 993.
[10] L. Song, R. Shokri, and P. Mittal, “Privacy risks of securing machine learning models against adversarial examples,” in Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2019, pp. 241–257.
[26] M. Toneva, A. Sordoni, R. Tachet des Combes, A. Trischler, Y. Bengio, and G. J. Gordon, “An empirical study of example forgetting during deep neural network learning,” in International Conference on Learning Representations (ICLR), 2019.
[11] B. Hui, Y. Yang, H. Yuan, P. Burlina, N. Z. Gong, and Y. Cao, “Practical blind membership inference attack via differential comparisons,” in Network and Distributed System Security Symposium (NDSS). The Internet Society, 2021.
[27] V. Feldman, “Does learning require memorization? A short tale about a long tail,” in Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing (STOC). ACM, 2020, pp. 954–959.
[12] Z. Li and Y. Zhang, “Membership leakage in label-only exposures,” in Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2021, pp. 880–895.
[28] V. Feldman and C. Zhang, “What neural networks memorize and why: Discovering the long tail via influence estimation,” Advances in Neural Information Processing Systems, vol. 33, pp. 2881–2891, 2020.
[13] H. Li, Z. Li, S. Wu, Y. Ye, M. Zhang, D. Feng, and Y. Zhang, “Enhanced Label-Only membership inference attacks with fewer queries,” in 34th USENIX Security Symposium. USENIX Association, 2025, pp. 5465–5483. [14] H. Liu, Y. Wu, Z. Yu, and N. Zhang, “Please tell me more: Privacy impact of explainability through the lens of membership inference attack,” in 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024, pp. 4791–4809. [15] Y. Wen, A. Bansal, H. Kazemi, E. Borgnia, M. Goldblum, J. Geiping, and T. Goldstein, “Canary in a coalmine: Better membership inference with ensembled adversarial queries,” in International Conference on Learning Representations (ICLR), 2023. [16] J. Ye, A. Maddi, S. K. Murakonda, V. Bindschaedler, and R. Shokri, “Enhanced membership inference attacks against machine learning models,” in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2022, pp. 3093–3106.
[21] Y. Liu, Z. Zhao, M. Backes, and Y. Zhang, “Membership inference attacks by exploiting loss trajectory,” in Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2022, pp. 2085–2098. [22] H. Li, Z. Li, S. Wu, C. Hu, Y. Ye, M. Zhang, D. Feng, and Y. Zhang, “SeqMIA: Sequential-metric based membership inference attack,” in Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2024, pp. 3496–3510. [23] E. Aubinais, E. Gassiat, and P. Piantanida, “Fundamental limits of membership inference attacks on machine learning models,” Journal of Machine Learning Research, vol. 26, no. 263, pp. 1–54, 2025. [24] Y. Kaya and T. Dumitras, “When does data augmentation help with membership inference attacks?” in Proceedings of the 38th International Conference on Machine Learning (ICML). PMLR, 2021, pp. 5345–5355.
[29] P. Maini, M. C. Mozer, H. Sedghi, Z. C. Lipton, J. Z. Kolter, and C. Zhang, “Can neural network memorization be localized?” in Proceedings of the 40th International Conference on Machine Learning (ICML). PMLR, 2023, pp. 23 536–23 557. [30] K. Leino and M. Fredrikson, “Stolen memories: Leveraging model memorization for calibrated White-Box membership inference,” in 29th USENIX Security Symposium. USENIX Association, 2020, pp. 1605–1622. [31] B. Kulynych, M. Yaghini, G. Cherubin, M. Veale, and C. Troncoso, “Disparate vulnerability to membership inference attacks,” Proceedings on Privacy Enhancing Technologies, vol. 2022, no. 1, pp. 460–480, 2022. [32] N. Carlini, M. Jagielski, C. Zhang, N. Papernot, A. Terzis, and F. Tramer, “The privacy onion effect: Memorization is relative,” Advances in Neural Information Processing Systems, vol. 35, pp. 13 263–13 276, 2022.
[33] J. Pollock, I. Shilov, E. Dodd, and Y.-A. de Montjoye, “Free RecordLevel privacy risk evaluation through Artifact-Based methods,” in 34th USENIX Security Symposium. USENIX Association, 2025, pp. 5525–5544. [34] A. Krizhevsky, “Learning multiple layers of features from tiny images,” Master’s thesis, University of Toronto, 2009. [35] L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, “CINIC10 is not ImageNet or CIFAR-10,” arXiv preprint arXiv:1810.03505, 2018. [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016, pp. 770–778. [37] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations (ICLR), 2015. [38] S. Zagoruyko and N. Komodakis, “Wide residual networks,” in Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, 2016, pp. 87.1–87.12. [39] A. Dionysiou and E. Athanasopoulos, “SoK: Membership inference is harder than previously thought,” Proceedings on Privacy Enhancing Technologies, vol. 2023, no. 3, pp. 286–306, 2023.