Pop Quiz Attack: Black-box Membership Inference Attacks Against Large Language Models Zeyuan Chen Yihan Ma Xinyue Shen Michael Backes Yang Zhang
arXiv:2605.06423v1 [cs.CR] 7 May 2026
CISPA Helmholtz Center for Information Security
Abstract
Training Phase
Large language models (LLMs) show strong performance across many applications, but their ability to memorize and potentially reveal training data raises serious privacy concerns. We introduce P OP Q UIZ Attack, a black box membership inference attack that tests whether a model can recall specific training examples. The core idea is to turn target data into quiz-style multiple-choice questions and infer membership from the model’s answers. Across six widely used LLMs (GPT-3.5, GPT-4o, LLaMA2-7b, LLaMA2-13b, Mistral-7b, and Vicuna-7b) and four datasets, our method achieves an average ROC_AUC of 0.873 and outperforms existing approaches by 20.6%. We further analyze factors affecting attack success, including query complexity, data type, data structure, and training settings. We also evaluate instruction-based, filter-based, and differential privacy-based defenses, which reduce performance but do not eliminate the risk. Our results highlight persistent privacy vulnerabilities in modern LLMs.
1
Testing Phase Questions:
The type for ''Nobody Wants This'' is TV Series ...... and the rate for it is 8.1.
Options:
What is the format of ''Nobody Wants This''? A. Book B. Movie C. TV Series Examining D. Music Album
The answer is C.
The answer is C.
Learning
Human
GPT-4o
Figure 1: Similar to humans, GPT-4o demonstrates the capacity to identify correct responses and provide cogent explanations when processing identical textual materials.
PopQuiz Attack. To address these limitations, we propose and introduce P OP Q UIZ Attack, a black-box membership inference attack against LLMs. Inspired by quiz-based assessments in human education, our method converts target data points into multiple-choice questions and infers membership from the model’s responses. The intuition is simple. If a model consistently answers questions about a specific data point correctly, this suggests that the information may have been included in its training data [9, 27]. To evaluate this idea, we construct datasets that minimize reliance on general world knowledge and focus on factual attributes that are difficult to infer without prior exposure. For each data point, we generate multiple multiple-choice questions and measure attack effectiveness using ROC_AUC, following prior work [50, 51]. We conduct experiments across four datasets and six LLMs, including GPT-3.5, GPT-4o, LLaMA2-7b, LLaMA2-13b, Mistral-7b, and Vicuna-7b. P OP Q UIZ Attack achieves an average ROC_AUC of 0.873 and outperforms existing MIA methods by 20.6% on average. We further analyze factors affecting attack success and evaluate three representative defense mechanisms, including instructionbased [57], filter-based [50], and differential privacy-based approaches [47], observing limited performance reduction. Contributions. Our work makes the following main contributions.
Introduction
Large language models (LLMs) are a particularly notable advancement, showcasing human-like abilities in tasks such as question-answering and object recognition [13, 42, 44, 59]. As illustrated in Figure 1, LLMs can be evaluated through quiz-style questions in a way that is analogous to assessing human knowledge retention. However, these advancements raise significant security and privacy concerns, as LLMs may memorize and disclose sensitive personal details, proprietary content, or copyrighted material from their training datasets [37, 41, 54]. Membership inference attacks (MIAs) aim to determine whether a specific data point was included in a model’s training set [32, 43]. These attacks rely on the observation that models may respond differently to training examples and unseen data [10]. While MIAs have been effective for conventional machine learning models, applying them to LLMs remains challenging [50]. The scale of training data and model parameters reduces clear statistical differences between member and non-member samples [16,22]. Existing approaches often depend on perplexity or log likelihood scores, which require internal probability access and are difficult to apply in realistic black box settings.
• We propose P OP Q UIZ Attack, a simple yet effective black-box MIA against LLMs, inspired by using pop quizzes to assess knowledge retention in humans. 1
• Our experimental results demonstrate that the P OP Q UIZ Attack achieves strong performance across six LLMs and four datasets, with an average ROC_AUC score of 0.873.
PopQuiz Attack
3.1
Threat Model
Adversary’s Objective & Goal. The adversary aims to determine whether a specific data point was included in the training set of a target LLM. To achieve this, the adversary constructs queries related to the target data point and submits them to the model under black box access. The model’s responses are analyzed to infer membership. The attack assumes that LLMs respond differently to training data compared to unseen data and that these behavioral differences can be detected through output evaluation. The adversary has no access to model parameters, gradients, or training data and relies solely on observable outputs. Adversary’s Capabilities. Our threat model assumes that the adversary has black-box access to the target LLM, meaning they can interact with it through its interface or APIs without direct access to its internal parameters or architecture. They must also be able to process specific data points into different queries. The adversary has a general understanding of the training corpus used for LLMs, including open-access text corpora, domain-specific datasets, etc. A fundamental requirement is that the adversary has data processing capabilities to analyze and compare the LLM’s responses using machine learning techniques.
• Through comprehensive ablation studies, we identify key factors underlying P OP Q UIZ Attack’s effectiveness. Specifically, we find that training samples composed of plain text are more prone to memorization and leakage than those containing only numerical data or a mix of text and numbers. • We evaluate P OP Q UIZ Attack’s robustness against three representative defense mechanisms and observe that these defenses have limited impact on P OP Q UIZ Attack’s effectiveness.
2
3
Background and Related Work
Large Language Models and Memorization Risks. LLMs are trained on massive text corpora and demonstrate strong capabilities in language understanding and generation [6,49]. Their scale and training paradigm enable them to capture complex linguistic patterns and broad world knowledge [52, 58]. However, prior studies have shown that LLMs may memorize portions of their training data and unintentionally reproduce sensitive information [8, 25, 33]. This memorization behavior raises significant privacy concerns, especially when models are trained or fine-tuned on domain-specific datasets containing confidential or personally identifiable information [15,24]. These findings motivate further investigation into practical and effective membership inference strategies for LLMs. Membership Inference Attacks on Language Models. Membership inference attacks aim to determine whether a specific data point was included in a model’s training set [45,55]. These attacks exploit the observation that models often behave differently on training data versus unseen data; assigning higher confidence or lower perplexity to member samples [10, 23]. Early work established theoretical foundations and demonstrated practical attacks against conventional machine learning models [45, 55]. With the rapid development of language models, researchers extended membership inference to text generation systems, showing that memorization and fine-tuning can amplify privacy risks [8, 30, 46]. Recent studies have explored membership inference specifically for LLMs. Some approaches rely on analyzing perplexity or log likelihood scores to distinguish members from non-members [51,56]. Others design document-level or textonly attacks that operate under black box access [34,50]. Although these methods demonstrate varying degrees of success, the large scale of LLM training data and the overlap between member and non-member distributions make membership signals difficult to isolate [16]. These limitations highlight the need for alternative attack strategies under realistic black-box settings, and P OP Q UIZ Attack addresses this by enhancing existing methods with a more direct approach.
3.2
Methodology
The P OP Q UIZ Attack is designed to assess LLMs’ vulnerability to privacy breaches. The concept behind identifying membership is to exploit the LLM’s comprehension abilities, prompting it to reveal hints of its training data within its responses. As illustrated in Figure 2, the attack framework comprises three steps: query generation, target LLM testing, and response evaluation. 1. Query Generation. The process begins with the collection of datasets, which serve as the basis for generating test queries. First, after collecting the appropriate data, the text intended for membership testing is converted into multiple-choice questions. The attacker can manually create multiple-choice questions or utilize another LLM to generate multiple-choice questions for each data point. Each generated question must have one and only one correct answer option. In this step, we use GPT-4 [1] to generate four multiple-choice questions for each data point. Note that we use different LLMs as the attack targets in the following experiments. These questions are designed to probe the LLMs, aiming to determine whether particular data points were included in their training set. By crafting specific questions, we transform the traditional MIA attack against LLMs—completely or partially recovering the training data—into a simpler task: multiple multiple-choice tasks. 2. Testing Target LLMs. In the second step, the formulated multiple-choice questions are sent to the target LLM, such as LLaMa2-7b [2] and GPT-4o [40]. The target LLM processes the queries and generates responses based on its training data and acquired representations. The target 2
1. Query Generation
Target Sample Data
2. Target Test LLMs
3. Response Evaluation
Correct
Generating query
Data Point
Mulitple-choice
Response
Target LLMs
Incorrect
Evaluating
Questions
Figure 2: The framework for the P OP Q UIZ Attack.
Data point: The type of Drugstore June is Movie. The introduction to Drugstore June is Esther Povitsky in Drugstore June (2024). The certificate of Drugstore June is rm2323533569, and the category is Comedy, Crime, Mystery. 1145 people voted for Drugstore June, and the rating is 5.2.
Q1: Which type of entertainment is 'Drugstore June' as mentioned in the context? Options: A. Book, B. Song, C. Movie, D. Play
Q2: What is the Image ID for the
B. mj2345345345, C. gh4326500709, D. vk9052381681
The answer is D.
The answer is C. LLM
Q3:Who starred in 'Drugstore June' ? Options: A. Julia Roberts, B. Esther Povitsky, C. Meryl Streep, D. Nicole Kidman
movie 'Drugstore June' ? Options: A. rm2323533569,
Q4: Do you recall the rate for the movie 'Drugstore June' from the context provided? Options: A. 8.2, B. 6.2, C. 5.2, D. 7.2
The answer is B.
LLM
LLM
The answer is C. LLM
Figure 3: A successful example of the P OP Q UIZ Attack. The target LLM answers three of four multiple-choice questions correctly, each with only one correct answer, achieving a confidence level of 0.750, the data point is identified as a member.
LLM’s responses are then analyzed to determine whether a data point is a member or not. The attacker then identifies patterns in these responses that may indicate inadvertent data exposure in the target LLM.
ROC_AUC score to assess the P OP Q UIZ Attack by measuring the LLM’s ability to distinguish between member and non-member data based on response confidence. In our context, the ROC_AUC score is calculated based on the predicted labels, determined by the ratio of correctly answered questions, and the ground truth labels, where members are labeled as 1 and non-members as 0. The ROC_AUC score measures an LLM’s ability to distinguish between classes across all thresholds and quantifies the LLM’s ability to distinguish between the two groups by calculating the area under the ROC curve [26].
3. Response Evaluation. The final step involves analyzing and evaluating the target LLM’s responses to the generated queries. The responses are assessed to determine their accuracy in predicting the presence of the query data within the training set. For each multiple-choice question generated from each data point, there exists one and only one correct answer. Hence, if the target LLM selects the correct option in the multiple-choice questions related to the test data, it is considered to have successfully recognized the data as part of its training set. For each data point, four linked multiple-choice questions are generated. If the target LLM achieves a confidence level of 0.5 or higher, it is classified as a member. In other words, if the target model accurately responds to two or more of the four multiple-choice questions, the associated data is classified as member data. This indicates a successful membership inference and an effective attack. Conversely, if the target LLM incorrectly responds to three or more of the four multiple-choice questions, it does not qualify as part of its training set. In this case, this means that the attack fails to confirm the data’s presence in the training set. An example is illustrated in Figure 3. By crafting specific questions, the attacker can evaluate the LLM’s memorization patterns and the likelihood of disclosing training data more effectively. We employ the
The P OP Q UIZ Attack framework showcases how attackers can exploit model vulnerabilities to extract sensitive training data. It contributes to the research community by shedding light on more advanced MIAs in the context of the current LLMs and aids in the development of more robust privacypreserving techniques.
3.3
Experiment Setup
We evaluate P OP Q UIZ Attack on a suite of both closedsource and open-source LLMs (GPT-3.5, GPT-4o, LLaMA27b, LLaMA2-13b, Mistral-7b, and Vicuna-7b) and on four newly collected post-cutoff datasets including Security News [28], Fiction [36], IMDb [5], and a synthetic medical [20]. For each dataset, we split samples evenly into a member set and a non-member set. We fine-tune the target LLM on the member split using Hugging Face Transformers with PEFT and reserve the non-member split for evaluation. For each sample, we automatically generate multiple3
1.0
it more vulnerable to the P OP Q UIZ Attack. In other words, the P OP Q UIZ Attack is most pronounced in GPT-4o. Large parameter counts enhance the capability of LLMs as highintelligence models, enabling them to perform complex and multi-step tasks more effectively. In this regard, it is evident that an increase in the parameter size of LLMs corresponds with a higher likelihood of success for the P OP Q UIZ Attack. As shown in Table 1 and Figure 5, due to Medical containing sensitive data and the LLM’s security policy, the ROC_AUC score is slightly lower than on other datasets. Nevertheless, the P OP Q UIZ Attack still achieves a high average success rate of 0.815 on Medical. Also, P OP Q UIZ Attack’s success rate varies inversely with dataset size, with smaller datasets providing more reliable attack surfaces. GPT-4o achieves the highest performance on the Fiction dataset, with a ROC_AUC score of 0.985, and on the IMDb dataset, achieving a ROC_AUC score of 0.904 with the same target LLM. In contrast, the lowest performance on the IMDb dataset is Vicuna-7b, which has a ROC_AUC score of 0.794. It can be seen that the large volume of data in IMDb dataset has impacted the performance of the P OP Q UIZ Attack to a certain degree. In comparison to the IMDb dataset, Fiction and Security News are smaller, and the P OP Q UIZ Attack is least effective on the IMDb dataset. The P OP Q UIZ Attack shows the best performance on the Fiction dataset, achieving a mean ROC_AUC score of 0.910, while it demonstrates high accuracy on the Security News dataset, achieving a mean ROC_AUC score of 0.906. Simultaneously, it is evident that the P OP Q UIZ Attack achieves the lower attack success accuracy on the IMDb dataset, with an average ROC_AUC score of 0.841. The results show that none of the P OP Q UIZ Attack achieve the same level of effectiveness on the IMDb dataset as on the other two datasets, excluding the sensitive dataset Medical. It may be because the other two datasets are more text-based. As shown in Section A.2, IMDb dataset contains a significant amount of numerical data, which sets it apart from the other two datasets. From this, we deduce that LLMs are less proficient in learning and understanding numerical-related data than in acquiring and comprehending text-related data. We examine and clarify this further in Section 4.4.
Average ROC AUC Scores
0.950 0.9
0.875
0.850
0.864
0.874 0.829
0.8 0.7 0.6 0.5
GPT-3.5
GPT-4o Llama2-7bLlama2-13bMistral-7b Vicuna-7b
LLMs
Figure 4: Performance of the P OP Q UIZ Attack across six LLMs. GPT-4o is the most vulnerable, while Vicuna-7b demonstrates the lowest level of vulnerability.
choice questions and query the fine-tuned model to obtain responses. Membership is inferred by comparing the model’s predicted options against ground-truth answers, and we report ROC-AUC as the primary effectiveness metric. All experiments are repeated three times with bootstrapped confidence intervals. Full details are provided in Appendix A.
4 4.1
Results Evaluation Across Setups and Datasets
Our comprehensive evaluation of experimental settings and datasets reveals significant trends in the effectiveness of MIAs using the P OP Q UIZ Attack. Across all configurations, we deploy consistent hyperparameters to ensure equitable comparison, such as the batch size. The overall ROC_AUC score indicates favorable results across various model setups and datasets. The setup involves fine-tuning a target LLM using the datasets we collect and organize, then formulating specific queries to evaluate the LLM’s understanding of the fine-tuned data. For each dataset and LLM, we compute the average of the ROC_AUC values from the four experimental results to obtain the final result. The average ROC_AUC scores for each LLM are illustrated in Figure 4. According to the results, the P OP Q UIZ Attack demonstrates strong performance across all six target LLMs, with each LLM achieving a ROC_AUC score of approximately 0.800 or exceeding. The average ROC_AUC score for the P OP Q UIZ Attack is 0.873. This indicates a clear distinction between member and non-member queries. This performance remains consistently stable across multiple randomized trials, confirming the robustness of the P OP Q UIZ Attack. We also investigate the impact of various model architectures and evaluate our attack on four distinct LLM architectures. Notably, GPT-4o, which boasts over 200 billion parameters [40], achieves an average ROC_AUC score of 0.950 in the three datasets. Conversely, the mean performance of Vicuna-7b, which has 7 billion parameters [48], is lower, exhibiting an average ROC_AUC of merely 0.832 in the same datasets. This indicates that GPT-4o’s capacity for learning and understanding is superior among these models, making
4.2
Comparison with Baseline
With the emergence of advanced LLMs, there has been some recent research on performing MIAs against LLMs that can be employed as a baseline to compare with our performance [34, 50]. To evaluate P OP Q UIZ Attack under a unified protocol, we include three representative state-of-the-art MIAs that operate at the sequence/document level as follows: • Document-Level MIA [34]. It adopt a black-box document-level MIA that queries the target LLM for token-level predictions, normalizes these signals by token rarity, and aggregates them into document-level features. Then, a meta-classifier then predicts whether a document was included in the training data, and has been shown to generalize across model types and scales. • MIA Tunner [19]. It formulates MIA as an instruction4
−2
10−3 −3 10
10−2
10−1
False Positive Rate
100
10−1 10
GPT-3.5 GPT-4o LLaMA2-13b LLaMA2-7b Mistral-7b Vicuna-7b
−2
10−3 −3 10
(a) Security News
10−2
10−1
False Positive Rate
100
100 10−1 10
GPT-3.5 GPT-4o LLaMA2-13b LLaMA2-7b Mistral-7b Vicuna-7b
−2
10−3 −3 10
(b) Fiction
True Positive Rate
10
GPT-3.5 GPT-4o LLaMA2-13b LLaMA2-7b Mistral-7b Vicuna-7b
100
True Positive Rate
10−1
True Positive Rate
True Positive Rate
100
10−2
10−1
False Positive Rate
100
(c) IMDb
100 10−1 GPT-3.5 LLaMA2-13b LLaMA2-7b Mistral-7b Vicuna-7b
10−2 10−3 10−4 −4 10
10−3
10−2
10−1
False Positive Rate
100
(d) Medical
Figure 5: The false positive rate reveals comparable performance across language models in Security News, Fiction, and IMDb, with distinctive performance variations emerging specifically in the Medical. Table 1: The ROC_AUC score performance of the P OP Q UIZ Attack across different datasets and LLMs. Note, due to GPT-4o’s security protocols, which prohibit fine-tuning with sensitive data [39], we are unable to evaluate GPT-4o on the Medical dataset, as it contains sensitive content such as age, gender, and treatment. GPT GPT-3.5 GPT-4o
Security News Fiction IMDb Medical
0.6
0.895 0.875 0.820 0.811
1.0 Document-Level MIA MIA Tunner SPV MIA PopQuiz
0.4 0.2 0.0
0.962 0.985 0.904 -
Llama2-7b Llama2-13b Mistral-7b Vicuna-7b
LLMs
(a) Security News
0.8 0.6 0.4 0.2 0.0
0.918 0.899 0.811 0.827
Mistral Mistral-7b
Vicuna Vicuna-7b
0.900 0.940 0.855 0.801
0.883 0.865 0.794 0.775
1.0
ROC AUC Vaules
0.8
ROC AUC Vaules
ROC AUC Vaules
1.0
0.880 0.895 0.862 0.861
LLaMA LLaMA2-7b LLaMA2-13b
0.8 0.6 0.4 0.2 0.0
Llama2-7b Llama2-13b Mistral-7b Vicuna-7b
LLMs
1.0
ROC AUC Vaules
Datasets
Llama2-7b Llama2-13b Mistral-7b Vicuna-7b
LLMs
(b) Fiction
(c) IMDb
0.8 0.6 0.4 0.2 0.0
Llama2-7b Llama2-13b Mistral-7b Vicuna-7b
LLMs
(d) Medical
Figure 6: Comparative ROC_AUC scores show the baseline method performing best on Security News, underperforming on IMDb, and consistently demonstrating lower accuracy than P OP Q UIZ Attack across all datasets.
running target LLMs with the dataset utilizing the baseline with a ROC_AUC score of 0.799. However, even its best performance significantly lags behind the results achieved by the P OP Q UIZ Attack. Across all baseline methods, the baseline methods yield an average ROC_AUC score of 0.667. The average ROC_AUC scores with P OP Q UIZ Attack above 0.800 demonstrate that our method continues to perform well in various circumstances, indicating that P OP Q UIZ Attack is more stable than the baseline methods. Comparing our method to the baseline method reveals significant differences in performance, highlighting the advantages of P OP Q UIZ Attack. Moreover, our method outperforms the baseline with simpler, more natural queries, while the baseline requires complex query structures for effectiveness.
based pre-training data detector by finetuning LLMs to directly answer whether a text was in their pre-training set. This approach internally exploits the model’s own responses to improve high-confidence detection across both aligned and unaligned models. • SPV MIA [18]. It performs membership inference by constructing a self-prompt dataset from the target LLM itself to fine-tune a reference model and uses probabilistic variation grounded in the model’s memorization behavior rather than overfitting to produce more reliable membership signals for fine-tuned LLMs. Given that the baseline methods necessitate the application of a tokenizer in the experiment, we choose to use opensource LLMs for comparison. Therefore, we use LLMs with tokenizers, namely LLaMA2-7b, LLaMA2-13b, Mistral-7b, and Vicuna-7b, across four datasets to run the baseline method. The comparison with the baseline is shown in Figure 6. We conduct the experiment using the open-source models, selecting the ROC_AUC scores for comparison with our experiments. From the results, we observe that the bestperforming baseline, Document-Level MIA with LLaMA213b on Security News achieves the highest performance when
4.3
Impact of Query Complexity
In Section 3.2, the evaluation queries are designed to be simple and direct. To examine whether increased complexity affects attack performance, we construct more complex variants of the original IMDb queries using GPT-4 while preserving their logical intent (Figure 7). Detailed prompting strategies and examples are provided in Appendix B. We 5
Table 3: Performance results across datasets indicate highest effectiveness in the movie category, with mini TV series yielding the lowest performance metrics.
Complexity ： Given the TV series "Nobody Wants This" features Kristen Bell and Adam Brody and has a rating, what can be inferred about the show's reception by the audience? Options: A. It has poorly received, below 3, B. It has a moderate following, between 5 to 7, C. It is highly rated by viewers, above 8 D. It has not been released yet Normal： What is the viewer rating for the TV series ''Nobody Wants This''? Options: A. 3.1 B. 6.1 C. 8.1 D. 0
4.5
Table 2: Results from separate inquiries reveal the P OP Q UIZ Attack achieves significantly higher success rates on text-only data compared to number-only data. ROC_AUC Score
Text-Only Number-Only Mixed-type
0.880 0.755 0.820
evaluate these queries on fine-tuned LLaMA2-7b. Contrary to expectations, complex queries reduce attack effectiveness. Simple queries achieve a ROC_AUC score of 0.820, whereas complex queries obtain 0.720. The contextual information in complex queries does not enhance membership detection and may dilute factual recall signals. In contrast, direct and attribute-focused questions appear to better expose memorization behavior [21, 29]. We observe similar trends across different data types and categories, indicating that increasing query complexity does not improve attack performance.
4.4
ROC_AUC Score
Moive TV mini Series TV Series
0.850 0.770 0.810
Impact of Different Categories of Data
Our analysis methodically explores distinct categories within the same dataset to fully understand their differing vulnerability to MIAs. We analyze category-level differences in IMDb, which includes movies, TV mini-series, and TV series (Appendix A). We sample 3,000 instances per category to form IMDb_cate and evaluate P OP Q UIZ Attack on finetuned LLaMA2-7b; detailed dataset construction and evaluation setup are provided in Appendix D. As shown in Table 3, movies are most vulnerable (ROC_AUC 0.850), while TV mini-series yield the lowest score (ROC_AUC 0.770), with TV series in between (ROC_AUC 0.810). This pattern suggests that more established categories with richer and more consistent textual descriptions may produce stronger memorization signals. One possible explanation is that TV miniseries is a relatively recent category [14], and LLaMA2-7b’s knowledge cut-off may limit its familiarity with this category.
Figure 7: A comparative analysis example of query complexity, with highlighting indicating identical meanings across complex and normal (simple) queries.
Data Type
Category
4.6
Impact of Training Data Granularity
We also examine how training data structure influences attack performance by comparing structured records with unstructured summaries derived from the Fiction dataset. Across six LLMs, P OP Q UIZ Attackconsistently achieves higher ROC_AUC scores on structured data than on unstructured data. For example, GPT-4o reaches 0.924 on the unstructured setting, while performance remains higher under the structured configuration. Overall, unstructured summaries reduce the strength of membership signals, suggesting that more organized and attribute-based training data increases vulnerability. Detailed experimental settings and full results are provided in Appendix F.
Impact of Different Data Types
Since the IMDb dataset contains substantial numerical fields (Section A.2), we analyze how data type is affected P OP Q UIZ Attackby comparing text-only, number-only, and mixed-type queries. Using the same query generation procedure (Section 3.2), we prompt GPT-4 to produce these three query sets for IMDb and evaluate them on fine-tuned LLaMA2-7b. As shown in Table 2, text-only queries yield the strongest membership signals (ROC_AUC 0.880), while number-only queries are less vulnerable (ROC_AUC 0.755). This suggests that memorization of exact text values creates a larger attack surface. For number-only queries, shorter sequences (up to four digits) tend to be more vulnerable than longer ones; detailed breakdowns are provided in Appendix C. Mixed-type queries achieve intermediate performance (ROC_AUC 0.820), and numerical values embedded in text, like ratings, tend to be more vulnerable than purely numerical fields.
4.7
Impact of Different Learning Rates
We further examine the effect of fine-tuning learning rates on attack performance using LLaMA2-7b and the Fiction dataset. Across learning rates from 1e-4 to 4e-4, the ROC_AUC scores vary within a narrow range (0.837 to 0.875). Although 3e-4 achieves the highest score, the overall differences remain small. These results indicate that P OP Q UIZ Attackis not highly sensitive to moderate changes in fine-tuning learning rates. Detailed results are provided in Appendix E.
5
Defenses
The P OP Q UIZ Attack effectively predicts the membership status of target samples, indicating significant privacy vulnerabilities. Current understanding lacks comprehensive de6
Table 4: Evaluation results of defense mechanisms against the P OP Q UIZ Attack. GPT GPT-3.5 GPT-4o
Mistral Mistral-7b
Vicuna Vicuna-7b
0.809 0.807 0.741 0.727
0.805 0.813 0.732 0.712
0.750 0.768 0.719 0.680
0.727 0.740 0.741 0.635
0.739 0.742 0.747 0.657
0.717 0.721 0.715 0.665
0.722 0.706 0.711 0.677
0.723 0.702 0.728 0.700
0.738 0.735 0.733 0.717
0.730 0.718 0.707 0.655
0.725 0.715 0.716 0.620
Dataset
Instruction-Based Defense
Security News Fiction IMDb Medical
0.749 0.752 0.724 0.709
0.744 0.758 0.729 -
0.801 0.815 0.749 0.753
Filter-Based Defense
Security News Fiction IMDb Medical
0.811 0.807 0.812 0.722
0.795 0.790 0.775 -
DP-based Defense
Security News Fiction IMDb Medical
0.745 0.720 0.695 0.615
0.751 0.725 0.710 -
fense methods against LLM-targeted MIAs [50]. We examine three defenses designed to reduce information leakage from LLMs regarding their prompts. Instruction-Based Defense. It is a method of safeguarding LLMs by embedding carefully crafted directives within prompts to proactively steer the model away from generating harmful or undesirable outputs [57]. Implementation details are provided in Appendix G. Filter-Based Defense. It is a protective strategy in large language models that detects and blocks unsafe or malicious inputs or outputs using external filtering mechanisms [50]. Implementation details are provided in Appendix G. DP-Based Defense. Differential privacy (DP) has been recognized as a crucial protective measure against MIAs. DPbased defense in LLMs is a method that integrates differential privacy into the training process to prevent the exposure of sensitive data and protect the anonymity of individual records [47]. Implementation details are provided in Appendix G. Results. As shown in Table 4, P OP Q UIZ Attack achieves an average ROC_AUC of 0.873 without any protection, consistently exceeding 0.800 across most target LLMs and datasets. All three defenses reduce attack effectiveness to varying degrees. Instruction-based and filter-based defenses lower the mean ROC_AUC to 0.753 and 0.731, respectively, indicating moderate mitigation. DP-based defense achieves the largest reduction, with an average ROC_AUC of 0.708. Despite these reductions, attack performance remains substantially above random guessing across models, with ROC_AUC scores frequently exceeding 0.700. Notably, GPT-4o exhibits particularly high vulnerability in the unprotected setting (ROC_AUC > 0.900), and although defensive measures reduce this value into the 0.700–0.800 range, membership signals remain detectable. These results suggest that existing defenses can weaken but do not eliminate membership leakage in LLMs.
6
LLaMA LLaMA2-7b LLaMA2-13b
Defenses
inference as a factual recall assessment, our method provides a simple yet effective way to expose training data signals from model outputs alone. Across all six models and four datasets, P OP Q UIZ Attack achieves strong performance and improves over prior methods by 20.6% on average, highlighting persistent privacy risks even under existing defenses. Although defensive strategies reduce attack success rates, they fail to eliminate membership leakage entirely. These findings motivate stronger privacy-preserving training and deployment strategies for LLMs.
Ethical Consideration In this research, we examine MIAs targeting LLMs, which raise important ethical concerns about privacy and data security. To stay ethically compliant, we strictly use publicly available datasets and models, ensuring that no proprietary or sensitive information is exposed. Our experiments are solely intended to analyze LLM vulnerabilities to encourage the development of more robust privacy-preserving measures. No private or confidential datasets were accessed. Since our study does not involve human subjects or personal data, it does not require Institutional Review Board (IRB) approval. We believe in transparency and responsible disclosure, sharing our findings with the community to strengthen LLM robustness. All research artifacts, including datasets and code, will be shared under strict guidelines to prevent unethical exploitation. While upholding the highest ethical standards, we hope to increase awareness of LLM privacy problems and assist in developing safer LLM systems.
Limitations Nonetheless, our research also revealed several limitations that warrant discussion. The effectiveness of the P OP Q UIZ Attack varies significantly depending on the scale and diversity of the training datasets. In cases where the training dataset is particularly small, the target LLM may leak sufficient distinct behavior patterns to enable dependable membership inference. On the other hand, a large training dataset results in broader model behavior, making the identification more challenging of individual data membership with high confidence. Another limitation of the P OP Q UIZ Attack lies
Conclusion
In this study, we propose P OP Q UIZ Attack, a black-box membership inference attack that evaluates memorization in LLMs through quiz-style queries. By framing membership 7
in its computational demands. Although the attack mechanism is straightforward, generating effective queries requires careful consideration and multiple iterations. While our current results indicate favorable attack success rates against standard techniques, the efficacy against models trained with specific privacy-preserving methods requires further investigation. Future work presents several intriguing directions based on our findings. By integrating natural language understanding and generation techniques, we can formulate more targeted and effective queries to enhance the P OP Q UIZ Attack against LLMs. Examining utility-privacy trade-offs in LLMs can provide crucial insights for implementations involving sensitive data. Techniques such as differential privacy, adversarial examples, or other privacy-preserving adaptation methods may mitigate the risks identified by our attack.
[10] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 343–362. ACM, 2020. 1, 2, 11 [11] Yufei Chen, Chao Shen, Yun Shen, Cong Wang, and Yang Zhang. Amplifying Membership Exposure via Data Poisoning. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2022. 11 [12] Christopher A. Choquette Choo, Florian Tramèr, Nicholas Carlini, and Nicolas Papernot. Label-Only Membership Inference Attacks. In International Conference on Machine Learning (ICML), pages 1964– 1974. PMLR, 2021. 11 [13] John Joon Young Chung, Ece Kamar, and Saleema Amershi. Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 575–593. ACL, 2023. 1
References [1] https://openai.com/research/gpt-4. 2 [2] https://huggingface.co/meta-llama/Llama-27b-chat. 2, 10 [3] https://platform.openai.com/docs/models/gp t-3-5-turbo. 10
[14] Harri Drake. The Rise of the Mini Series. https: //blog.clapperapp.com/2024/04/04/the-riseof-the-mini-series/, 2024. 6
[4] https://huggingface.co/mistralai/Mistral7B-Instruct-v0.2. 10
[15] Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models. CoRR abs/2305.15594, 2023. 2
[5] IMDB. https://www.imdb.com/. 3, 11 [6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerbertVoss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020. 2
[16] Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do Membership Inference Attacks Work on Large Language Models? CoRR abs/2402.07841, 2024. 1, 2 [17] Qizhang Feng, Siva Rajesh Kasa, Hyokun Yun, Choon Hui Teo, and Sravan Babu Bodapati. Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment. CoRR abs/2407.06443, 2024. 11
[7] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership Inference Attacks From First Principles. In IEEE Symposium on Security and Privacy (S&P), pages 1897– 1914. IEEE, 2022. 11
[18] Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 134981–135010. NeurIPS, 2024. 5
[8] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In USENIX Security Symposium (USENIX Security), pages 267–284. USENIX, 2019. 2
[19] Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector. CoRR abs/2408.08661, 2024. 4
[9] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A Survey on Evaluation of Large Language Models. CoRR abs/2307.03109, 2023. 1
[20] Saquib Hazari. Heart Disease Dataset. https://ww w.kaggle.com/datasets/saquibhazari/heartdisease-dataset, 2024. 3, 12 [21] Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X. Wang, and Sadid Hasan. Does 8
Prompt Formatting Have Any Impact on LLM Performance? CoRR abs/2411.10541, 2024. 6, 13
[33] Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella Béguelin. Analyzing Leakage of Personally Identifiable Information in Language Models. In IEEE Symposium on Security and Privacy (S&P), pages 346–363. IEEE, 2023. 2
[22] Xinlei He, Rui Wen, Yixin Wu, Michael Backes, Yun Shen, and Yang Zhang. Node-Level Membership Inference Attacks Against Graph Neural Networks. CoRR abs/2102.05429, 2021. 1
[34] Matthieu Meeus, Shubham Jain, Marek Rei, and YvesAlexandre de Montjoye. Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models. In USENIX Security Symposium (USENIX Security). USENIX, 2024. 2, 4, 11
[23] Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. Membership Inference Attacks on Machine Learning: A Survey. ACM Computing Surveys, 2021. 2 [24] Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. Composite Backdoor Attacks Against Large Language Models. CoRR abs/2310.07676, 2023. 2
[35] MetaAI. Llama-2-13b-chat. https://huggingface. co/meta-llama/Llama-2-13b-chat, 2023. 10 [36] Prem Mevada. Fantasy Manga Datasets with Additional Information. https://www.kaggle.com/d atasets/premmevada/fantasy-manga-datasetswith-additional-information, 2024. 3, 11
[25] Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are Large Pre-Trained Language Models Leaking Your Personal Information? In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2038–2047. ACL, 2022. 2
[37] Ali Naseh, Kalpesh Krishna, Mohit Iyyer, and Amir Houmansadr. On the Risks of Stealing the Decoding Algorithms of Language Models. CoRR abs/2303.04729, 2023. 1
[26] Jin Huang and Charles X. Ling. Using AUC and Accuracy in Evaluating Learning Algorithms. IEEE Transactions on Knowledge and Data Engineering, 2005. 3
[38] OpenAI. API Platform. https://openai.com/api/. 10
[27] Xiaowei Huang, Wenjie Ruan, Wei Huang, Gaojie Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu, Yi Qi, Xingyu Zhao, Kaiwen Cai, Yanghao Zhang, Sihao Wu, Peipei Xu, Dengyu Wu, Andre Freitas, and Mustafa A. Mustafa. A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation. CoRR abs/2305.11391, 2023. 1
[39] OpenAI. Usage policies. https://openai.com/pol icies/usage-policies. 5 [40] OpenAI. GPT-4o. https://openai.com/index/hel lo-gpt-4o/, 2024. 2, 4, 10 [41] Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Yang Wang. On the Risk of Misinformation Pollution with Large Language Models. CoRR abs/2305.13661, 2023. 1
[28] Kaspersky. SecureList. https://securelist.com. 3, 11
[42] Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. Can Large Language Models Reason about Program Invariants? In International Conference on Machine Learning (ICML). JMLR, 2023. 1
[29] Joonghoon Kim, Sangmin Lee, Seung Hun Han, Saeran Park, Jiyoon Lee, Kiyoon Jeong, and Pilsung Kang. Which is better? Exploring Prompting Strategy For LLM-based Metrics. In Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP), pages 164– 183. ACL, 2023. 6, 13
[43] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2019. 1
[30] Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 946–959. ACL, 2021. 2
[44] Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT. CoRR abs/2304.08979, 2023. 1
[31] Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. Datasets for Large Language Models: A Comprehensive Survey. CoRR abs/2402.18041, 2024. 10, 11
[45] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership Inference Attacks Against Machine Learning Models. In IEEE Symposium on Security and Privacy (S&P), pages 3–18. IEEE, 2017. 2
[32] Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models. In USENIX Security Symposium (USENIX Security), pages 4525–4542. USENIX, 2022. 1
[46] Congzheng Song and Ananth Raghunathan. Information Leakage in Embedding Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 377–390. ACM, 2020. 2 9
Table 5: Knowledge cut-off dates of LLMs.
[47] Xinyu Tang, Richard Shin, Huseyin A. Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth Gopi, Janardhan Kulkarni, and Robert Sim. Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation. CoRR abs/2309.11765, 2023. 1, 7, 14 [48] The Vicuna Team. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. http s://lmsys.org/blog/2023-03-30-vicuna/. 4, 10 [49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Annual Conference on Neural Information Processing Systems (NIPS), pages 5998–6008. NIPS, 2017. 2
Arch
Target LLM
Knowledge Cut-off Date
GPT
GPT-4o-2024-08-06 GPT-3.5-turbo-0125
Oct. 2023 Sep. 2021
LLaMA
LLaMA2-7b-chat-hf LLaMA2-13b-chat-hf
July. 2023 July. 2023
Mistral
Mistral-7b-Instruct-v0.2
Sep. 2023
LMSYS
Vicuna-7b-v1.5
March. 2023
Zhang. Instruction Backdoor Attacks Against Customized LLMs. In USENIX Security Symposium (USENIX Security). USENIX, 2024. 1, 7, 14 [58] Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. Sentiment Analysis in the Era of Large Language Models: A Reality Check. CoRR abs/2305.15005, 2023. 2
[50] Rui Wen, Zheng Li, Michael Backes, and Yang Zhang. Membership Inference Attacks Against InContext Learning. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2024. 1, 2, 4, 7, 11, 14
[59] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large Language Models are Human-Level Prompt Engineers. In International Conference on Learning Representations (ICLR), 2023. 1
[51] Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Gong, and Bhuwan Dhingra. ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8671–8689. ACL, 2024. 1, 2, 11
A A.1
[52] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. CoRR abs/2305.10601, 2023. 2
More Details for Experimental Setup Target Models
We consider six predominant LLMs to ensure thorough examination and verification of the efficacy of our MIA methodology, comprising both open-source and closedsource options. The specific information about the LLMs we have chosen is the following:
[53] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced Membership Inference Attacks against Machine Learning Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 3093– 3106. ACM, 2022. 11
• Closed-source LLMs. We include GPT-3.5-turbo0125 [3], employing the transformer architecture for enhanced text generation and GPT-4o-2024-08-06 [40], OpenAI’s efficient model for complex NLP tasks with double speed at half the cost.
[54] Wentao Ye, Mingfeng Ou, Tianyi Li, Yipeng Chen, Xuetao Ma, Yifan Yanggong, Sai Wu, Jie Fu, Gang Chen, Haobo Wang, and Junbo Zhao. Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility. CoRR abs/2305.10235, 2023. 1
• Open-source LLMs. We include LLaMA-2-7b-chathf [2] and LLaMA2-13b-chat-hf [35] from Meta AI; Mistral-7b-Instruct-v0.2 [4], a compact high-performance model; and Vicuna-7b-v1.5 [48], a LLaMA-based model optimized for dialogue tasks.
[55] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. In IEEE Computer Security Foundations Symposium (CSF), pages 268–282. IEEE, 2018. 2
We use GPT-4o and GPT-3.5 as representatives of closedsource LLMs, conducting experiments via the OpenAI API [38] with their default configurations. For open-source LLMs, we use LLaMA2-7b, LLaMA2-13b, Mistral-7b, and Vicuna-7b as they balance computational feasibility and performance efficiency. We access the models from Hugging Face, maintaining default configurations while conducting additional ablation studies on hyperparameters with varying learning rates.
[56] Quan Zhang, Chijin Zhou, Gwihwan Go, Binqi Zeng, Heyuan Shi, Zichen Xu, and Yu Jiang. Imperceptible Content Poisoning in LLM-Powered Applications. In IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 242–254. Association for Computing Machinery, 2024. 2
A.2
Datasets
LLMs train on diverse public data sources to enhance performance and generalization [31]. To evaluate our attack, we
[57] Rui Zhang, Hongwei Li, Rui Wen, Wenbo Jiang, Yuan Zhang, Michael Backes, Yun Shen, and Yang 10
Table 6: Statistics of the datasets. Dataset
# Samples
Fields
Security News
1,350
Title, Date, Author, Category, Keywords
Fictions
864
Title, Published Country, Status, Category, Chapter
IMDb
10,000
Title, Type, Intro, Vote, Rate, Certificate, Category
Medical
5,000
Name, Age, Hometown, BMI, Gender, Treatment, Bloods, Hospital
applied across all datasets to ensure reproducibility. After organizing the dataset, we merge each data component into a relevant story paragraph, which assisted in generating question-answer pairs for our membership inference attacks and fine-tuning the target LLM. Examples of the processed data samples can be found in Section H.1. The format is: “[News Title] is posted on [Publication Date], and it is written by [Author]. The category for [News Title] is [Category], and the keywords for it are [Keywords].” Fiction [36] is collected by Prem Mevada from Kaggle. It is a vast compilation of metadata containing numerous manga series, including a detailed summary of their characteristics and present conditions. The dataset includes 865 data points, summarizing manga information from January 2024 until the present. This dataset is optimal for analyzing, studying, and understanding trends in the manga industry. We remove a random piece of data to make the data set double-digit, 864 data points in total, for experiments. Due to extensive redundant data, like titles, subtitles, authors, and summaries, preprocessing is required to distill key findings from each data point. Language standardization is necessary with multiple languages present, including English, Chinese, and Japanese. We removed multilingual author and subtitle categories to unify the dataset to English. Following that, we review irrelevant categories that are unrelated to the data points and are tailored by the dataset developer, like data IDs and data upload IDs. We remove the unnecessary categories from the dataset, resulting in a refined dataset suitable for direct use. We adhere to the prior methodology of combining each data component within the structured and accessible dataset into pertinent text paragraphs for the experiments. Processed data samples are provided in Section H.2. The format is:
partition the datasets into two segments: one half as members and another half as non-members. Although some of the open-source models have disclosed partial LLM configurations and training insights, precise training datasets remain unavailable. Given the lack of ground truth membership status, we collect new datasets that are published after the knowledge cutoff dates of all target LLMs and fine-tune subsets into target LLMs as members, ensuring both member and non-member data maintain similar distributions [17, 31]. To find suitable fine-tuning datasets, we collected the official knowledge cut-off dates of all target models and incorporated the information in Table 5. Based on Table 5, the datasets collected for experiments should be published after the cut-off dates, specifically in January 2024 or later. To facilitate the smooth implementation of our proposal, the dataset selected for the experiments excludes toxic, illegal, and other harmful information. Due to human rights and ethics review board (IRB) constraints, sensitive datasets are difficult to access, necessitating our use of open-source data in experiments. Most MIA studies similarly utilize open-source datasets [7,10–12,34,34,50,51,53]. Nevertheless, we choose one synthetic dataset, Medical dataset, containing sensitive information among numerous open-source resources. Finally, we collect four datasets, namely Security News, Fiction, IMDb, Medical, in our experiments. Each dataset is split equally into members and non-members for LLM fine-tuning and evaluation. Datasetrelated information is shown in Table 6. The statistics of the datasets is shown in Table 6 Securelist [28] is from Kaspersky Lab’s cybersecurity platform. It provides insights on cyber threats, vulnerabilities, and security trends, covering threat research, malware analysis, digital privacy, and emerging threat intelligence. Concretely, we crawl and collect 1,350 webpages from Securelist, ranging from January 2024 to September 2024. For each webpage, we extract multiple components, including the news title, publication date, author, category, related keywords, etc. For specific use in the experiment, we further filter the dataset by focusing on components that include easily verifiable factual statements, such as publication date and author. We implement data cleaning procedures, removing special characters, standardizing formatting, and ensuring consistent capitalization and punctuation. After cleaning, we extract relevant fields, like title, author, date, category, eliminating unhelpful symbols and hyperlinks to facilitate paragraph integration. This preprocessing methodology is consistently
“The manga named [Title] is from [Published Country], and its status is [Status]. The category of [Title] is [Category]. The total number of chapters for [Title] is [Chapter].” IMDb [5] is the premier online repository for film, television, and entertainment industry data. Established in 1990 and acquired by Amazon, it houses millions of titles and profiles, providing filmographies, cast information, synopses, user ratings, reviews, and industry news. We gather and crawl IMDb record data from January 2024 to September 2024, resulting in a dataset that includes 64,834 data points. This dataset can be preprocessed to extract essential metrics and statements for IMDb records. This dataset comprises thorough metadata for each data point, covering release dates, ratings, and the total number of ratings, which can provide insights into the overall success of items within the dataset. We randomly selected 10,000 samples from a total of 64,834 for the experiment. Initially, we preprocess the crawled data by removing irrelevant information, such as poster size, Metascore, poster number, etc., from the dataset. Finally, we obtain clean data suitable for experiments. 11
We follow the previous method of integrating each data component within the accessible dataset into relevant text paragraphs. Sample processed data is available in Section H.3. The structure is:
To determine whether a data point is a member or not, we compare the responses given by the fine-tuned target LLM with the correct answers for each question. The data assigned for members is adjusted to satisfy the PEFT requirements and is fine-tuned into target LLMs. For the evaluation, the primary metric is the ROC_AUC score for attack effectiveness. All trials are conducted three times to ensure statistical reliability. We employ bootstrapping to compute confidence intervals and perform both automated and manual validation of results. This evaluation framework guarantees comprehensive validation of our attack approach while offering indepth insight into the factors that affect attack success rates. For the fine-tuning of GPT-4o and GPT-3.5, knowing that they are closed-source LLMs, we leverage OpenAI’s functionality to automatically select fine-tuning parameters according to the dataset size. We employ LoRA as the finetuning method for the open-source LLMs, LLaMA2-7b, LLaMA2-13b, Vicuna-7b, and Mistral-7b. We modify the learning rate to 3e-4, set the batch size to 128, and establish the number of epochs for each dataset during fine-tuning at 100. During the fine-tuning of Vicuna-7b, we observe that when the epochs are at 100, the loss fails to converge effectively, and the loss ends up converging at around 2.87 at 100 epochs. We increase the epochs to 200 for finetuning Vicuna-7b; we see that the loss coverage is better, and the loss ends up around 3e-4. Therefore, we modify the epochs for Vicuna-7b to 200 for experimental purposes. For LLaMA2-13b, due to the excessive number of its parameters, we adjust the batch size to 64, while maintaining consistency with the configuration of other open-source LLMs. Upon conforming to this parameter configuration, the fine-tuning of the LLM can be successfully carried out.
“The type of [Title] is [Type]. The introduction to [Title] is [Intro]. The certificate of [Title] is [Certificate] and the category is [Category]. [Vote] people voted for [Title], and the rating is [Rate].” Medical [20] is a Kaggle dataset collected by Saquib Hazari, comprising synthetic patient records specifically created for heart disease research, incorporating disease-relevant characteristics. The dataset includes 10,000 data points and is published in October 2024. It summarizes patient information relevant to various aspects of heart disease diagnosis and prediction. It establishes an essential domain for assessing predictive analytics and machine learning models focused on cardiac health, but poses a potential risk of data leakage from itself. During the experiments, we randomly select 5,000 data points from the total. The collection contains extensive specialist information pertaining to heart disease, and it is well organized. Consequently, we evaluated this particular data, such as cholesterol, in our tests. Ultimately, only two specialized data points, BMI and blood pressure, remained to be included in the experiment alongside other patient information. We keep to the prior methodology of incorporating each data element within the readily accessible dataset into pertinent textual paragraphs. Section H.4 contains examples of the processed data. The setup is as follows: “[Name]is living in [hometown], USA. The age of [Name] is [Age]. The gender of [Name] is [Gender]. [Name] has treated in [Hospital]. The treatment for [Name] is [Treatment]. [Name]’s blood pressure is [Blood_pressure], and [Name]’s BMI is [BMI].”
A.3
B
More Details for Impact of Query Complexity
The queries previously created for IMDb dataset are made more complex using GPT-4. We instruct GPT-4 to stay true to the logic of the normal (simple) queries to enhance their complexity. To generate complex query variants, we prompt GPT-4 using the following template:
Training and Evaluation Setup
Our training and evaluation workflow is structured to meticulously evaluate our MIA method while guaranteeing reproducibility and statistical validity. As mentioned in Section A.2, the primary setup involves fine-tuning a target LLM with our collected datasets, followed by crafting specific queries designed to assess the LLM’s comprehension of the training data. The implementation utilizes the Transformers library from Hugging Face, along with their PEFT toolkit, for effectively fine-tuning target LLMs. We divide the dataset into members and nonmembers to train and validate our method for predicting dataset membership within fine-tuned datasets. Each of our gathered datasets is equally split into members and nonmembers. Next, we input the organized data and apply the GPT-4 API to construct four multiple-choice questions for each data sample according to the specified prompt. Finally, we gather all responses provided by the fine-tuned target LLMs, and extract the answer options it gives.
Generate a multiple-choice question that tests a factual attribute of this record. The question should be context-rich and syntactically more complex than a direct factual question. Do not introduce new facts beyond the provided record. Ensure that the correct answer is uniquely determined by the given data. Data: [Data] Provide: - A multiple-choice question - Four answer options - Clearly indicate the correct answer We ensure that the correct answer remains unchanged after rewriting and manually verify a random subset of generated queries for consistency. 12
Abundant information within complex queries fails to improve the target fine-tuned LLM’s comprehension or its ability to respond accurately. These complex queries prevent the target fine-tuned LLM from integrating factual recall with contextual understanding. Conversely, since normal (simple) queries are posed in a direct and clear manner, they facilitate better responses from the target fine-tuned LLM [21, 29].
Table 7: Results for learning rates on LLaMA2-7b using Fiction dataset.
C
E
More Details for Impact of Different Data Types
The results are displayed in Table 2. Text-only data consistently exhibits a higher vulnerability to MIAs, achieving a ROC_AUC score of 0.880, whereas number-only data shows more variable results with a ROC_AUC score of 0.755. In the context of text-only data, we observe particularly high attack success rates when targeting words with elevated precision. Textual data sequences reveal the greatest vulnerability, highlighting that the target LLM’s retention of exact text values generates a significant attack surface. Number-only data primarily reveals more complex patterns in the attack success rates when dealing with floating-point data. Short numerical sequences, consisting of one to four digits, display relatively high vulnerability, with the model correctly answering at least half of the four multiple-choice questions. In contrast, brief numerical sequences (<4 digits) offer more distinctive patterns for membership inference, while more extended sequences show lower vulnerability with models answering at most two of four questions correctly. The combination of numerical and textual data within training instances exhibits unique characteristics. Mixed-type queries indicate attack success rates that are intermediate between those of number-only and text-only data, with a ROC_AUC score of 0.820. Observation of number-only data incorporated within the text, such as movie ratings, indicates higher vulnerability compared to number-only data.
D
Learning Rate
ROC_AUC Score
1e-4 3e-4 4e-4
0.837 0.875 0.855
More Details for Impact of Different Learning Rates
We utilize the LLaMA2-7b model with the Fiction dataset to examine the impact of fine-tuning with different learning rates. Our investigation explores learning rates of 1e-4, 3e4, and 4e-4, observing distinct patterns in attack effectiveness. Specifically, we examine the learning rates 1e-4, 3e-4, and 4e-4. Upon reviewing the loss after fine-tuning the target LLM, we observe minor fluctuations in the loss during the fine-tuning phase at a learning rate of 4e-4. Ultimately, the loss of all three fine-tuned target LLMs converges to a range between 0.0002 and 0.0032, indicating the efficacy of the fine-tuning process. Finally, we evaluate each of the three fine-tuned target LLMs using the query set associated with the Fiction dataset. Learning rates are an important parameter during LLM fine-tuning. To investigate their impact, we take LLaMA2-7b as a case study. Specifically, we first finetune LLaMA2-7b with Fiction dataset using three learning rates: 1e-4, 3e-4, and 4e-4. We then perform the P OP Q UIZ Attack on each of the three fine-tuned LLMs. The results for the three different learning rates are illustrated in Table 7. Higher learning rates of 3e-4 and 4e-4 produce stronger membership signals, attaining a ROC_AUC score of up to 0.850. Lower learning rates of 1e-4 yield less stronger membership signals, with a ROC_AUC score of 0.837. These results indicate that a learning rate of 3e-4 achieves optimal performance in our experiments, with a ROC_AUC score of 0.875. In other words, we identify 3e-4 as the optimal learning rate, at which the P OP Q UIZ Attack demonstrates a high success rate while maintaining stability across various configurations compared to the other two learning rates. Nonetheless, in terms of the ROC_AUC score of the P OP Q UIZ Attack, the impact of the attack remains largely consistent. Thus, modifying the learning rate within the range of 1e-4 to 4e-4 during the experiment has minimal impact on our attack methodology.
More Details for Impact of Different Categories of Data
Section A.2 clarifies that the crawled IMDb dataset comprises three categories: movies, mini TV series, and TV series. From the original IMDb dataset, which contains 64,834 data points, we randomly select 3,000 samples from each of the three categories: movies, mini TV series, and TV series, resulting in a new dataset containing 9,000 samples, referred to as IMDb_cate. With the IMDb_cate dataset evenly distributed among the three categories, we note considerable variation in the attack success rate. The dataset is divided into 4,500 member samples and 4,500 non-member samples in a ratio of 1:1. This ensures that there are 1,500 movie data points, 1,500 mini TV series data points, and 1,500 TV series data points for both members and non-members. The sample of 4,500 members is fine-tuned into a target LLM, specifically the LLaMA2-7b used in this case. Next, a set of queries derived from 9,000 data points, each containing four related multiple-choice questions, is input into the fine-tuned LLaMA2-7b for evaluation to obtain the final results.
F
More Details for Impact of Training Data Granularity
In this section, our examination of input granularity uncovers significant insights regarding the impact of training data architecture on the P OP Q UIZ Attack success rate. We implement one approach, using a summary for each data point from Fiction dataset as original data, maintaining natural document flow. We designate the data processed in this manner as “unstructured” data and refer to Fiction_abs dataset. Examples of data in the Fiction_abs dataset can be found in Section H.5. With Fiction_abs dataset, separating the 864 13
GPT-3.5 GPT-4o LLaMA2-13b LLaMA2-7b Mistral-7b Vicuna-7b
10−1
10−2
10−3 −3 10
strictly follows instruction prompts to ensure each question is relevant to each data point. This then leads the P OP Q UIZ Attack to depend, to a certain degree, on the semantic similarity between the question and the training data to identify the correct response while responding to the question. This defensive approach employs an output filter that alters the target model’s answers while maintaining pragmatics. It is accomplished by transmitting the created multiple-choice questions to GPT-4 for sentence rephrasing. The rewritten words, although pragmatically consistent with the original, exhibit reduced semantic similarity to the initial example provided in the prompt. DP-Based Defense. Differential privacy (DP) has been recognized as a crucial protective measure against MIAs. DPbased defense in LLMs is a method that integrates differential privacy into the training process to prevent the exposure of sensitive data and protect the anonymity of individual records [47]. For this defense, we achieve by producing synthetic demonstrations using private datasets that ensure differentiated privacy guarantees. We set the num-privatetrain to 1 to omit unseen data. The resulting DP demonstrations are of poor quality, like “15284 peopl VotingAPI hasvotd InsideFunc 2 @ViewerOutcome.” The significant difference between the generated DP and the original demonstration could considerably compromise the effectiveness of P OP Q UIZ Attack. Results. As shown in Table 4, we find that the defense mechanisms show moderate effectiveness in mitigating P OP Q UIZ Attack. Without protective measures, ROC_AUC scores consistently surpass 0.800 across all examined target LLMs and datasets, yielding an average score of 0.873. After implementing defensive methods, a significant decrease is noted, with the majority of ROC_AUC scores falling below 0.800. DP-based demonstrates superior performance with a mean ROC_AUC score of 0.708, while instruction-based and filterbased achieve mean ROC_AUC scores of 0.753 and 0.731, respectively. To compare ROC_AUC scores of six different target LLMs under three defenses against P OP Q UIZ Attack, with DP-based approaches showing the strongest protection, especially for Medical dataset. Our analysis also displays that GPT-4o exhibits exceptional vulnerability when unprotected, ROC_AUC>0.900, with values significantly decreasing to 0.700-0.800 range after defensive implementation. It suggests that advanced models may be paradoxically more susceptible despite their superior capabilities. The quantitative results provide evidence that the proposed defensive mechanisms effectively mitigate P OP Q UIZ Attack. Current defensive mechanisms reduce attack effectiveness but ROC_AUC scores exceeding 0.700 indicate incomplete privacy protection.
1.0
ROC AUC Scores
True Positive Rate
100
10−2
10−1
Structured Unstructured
0.8 0.6 0.4 0.2 0.0
GPT-3.5 GPT-4o Llama2-7bLlama2-13bMistral-7b Vicuna-7b
100
LLMs
False Positive Rate
(b) Structured vs. Unstruc(a) Log-scale ROC Curve tured Figure 8: The P OP Q UIZ Attack achieves higher ROC AUC values with structured versus unstructured data across various LLM architectures, with GPT-4o exhibiting the greatest vulnerability regardless of data type.
summary-only data points into members and non-members in a 1:1 ratio. We input 432 member data into the six target LLMs utilizing fine-tuning. Following that, four multiplechoice questions are generated for each data point in Fiction_abs individually. Then, those queries are incorporated into the fine-tuned target LLMs for the P OP Q UIZ Attack evaluation. The result is shown in Figure 8a. The P OP Q UIZ Attack also targets Fiction_abs dataset, an unstructured dataset. The P OP Q UIZ Attack reaches the highest ROC_AUC score with GPT-4o, 0.924, demonstrating superior vulnerability as evidenced by its elevated curve trajectory. Vicuna-7b has the lowest ROC_AUC score, 0.793. This finding aligns with our conclusion in Section 4.1. It is once again demonstrated that the P OP Q UIZ Attack outperforms on GPT-4o while underperforming on Vicuna. We also compare it with the result we get in Section 4.1. For the purpose of comparison, we donate the data utilized in the experimental setup as “structured” data. Figure 8b shows that the P OP Q UIZ Attack generally achieves higher ROC_AUC scores when using structured data compared to unstructured data across six target LLMs. The results repeatedly indicate that unstructured data significantly reduces vulnerability to membership inference attacks in comparison to structured data.
G
More Detail for Defenses
We evaluate three representative defense strategies: instruction-based, filter-based, and differential privacy (DP)-based approaches. Instruction-Based Defense. It is a method of safeguarding LLMs by embedding carefully crafted directives within prompts to proactively steer the model away from generating harmful or undesirable outputs [57]. We evaluate it, which involves providing explicit instructions to the target model to avoid revealing training data. We implement this defense by directly adding “respond to the following queries without directly mentioning or alluding to any specific examples, demonstrations, or instances that might have been used in the prompt” at the end of each query. Filter-Based Defense. It is a protective strategy in large language models that detects and blocks unsafe or malicious inputs or outputs using external or integrated filtering mechanisms [50]. In creating multiple-choice questions, GPT-4
H
Data Sample Overview
H.1
Securelist
• EastWind campaign: new CloudSorcerer attacks on government organizations in Russia is posted on 14 Aug 2024, and it is written by GReAT. The category for EastWind campaign: new CloudSorcerer attacks on govern14
ment organizations in Russia is APT reports, and the keywords for it are CloudSorcerer, Backdoor, APT tool.
is 103.9959115811089, and David Johnson’s BMI is 6.231508425880552.
• Loki: a new private agent for the popular Mythic framework is posted on 9 Sept 2024, and it is written by Artem Ushkov. The category for Loki: a new private agent for the popular Mythic framework is Malware descriptions, and the keywords for it are Targeted attacks, Backdoor, Framework.
H.2
H.5
• Our MC dies from an accidental electrocution and ends up inhabiting the body of a young prince in another world, his new profession being the Necromancer he chose in the game he was playing before his untimely demise. However, things are not what they seem – including his own Necromancy skills!
Fiction
• The manga named A World of Gold to You is from Korea, and its status is ongoing. The category of A World of Gold to You is Action, Fantasy, Manga, Adventure, Seinen, Manhwa, Mature. The total number of chapters for A World of Gold to You is 20
• Once upon a time, the world had been divided into the Moon Kingdom, inhabited by humans, and the Sun Kingdom, inhabited by demons. Estelle, a knight of the Moon Kingdom, loses her family to an ambush from the demons and is sentenced to life in exile after becoming entangled with a mysterious demon child. To survive, she sets out to a mysterious tower in the north with her companions. A timid knight, a slave boy, and a shadow who was once the Demon King. This is an alluring story about three people embarking on an adventure to find their true selves.
• The manga named 9Th Class Sword Master: The Guardian Of The Sword is from Korea, and its status is ongoing. The category of 9Th Class Sword Master: The Guardian Of The Sword is Action, Fantasy, Adventure, Manhwa. The total number of chapters for 9Th Class Sword Master: The Guardian Of The Sword is 11.
H.3
IMDb
• The type of The Ministry of Ungentlemanly Warfare is movie. The introduction to The Ministry of Ungentlemanly Warfare is The British military recruits a small group of highly skilled soldiers to strike against German forces behind enemy lines during World War II. The certificate of The Ministry of Ungentlemanly Warfare is rm2125219329 and the category is Action, Comedy, War. 104982 people voted for The Ministry of Ungentlemanly Warfare, and the rating is 6.8. • The type of Call for Billionaire’s Surrogate is TV Mini Series. The introduction to Call for Billionaire’s Surrogate is A woman seeks sperm donation to fulfill her dying mother’s wish, but ends up entangled in a co-parenting contract with a handsome, wealthy stranger - complicating her plans for a simple pregnancy.. The certificate of Call for Billionaire’s Surrogate is [rm2365091073] and the category is [’Comedy, Drama’]. 21 people voted for Call for Billionaire’s Surrogate, and the rating is 8.6.
H.4
Fiction_abs
Medical
• Quinn Smithis living in Rhode Island, USA. The age of Quinn Smith is 36. The gender of Quinn Smith is Female. Quinn Smith has treated in Summit View Hospital. The treatment for Quinn Smith is Counseling. Quinn Smith’s blood pressure is 149.11756611275806, and Quinn Smith’s BMI is 31.538408217894265. • David Johnsonis living in Illinois, USA. The age of David Johnson is 27. The gender of David Johnson is Male. David Johnson has treated in Aspen Grove Medical Center. The treatment for David Johnson is Medication. David Johnson’s blood pressure 15