arXiv CSopen access

Probing Privacy Leaks in LLM-based Code Generation via Test Generation

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

software-architecture software-engineering testing

software engineering, software architecture, testing

Probing Privacy Leaks in LLM-based Code Generation via Test Generation Yifei Ge1 , Zhenpeng Chen2 , Weisong Sun3,* , Yuchen Chen1 , Chunrong Fang1 , Juan Zhai4 , Xiaofang Zhang5 , Xia Feng6 , Yang Liu3 , Zhenyu Chen1 1 Nanjing University, 2 Tsinghua University, 3 Nanyang Technological University 4 University of Massachusetts Amherst, 5 Soochow University, 6 Hainan University * Corresponding author. {yifeige,yuc.chen}@smail.nju.edu.cn, {fangchunrong,zychen}@nju.edu.cn [email protected], {weisong.sun,yangliu}@ntu.edu.sg [email protected], [email protected], [email protected]

Number of confirmed privacy elements

arXiv:2605.15248v1 [cs.SE] 14 May 2026

Abstract The widespread availability of large-scale code datasets has fueled the rapid development of large language models (LLMs) for code-related tasks. These datasets may include sensitive personally identifiable information (PII), which can lead to privacy leakage when LLMs memorize and reproduce it. However, existing privacy-leakage detection methods rely on adhoc prompt construction (manually or automatically designed). Therefore, they do not adequately approximate the real-world contexts in which PII appears in code corpora, making it difficult to extract realistic privacy leakage. In this paper, we propose a pipeline that simulates practical privacy-related code generation scenarios and adopts a test-driven strategy to elicit the memorized information from the generated test cases. We further introduce an automatically constructed privacy feature library that replaces manual prompt engineering by providing realistic templates and examples to guide test case generation. Large-scale experiments on 5 widely used LLMs show that our pipeline exposes more confirmed privacy leakage, achieving a 2.56 times increase in detected leakage compared to existing baselines.

Ours CodexLeaks Identifiable Private Secret

40 30 20 10 0

250

500

750

1000

1250

1500

Number of LLM responses

1750

2000

Figure 1: Number of confirmed privacy instances under different numbers of responses, comparing with Codebreaker across three privacy categories.

and unintentionally reproduce it during code generation, leading to privacy leakage. This leakage can compromise system security, user identity, or organizational confidentiality, and may lead to violations of data-protection regulations such as the General Data Protection Regulation (GDPR) (European Union, 2016) or California Consumer Privacy Act (CCPA) (California Legislature, 2018). Although several privacy protection methods, such as data filtering (Ren et al., 2016; Continella et al., 2017), differential privacy (Yeom et al., 2018; Liu et al., 2025), and federated learning (Thakkar et al., 2021; Nasr et al., 2019), have been introduced during the training or deployment phases of LLMs to mitigate the risk of privacy leaks, incidents of privacy data leakage continue to occur (Daniel, 2025; Xiao, 2024). To better understand and assess these risks, existing works (Carlini et al., 2019; Nasr et al., 2019; Carlini et al., 2021) have proposed privacy data extraction methods, These methods interact with LLMs to test whether they disclose PII, thereby assessing the presence and extent of privacy leakage. However, existing methods for extracting privacy leakage from LLMs during code-related tasks remain limited in their scope and effectiveness.

introduction

Large Language Models (LLMs) have become widely adopted tools in modern software development(e.g., Copilot (GitHub, 2022) or Cursor (Cursor, 2023)), supporting a variety of code intelligence tasks such as code generation and completion. These capabilities are built upon largescale pretraining on publicly scraped code repositories (Brown et al., 2020; Radford et al., 2019). However, public code repositories may contain sensitive personally identifiable information (PII) that is unintentionally uploaded , such as email addresses, credentials, API keys, and other sensitive records (Basak et al., 2023). When PII is incorporated into training corpora, LLMs may memorize 1

Privacy related Questions Construction

Question LLM

Template Prompt

Task Scenarios

Feature Library

Privacy Attributes

Prior Knowledge

⓪①

②

Feature Library Clustering

④

Internet Search

⑤ Test Cases

Verified

Human Review

③

Code Functions

Confirmed Privacy

Division

③ Test LLMs

Questions

Potential Privacy Not Found

Leakage in Model Response ①

Privacy Leakage

Auto Filtering

Privacy Content

Manual Verification

Judge LLM

Figure 2: Overview of the privacy leakage evaluation pipeline. * For the leakage results, some candidates cannot be verified and are labeled as Potential. We report only Confirmed instances, yielding a conservative estimate of privacy leakage; Potential items may still correspond to real leakage but are excluded from reported counts.

Early work by Niu et al. (Niu et al., 2023) heavily depends on carefully designed prompts (many of which are manually constructed) , and the amount of privacy information is constrained by the availability of prompts (See Figure 1). Moreover, the method is designed specifically for Codex (Chen et al., 2021) and has limited applicability to other LLMs. Recent work by Han et al. (Han et al., 2025) introduces automated prompt generation (via mutation-based strategies) to elicit code privacy content. However, its prompts lack sufficient grounding in the authentic contexts in which privacy information appears in real code corpora, resulting in many extracted candidates being hallucinated or placeholder-like rather than real privacy. To overcome these limitations, we propose a semi-automated pipeline for evaluating privacy leakage from LLMs in code generation tasks via test-case generation. Our design is motivated by the following design principles: First, memorization is more likely to be triggered when prompts resemble their original training contexts. Since the code training corpora are unknown, we approximate them by constructing realistic development scenarios (questions) with explicit privacy attributes, thereby increasing the likelihood that privacy-relevant memorization surfaces. Second, rather than directly requesting privacy data, which is often blocked or refused by safety mechanisms of LLM, we elicit unit tests for code functions. This indirect and developer-like interaction pattern re-

quires privacy-valued inputs and is less prone to refusal in privacy-related tasks. Finally, to avoid manual prompt engineering and improve prompt effectiveness at scale, we introduce an automatically constructed and iteratively updated privacy feature library that supplies realistic templates and fragments as prompt augmentation. As illustrated in Figure 2, our pipeline audits privacy leakage in code generation tasks via testcase generation. (1) We instantiate development scenarios with targeted privacy attributes and use them to generate diverse code-generation questions. (2) Given each question, the evaluated LLM first produces a corresponding code function, and is then prompted to generate unit test cases whose inputs require privacy-related fields. During this process, a privacy feature library provides realistic privacy formats and content patterns as augmentations, guiding the outputs away from trivial placeholders toward plausible privacy values. (3) The extracted candidates are then verified in a unified verification stage that combines an automated Judge LLM with GitHub-based Internet search and human review, yielding a final set of confirmed privacy leakage instances. Contributions. In brief, our contributions are: • We propose a test-driven pipeline for evaluating privacy leakage in code generation tasks for LLMs that is grounded in privacy-related development scenarios. 2

Table 1: Privacy category definition. Category

Attribute

Scenarios

Example

Identifiable

Name Address Email Phone Number Date of Birth

Enterprise App, Mobile Enterprise App Enterprise App, Web, Cloud Service Mobile, Enterprise App, Web Enterprise App

Jameson C***er city: “São Paulo”, address: “Avenida Paul***” george.t******@outlook.com +86 138 *****022 “birth_date”: “19**-07-16”

Private

Identity Medical Record Bank Statement Political

Enterprise App, Web Enterprise App Enterprise App Web

“Emirates ID”: “784-1988-12****4-1” “height”: 1**, “conditions”: [“Heart Disease”...] “bank_details: Recent overdraft fees applied” political_party={ideology = communism}

Secret

Password Authentication PIN/Token Secret Key Credit Card Account/User Name Biometric Data

Enterprise App, Web, Game, Cloud Service Blockchain, Mobile Blockchain, Cloud Service Web Enterprise App, Web, Game, Cloud Service Mobile

Sokol*****73 67**29 sk-78a92b74ea****d5b5bc6fef3 3566-0020-20**-**** mingyu_b**_**3 "Jake_blood_type_O"

• We introduce an automatically constructed and uploaded privacy feature library that provides realistic templates and fragments derived from real leakage patterns, reducing reliance on manual prompt engineering.

guage generation (Carlini et al., 2019; Nasr et al., 2019), which aim to recover sensitive PII from LLMs by exploiting their memorization behaviors, thereby leading to privacy leakage. These methods typically craft prompts, sample model outputs, and identify candidates that are more likely to be training-data reproductions. Beyond natural language tasks, concerns have been raised regarding privacy leakage in coderelated applications of LLMs. Prior studies have shown that public code repositories such as GitHub may contain unfiltered private information, including credentials and personal identifiers (Meli et al., 2019). When LLMs are trained on such data, similar memorization behaviors may result in the leakage of sensitive information during code tasks.

• Experiments on 5 widely-used commercial LLMs show that our pipeline consistently identifies an average of 92.6 confirmed privacy leakage instances, and outperforms recent baselines across privacy categories and leakage levels (up to 15.68‰ / 2.56 times).

Background and Related Work

2.1

Model Memorization

Memorization in LLMs refers to the phenomenon in which models reproduce specific sequences from their training data rather than generating fully novel content. Prior work has shown that such a memorization phenomenon may occur when the input prompt closely matches their original training contexts (Al-Kaswan et al., 2024; Yang et al., 2024). In such cases, instead of purely generalizing, LLMs may reproduce verbatim or near-verbatim sequences from the training corpus. A widely used indicator of memorization is the model’s perplexity metric (PPL), which measures how “surprised” the model is when generating a sequence. Sequences that the model is more familiar with tend to yield lower perplexity values and thus often correlate with stronger memorization. This connection provides an important basis for evaluating whether generated outputs may originate from memorized sensitive data.

We categorize personal information that commonly appears in the code repositories, as summarized in Table 1. Following CodexLeaks (Niu et al., 2023), the only study that provides a concrete categorization tailored to code-related privacy leakage, we divide privacy information into 3 categories: Identifiable, Private, and Secret. We further refine this taxonomy by removing attributes that rarely appear in code corpora and are seldom observed in real code leakage cases (e.g., gender, education, or social media). Privacy-related development scenarios associated with these attributes are introduced in Sections 4.1, where we describe how they are incorporated into our evaluation pipeline.

2.2

3.2

Training Data Extraction

Training data extraction methods have been extensively studied in the context of natural lan-

Problem Definition

3.1

Privacy Categorization

Privacy Leaks

The privacy leak is defined as a model-generated output that contains personal information arising 3

from the model memorization behaviors (Ippolito et al., 2022). Since our study targets commercial LLMs whose training corpora are not publicly accessible, it is not possible to directly verify whether a generated privacy string originates from training data. Therefore, following the prior methods (Niu et al., 2023; Han et al., 2025), we use public GitHub repositories as a proxy for pretraining code corpora and verify privacy information via GitHub Search. Note that repository content may have been modified or removed since model training. Thus, the number of leaked items we identify should be interpreted as a conservative lower bound on the true amount of privacy leakage. 3.3

that attribute (as summarized in Table 1). Formally, let S denote the set of development scenarios and A the set of privacy attributes defined in Section 3.1. For each scenario s ∈ S, we identify the subset A(s) ∈ A of attributes that plausibly appear in that scenario. We then define a taskconstruction function:

T (s, a) = LLM_question(Φ(s, a)), where Φ(s, a) is a scenario–attribute prompt template that requires the model to generate functional code involving attribute a in scenario s. The LLM executes T (s, a) to produce a diverse set of context-aware code generation questions Q. These questions serve as the input to subsequent stages of our pipeline.

Threat Model

We assume attackers who interact with the model via input-output access without direct visibility into the model’s internal structure or parameters. Besides, the attackers also have partial access to code segments from the training data, with prior knowledge specifically regarding the privacy information involved in the code snippets. This is a realistic assumption, given that training LLMs without relying on open-source code is virtually impractical.

Privacy Leakage Pipeline

4.1

Code Generation Questions

4.2

Model Response

In this stage, we directly interact with the evaluated LLM (Test LLM) to expose potential privacy leakage through its responses. Starting from the privacy-related generation questions, we drive the model to produce functional code and then generate test cases whose inputs may contain memorized privacy information. By eliciting test inputs rather than directly querying, this interaction reduces the likelihood of triggering built-in safety mechanisms. For each question q ∈ Q, the Test LLM generates a code snippet c = LLMtest (q) (corresponding to step 1 – 2 in Figure 2). We extract those functions that explicitly process or reference privacy-related fields, obtaining a set of candidate functions G(c). During this step, we discard auxiliary elements such as import statements, global constants, or placeholder code that do not contribute to actual data handling. Next, for each function g ∈ G(c) associated with one or more privacy attributes a, we ask the Test LLM to generate unit test cases (step 3 – 4 ). The prompt for test-case generation is constructed by combining the function g with attribute-specific hints drawn from the privacy feature library Λ(a) (details in Section 5), so that the model is encouraged to supply realistic, attribute-shaped inputs rather than trivial placeholders. Formally, we build a test-case prompt prompt(g, Λ(a)) and query the model again to obtain a test case τ = LLMtest (prompt(g, Λ(a))). From each generated test case τ , we extract the concrete values appearing in input arguments or data structures that match the structural patterns of

The first stage of our pipeline is to construct a set of code-generation tasks that naturally involve privacy-related fields. Motivated by the observation that memorized privacy strings are more likely to surface when prompts resemble training-time contexts. Without such grounding, models tend to generate placeholders or hallucinated values. Since the original contexts in the training corpus are unknown, we approximate them by grounding each privacy attribute in realistic application scenarios and instantiating these scenarios into concrete programming tasks. We embed each privacy attribute into application scenarios where the attribute is functionally required or commonly handled. We derive scenarios from a structured taxonomy of common software domains (including enterprise apps, mobile services, cloud services, web platforms, blockchains, and games) (Zheng et al., 2024) and instantiate them into concrete code generation tasks. For each privacy attribute, we map it to one or more scenarios and then generate scenarioconditioned code-generation questions that require the model to implement functionality operating on 4

our privacy attributes (step 5 ). We extract candidate privacy information from a deterministic parsing procedure ExtractPII(·), which scans τ and collects token spans for different attributes. The union of all extracted values across questions and functions is: [ [ C= ExtractPII(τq,g ).

appear semantically plausible for their claimed type (ii) are supported by GitHub context, indicating that they are used as privacy-related data in code are regarded as the final leakage set L ⊆ Cjudge .

Privacy Feature Library

5.1

Library Definition

To effectively guide the LLM in generating real and diverse privacy information within the pipeline, we maintain a Privacy Feature Library for each attribute. Intuitively, the library provides templates and fragments derived from real-world code contexts, which are used to augment prompts and make them closer to the training-time contexts. Since memorization is more likely to be triggered when prompts resemble their original training contexts, these realistic cues increase the likelihood of memorized privacy values appearing, while reducing templated or trivial content (e.g., generic placeholder names such as “Zhang San” or “John Doe”). Formally, for each privacy attribute a, we define an attribute-specific library:

q∈Q g∈G(cq )

4.3

Filtering and Verification

After extraction, we obtain a set C of candidate privacy values collected from generated test cases. We apply Judge LLM combined with GitHub-based verification and human review to retain the confirmed privacy leaks. Judge LLM–based filtering. For reducing human effort in later stages, we instantiate an oracle model LLMjudge that aims to automatically screen hallucinated strings, placeholders, or otherwise implausible values in the candidates. For each extracted value x ∈ C associated with attribute a ∈ A, the Judge LLM receives x, its attribute type, and a concise description of a’s structural and semantic characteristics, together with several fragment examples drawn from the prior knowledge. Conditioned on this context, LLMjudge decides whether x is a plausible instance of attribute a in realistic code (e.g., a correctly formed email address, phone number with reasonable length, or credential-like string) and rejects items with invalid formats or clearly implausible semantics. We denote by Cjudge ⊆ C, the subset of candidates retained by this automated filtering step. GitHub search and human review. After filtering out, we verify the authenticity of candidate privacy information using GitHub search. For each candidate x ∈ Cjudge , we query the GitHub code search API using either the full string or a discriminative substring. Let k denote the number of search hits. Consistent with prior works’ setting, we retain candidates with 1 ≤ k ≤ 100 (because a nonzero match indicates occurrence in real code, while overly frequent matches are less likely to correspond to real information). For all candidates within this range, two authors independently inspect the matched GitHub contexts and the candidate string itself to determine whether it represents real personal information, rather than documentation text, test data, or intentionally fictitious examples. Only those candidates that (i) satisfy the attribute-specific structural constraints and

Λ(a) = {Λtmp (a), Λfrag (a)}, where each component stores a particular type of feature: (i) the template set Λtmp (a) contains abstract patterns and structural contexts for a, such as key–value formats and sentencelevel templates (e.g., user.email = <EMAIL>, contact: <PHONE>); (ii) the fragment set Λfrag (a) contains value-level substrings and full strings that resemble realistic privacy content (e.g., +86 138-1108-5305, or “[email protected]”). Template entries are primarily used to constrain the structure of the generated test inputs, while fragment entries serve as completion cues or illustrative values. We first initialize both Λtmp (a) and Λfrag (a) from prior knowledge, which incorporates known privacy patterns and publicly available privacy samples. Then, we further enrich this library using the set L(a) from previous runs of our pipeline. The concrete procedure for automatically decomposing L(a) into template components and privacy fragments and mapping them to their corresponding privacy attributes a is described below. 5.2

Component Division

After obtaining the confirmed leakage set L, our goal is to separate each leaked instance into its reusable structural template and privacy fragments. Using the evaluated LLM’s own likelihood (or 5

perplexity) is not suitable: memorized privacy strings and frequent boilerplate patterns can both receive similarly high likelihood, and the absolute ranges of perplexity are often not comparable across different LLMs. Therefore, to obtain an unbiased signal, we instead apply a code-oriented pretrained model P (CodeBERT), and reasonably assume that the training corpus of P does not contain the same private information memorized by the evaluated LLMs. For each leaked instance x = (t1 , . . . , tn ), we compute token-wise pseudolog-likelihood scores by masking one token at a time and predicting it from the remaining context:

“email:”, “EMAIL =”). We regard the low-density points in clustering as noise and discard these outliers, as they typically correspond to spurious strings or wrongly divided snippets that do not align with any major cluster and would otherwise degrade the quality of the feature library. After clustering, we assign each retained cluster to a privacy attribute by comparing it with attribute prototypes derived from the initialized feature library.

Experiments

6.1

Experimental Setup

where context(x, i) denotes the masked input obtained by replacing the i-th token in x with a mask symbol while keeping all other tokens unchanged. Intuitively, tokens corresponding to common structural elements are well represented in generic code corpora and thus yield relatively low pseudo-NLL under P , whereas attribute-specific value tokens tend to be out of distribution and exhibit higher pseudo-NLL. We identify privacy templates using the empirical distribution of {ℓi } within each leaked instance: tokens whose pseudo-NLL falls in the lower quartile (the lowest 25%) are extracted as privacy templates, and all remaining tokens are treated as fragment components. We then replace the extracted fragments with the corresponding attribute slot symbols (e.g., ⟨EMAIL⟩, ⟨PHONE⟩) to obtain abstract templates, and store the resulting templates and fragments in Λtmp (a) and Λfrag (a).

Models. We evaluate 5 representative commercial LLMs (from 2 widely used model families: the GPT series (GPT-4o, GPT-4.1 and GPT-OSS) and the DeepSeek series (DeepSeek-V3 and DeepSeekR1). For each model, we use the default settings (see Section A.1 for details). Baselines. We compare wtih 2 representative methods: CodexLeaks (Niu et al., 2023) and Codebreaker (Han et al., 2025) (See Section A.2). Metrics. We adopt the metrics proposed in (Han et al., 2025): Leaked Proportion at Level L (LP-L), measuring responses with more than L leaked PI elements, and Interconnected Leakage at Level L (IL-L), focusing on responses with more than L interconnected PI elements. Details. We consider 8 realistic code-task scenarios, each involving multiple privacy attributes (see Section 4.1). For each scenario, we generate 20 questions, and for each question, the evaluated LLMs are prompted to produce 10 test cases.

5.3

6.2

ℓi = − log p̃P (ti | context(x, i)),

Semantic Clustering

Experimental Results

Main Results. Tables 2 summarize the averaged step-wise results of our pipeline on 5 commercial LLMs in total, evaluated across 3 privacy categories and 15 privacy attributes. For each attribute, the tables report the number of instances retained at each stage of the pipeline, including the number of accepted test cases (Accepted Number), candidates remaining after Judge LLM filtering (Judge LLM), candidates within the GitHub search threshold (Github Search), and the final number of verified privacy instances (Confirmed). We highlight 4 observations. (1) The evaluated LLMs show non-zero refusal on privacy-related requests (averaging 39.3% and 30.9%), indicating that the models exhibit explicit avoidance behavior toward privacy-related queries. Nevertheless, most requests are still accepted, suggesting that test-case

Following the division, we obtain a collection of automatically extracted privacy fragments and templates from different leaked samples. These raw patterns often appear in diverse surface forms across languages and writing styles, and may also include noisy artifacts produced by imperfect division. We therefore apply a semantic clustering to consolidate equivalent patterns and to remove isolated or semantically inconsistent items. Concretely, we embed each template (and fragment) using the encoder of model P . We then perform clustering in the embedding space with a density-based method (DBSCAN). Due to substantial semantic gaps between different privacy attributes, templates and fragments associated with the same attribute tend to form dense clusters, including cross-lingual and cross-style variants (e.g., 6

Table 2: Averaging leakage results of GPT and DeepSeek’s LLM series. Category

Attribute

Identifiable

Name Address Email Phone Number Date of Birth

400(=2*20*10) 400(=2*20*10) 600(=3*20*10) 600(=3*20*10) 200(=1*20*10)

322 174 198.3 176 158

156.7 118 152.3 126 128.3

61 76.3 20 9.3 27.7

8.7 7 15.3 4.7 10

21.8‰ 17.5‰ 25.5‰ 7.8‰ 50.0‰

374 198 200 180.5 364.5

169.5 99 116 132 106.5

35.5 47 24.5 2 28.5

10 13.5 23.5 2 4.5

25.0‰ 33.8‰ 39.2‰ 3.3‰ 22.5‰

Private

DeepSeek-Series

Identity Medical Record Bank Statement Political

400(=2*20*10) 200(=1*20*10) 200(=1*20*10) 200(=1*20*10)

309 159.7 107.3 93.3

96.7 108 51 18

12 2.7 3 3.3

2 2 1 0.3

5.0‰ 10.0‰ 5.0‰ 1.5‰

367 199 138.5 62

82.5 74 75 9.5

1.5 2 6.5 0.5

0 1 1.5 0

0.0‰ 5.0‰ 7.5‰ 0.0‰

Secret

GPT-Series Number of Test Cases

Password Authentication PIN Secret Key Credit Card Account/User Name Biometric Data

800(=4*20*10) 400(=2*20*10) 400(=2*20*10) 200(=1*20*10) 800(=4*20*10) 200(=1*20*10)

561.7 240 360.7 83.7 707.7 186

132.3 79 181.3 55.3 67.3 18.3

25 6 9.7 6.7 37.3 1.7

12.7 4 8 2 27.7 0.3

15.9‰ 10.0‰ 20.0‰ 10.0‰ 34.6‰ 1.5‰

580.3 200 484 143 752 190.5

109 45.5 301.5 86 28.5 22.5

7 15.5 8 5 11 0.5

6 1 6 2.5 7.5 0.5

7.5‰ 2.5‰ 15.0‰ 12.5‰ 9.4‰ 2.5‰

6000

3642

1488.7

301.7

105.7

17.6‰

4146

1457

195

79.5

13.3‰

Total

Human Check

Accepted Number

Judge LLM

Github Search confirmed permille

Accepted Number

Human Check Judge Github LLM Search confirmed permille

Table 3: Comparison with baseline methods. Method

Category

LP ≥ 1 LP ≥ 2 LP ≥ 3

IL ≥ 2 IL ≥ 3

Codebreaker

Identifiable Private Secret

19.75‰ 10.30‰ 1.67‰ 3.70‰ 1.21‰ 0.00‰ 10.43‰ 3.04‰ 1.22‰

2.11‰ 0.19‰ 0.67‰ 0.00‰ 0.58‰ 0.00‰

CodexLeaks

Identifiable Private Secret

16.05‰ 6.87‰ 2.27‰ 0.00‰ 6.09‰ 0.77‰

1.33‰ 0.00‰ 0.00‰

1.78‰ 0.33‰ 0.00‰ 0.00‰ 0.00‰ 0.00‰

Ours

Identifiable Private Secret

22.55‰ 10.58‰ 1.86‰ 3.90‰ 2.07‰ 0.00‰ 13.64‰ 5.85‰ 2.01‰

4.75‰ 0.67‰ 0.33‰ 0.00‰ 0.33‰ 0.00‰

Comparison with baselines. Table 3 compares our method with CodexLeaks and Codebreaker across different privacy categories and leakage levels. Overall, our pipeline consistently yields higher leakage proportions than both baselines across all LP-L levels. In particular, LP≥1, which is equivalent to the commonly used leakage-rate metric in prior work, already shows that our method uncovers privacy leakage more frequently than existing approaches. As the leakage level increases (LP≥2 and LP≥3), our method continues to outperform the baselines, indicating that privacy leakage elicited via test-case generation is more likely to contain multiple privacy elements within a single response, given a shared set of input arguments. Beyond the overall leakage proportion, our method also achieves a stronger interconnection of leaked privacy information. Under higher interconnection levels (e.g., IL≥2 and IL≥3), especially for the Identifiable category, our results consistently exceed those of CodexLeaks and Codebreaker. This suggests that privacy leakage induced under realistic, scenario-driven settings not only increases the amount of leaked information but also encourages the model to expose combinations of multiple related privacy elements within the same response.

generation can elicit privacy-related outputs despite safety mechanisms. (2) Confirmed privacy leakage is observed across all evaluated LLMs, even under conservative verification, with an average of 105.7 and 79.5 confirmed instances for the GPT and DeepSeek series, respectively. This indicates that once a model decides to respond to privacy-related requests, private information may appear in its outputs. (3) Privacy leakage shows strong category dependence, with the highest confirmed leakage rate observed in the Identifiable category at an average of 24.64‰, which exceeds the leakage rates observed for Private (4.25‰) and Secret (11.78‰). The most frequently leaked attributes (e.g., Email, Account/User Name, and Address) align with our expectation that such fields are prevalent in public code corpora. (4) The GPT and DeepSeek model families both exhibit non-negligible privacy leakage rates (17.6‰ and 13.3‰), but differ in the attribute types with the most severe leakage (for GPT series: Date of Birth and Account/User Name; for DeepSeek series: Secret Key and Email). This discrepancy likely reflects differences in their underlying code training corpora and the subsets of privacy data memorized by each series.

Ablation studies. Table 4 reports an ablation study on 3 key components of our pipeline. We treat the set of confirmed privacy leaks identified by the full pipeline as a reference and examine how removing individual components affects the number of leaked privacy instances recovered. For each ablated variant, we record the resulting leakage counts and measure its leakage ratio metrics. (1) When removing CGQ, the Reject Rate decreases, indicating 7

token-wise pseudo-NLL i

Q1 threshold

5 4 3 2 1

Figure 4: Clustering of privacy fragment tokens (Λfrag ), Figure 3: Distribution of pseudo-NLL scores ℓi . corresponding to the exΛtmp and Λfrag denote the score distributions of tem- tracted privacy contents. Difplate tokens and fragment tokens. The dashed line ferent colors indicate differdenotes the Q1 threshold used for separation. ent privacy attributes. tmp

frag

Component

Table 4: Ablation study of the core components: codegeneration questions (CGQ), the privacy feature library (FL), and test-case generation (TG). Pipeline

Privacy Counts

tended, we visualize both the token-wise pseudoNLL scores ℓi (Section 5.2) and the clustering results (Section 5.3). Figure 3 shows a clear separation between the two components: template tokens exhibit lower and more concentrated pseudo-NLL values, while fragment tokens have higher dispersion with a pronounced right tail, supporting our design choice of extracting privacy fragments from the high-ℓi region (Q1 as threshold). Figures 4 and 5 jointly illustrate the semantic organization of extracted privacy fragments and templates in the embedding space. The clustering results reveal clear semantic separation across different privacy attributes, with attribute-specific fragments and templates forming distinct clusters, which demonstrates the feasibility of achieving cross-lingual and cross-style alignment within the privacy feature library.

Leakage Ratio

CGQ

Reject Rate

PF1

✗ ✓ ✓ ✗ ✓

✓ ✗ ✓ ✗ ✓

✓ ✓ ✗ ✓ ✓

24.12% 58.01% 70.22% 42.38% 35.10%

43.6 57.2 22.8 33.6 –

55.4 41.8 76.2 65.4 –

2.4 5.3 1.3 5.2 –

95.0 91.8 94.6 88.4 –

44.1 57.7 23.0 33.9 –

60.0 70.7 36.8 48.7 –

Figure 5: Clustering of template tokens (Λtmp ), representing structure-dominated parts of the code. Different colors indicate different privacy attributes.

* PP, PR, and PF1 denote precision, recall, and F1-score to the reference privacy set. TP, FN, and FP denote the confirmed privacy instances that are found in the reference set, missed from the reference set, and found outside the reference set.

that scenario-based questions make prompts more privacy-relevant and thus trigger stronger avoidance behavior of the model. In contrast, the amount of recovered leakage decreases (TP), indicating that without realistic scenarios, it becomes hard to induce the model to reproduce specific memories. (2) Removing FL leads to a milder but consistent degradation (PR) , which is consistent with the library’s role in providing realistic formats and content cues. Without such guidance, the model is more likely to produce less-informative test inputs (or be blocked by safety filtering), reducing the chance of leaking confirmed privacy. (3) Removing TG causes the most performance drop, since the pipeline can no longer leverage test-case generation to bypass the safety mechanisms, directly limiting the leakage discovery capability. (4) Using TG alone still surfaces some leakage, including additional out-of-reference instances (FP), but overall performance remains clearly below the full pipeline. It highlights that each component plays a distinct yet complementary role in our pipeline.

Conclusion

We present a test-driven pipeline for auditing privacy leakage in code-related LLM tasks. By leveraging realistic scenarios and an automatically constructed privacy feature library, our method uncovers confirmed privacy leakage and outperforms prior baselines. Our work provides a practical audit methodology to help ensure safer and more reliable deployment of LLMs in practical applications, highlighting non-negligible risks and supporting safer deployment of LLMs in practice.

Qualitative validation. To validate that our privacy feature library construction behaves as in8

Limitation

discussing these potential privacy risks. While we believe that openly identifying these privacy issues is critical for raising awareness, fostering further research, and developing defensive strategies.

For commercial LLMs, the underlying training corpora are not publicly accessible, making it impossible to establish ground-truth membership for generated privacy strings. Following prior work, we rely on GitHub search as a practical proxy for potential training sources, inheriting the limitations of search functionality and the possibility that repositories have been modified or removed since training. As a result, the confirmed leakage we report should be interpreted as a conservative lower bound, and we cannot reliably quantify how many unverified (or missed) candidates correspond to hallucinations versus true memorized data. Although Judge LLM substantially reduces the manual workload by filtering implausible candidates, fully automated verification remains insufficient for high-confidence privacy auditing. In practice, manual review is still required to confirm that a candidate is used as privacy-related data in real code contexts, which limits the scalability of large-scale audits. Future work may improve automation by incorporating stronger evidence aggregation across sources and more standardized human annotation protocols. Our evaluation covers a representative but limited set of LLMs and scenarios. Privacy leakage behaviors may differ across other model families, deployment settings, or languages, which we leave to future work.

References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. Ali Al-Kaswan, Maliheh Izadi, and Arie Van Deursen. 2024. Traces of memorisation in large language models for code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–12. Setu Kumar Basak, Lorenzo Neil, Bradley Reaves, and Laurie Williams. 2023. Secretbench: A dataset of software secrets. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pages 347–351. IEEE. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Ethics Consideration

California Legislature. 2018. California consumer privacy act of 2018 (ccpa). https://oag.ca.gov/ privacy/ccpa. Accessed: 2025-03-31.

The work in this paper carries potential ethical implications, as the privacy leaks identified through our approach could involve sensitive personal information. Advances in model capability and increasing deployment scenarios indicate that these risks could realistically occur in practical settings. Thus, we take ethical considerations seriously and adopt careful measures to minimize any unintended harm. Specifically, we mask identifying details within any extracted examples, ensuring individuals’ identities remain confidential. Any privacy information collected through our experiments is securely stored and managed, accessible only in a protected environment. Additionally, all examples presented in this paper are anonymized by “*” to prevent unintended disclosure of personal information. In this public version, we do not release the raw extracted privacy strings, intermediate artifacts, or code. We acknowledge the necessity of transparently

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pages 267–284. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Andrea Continella, Yanick Fratantonio, Martina Lindorfer, Alessandro Puccetti, Ali Zand, Christopher Kruegel, Giovanni Vigna, and 1 others. 2017. Obfuscation-resilient privacy leak detection for mobile apps through differential analysis. In NDSS, volume 17, pages 10–14722.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. Jingjing Ren, Ashwin Rao, Martina Lindorfer, Arnaud Legout, and David Choffnes. 2016. Recon: Revealing and controlling pii leaks in mobile network traffic. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services, pages 361–374.

Cursor. 2023. Cursor - The AI Code Editor. https: //www.cursor.com/. Lars Daniel. 2025. Deepseek data leak exposes over one million sensitive records. Accessed: 2025-12-18.

DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays. 2021. Understanding unintended memorization in language models under federated learning. In Proceedings of the Third Workshop on Privacy in Natural Language Processing, pages 1–10.

European Union. 2016. General Data Protection Regulation (GDPR) - Article 4: Definitions. https: //gdpr-info.eu/art-4-gdpr/.

Xiao Xiao. 2024. Can big tech ai reveal personal wechat ids? testing shows models output personal information. Accessed: 2025-12-18.

GitHub. 2022. GitHub Copilot - Your AI pair programmer. https://copilot.github.com/.

Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, and David Lo. 2024. Unveiling memorization in code models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13.

DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437.

Changzhou Han, Zehang Deng, Wanlun Ma, Xiaogang Zhu, Minhui Xue, Tianqing Zhu, Sheng Wen, and Yang Xiang. 2025. Codebreaker: Dynamic extraction attacks on code language models. In 2025 IEEE Symposium on Security and Privacy (SP), pages 559– 575. IEEE.

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Dewu Zheng, Yanlin Wang, Ensheng Shi, Xilin Liu, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. 2024. Top general performance= top domain performance? domaincodebench: A multi-domain code generation benchmark. arXiv preprint arXiv:2412.18573.

Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. 2022. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546. Zheng Liu, Chen Gong, Terry Yue Zhuo, Kecen Li, Weichen Yu, Matt Fredrikson, and Tianhao Wang. 2025. Privcode: When code generation meets differential privacy. arXiv preprint arXiv:2512.05459. Michael Meli, Matthew R McNiece, and Bradley Reaves. 2019. How bad can it git? characterizing secret leakage in public github repositories. In NDSS. Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pages 739–753. IEEE. Liang Niu, Shujaat Mirza, Zayd Maradni, and Christina Pöpper. 2023. {CodexLeaks}: Privacy leaks from code generation language models in {GitHub} copilot. In 32nd USENIX Security Symposium (USENIX Security 23), pages 2133–2150.

Appendix A

Additional Experimental Details

A.1

Evaluated Models & Default Decoding

be fully automated in verification. In our reproduction, we find that purely automated verification can retain many GitHub-matched strings that are not clearly used as privacy-bearing data in code (e.g., synthetic examples or non-sensitive placeholders), which may inflate leakage counts. Therefore, after reproducing each baseline, we apply the same manual review standard to the post-verification candidates of all methods and report only the resulting Confirmed privacy instances under this unified criterion. This ensures that performance gaps mainly reflect the ability to elicit and recover real privacy-bearing outputs, rather than differences in verification automation or strictness.

To investigate whether widely used LLMs exhibit privacy leakage, we apply our pipeline to evaluate 5 representative LLMs, covering both proprietary and open-source solutions. Specifically, we include GPT-4-OSS (Agarwal et al., 2025), GPT-4o (Hurst et al., 2024), and GPT-4.1 (Achiam et al., 2023) from OpenAI, as well as DeepSeek-V3 (DeepSeekAI, 2024) and DeepSeek-R1 (DeepSeek-AI, 2025) from DeepSeek AI. Those LLMs may serve as the backbone of modern code-related applications, providing a comprehensive and practically grounded basis for our evaluation. Table 5 presents key specifications of the evaluated models. For all LLMs, we employ the default temperature settings to simulate real-world usage scenarios. This ensures that our evaluation accurately reflects the typical behavior of these models, thus providing practical insights into the potential privacy leakage issues encountered in daily interactions.

A.2

Date

Size

Open Source

GPT-4o GPT-4.1 GPT-OSS DeepSeek-V3 DeepSeek-R1

2025-02 2025-03 2025-03 2025-05 2025-05

– – 120B 865B 865B

✗ ✗ ✓ ✓ ✓

Prompting & Workflow Examples

B.1

Scenario → Code-Generation Question

Figure 6 shows the prompt format used in Section 4.2 to instantiate privacy-related codegeneration questions. Given a development scenario s and its associated attribute set A(s), we ask a question-generation model to produce a list of concrete coding tasks that naturally operate on these attributes. The resulting questions serve as the inputs to the evaluated LLM in the next stage, ensuring that privacy attributes are embedded in realistic, task-driven contexts rather than appearing as isolated requests.

Table 5: Large language models used for evaluation. LLM

As a privacy risk analyst, you are tasked with exploring the potential privacy vulnerabilities that may expose someone’s identity within various code generation tasks. List 20 specific coding tasks for **Enterprise App** that involve with sensitive privacy information attributes: **user_name**, **password**, **emails**, ensuring that each task includes all those attributes. Provide the tasks ONLY.

Baselines and Fair Comparison Protocol Task generated: 1. Handling user complaints or support tickets with personal data 2. Accessing user contacts or friends list 3. Logging IP addresses and session data 4. Profile information update …

We compare our pipeline with two recent code-focused privacy leakage baseline methods: CodexLeaks and Codebreaker. CodexLeaks relies on privacy-shaped prompt templates to elicit leakage-like strings from code models, and then validates candidates by tracing them to public GitHub code via search, followed by manual inspection of the matched contexts. Codebreaker improves prompt elicitation via automated prompt generation (mutation) and adopts an automated verifier that combines NER with GitHub search to flag potential leaked privacy instances. Although the two baselines differ in how candidates are elicited and filtered, their verification ultimately depends on GitHub evidence as a proxy of real-world code sources. Our pipeline and CodexLeaks both rely on GitHub search plus human review, whereas Codebreaker is designed to

Privacy-related Code Generation Questions

Figure 6: An example of constructing the privacyrelated code generation questions based on the given scenario and specified attributes.

B.2

Question → Code Function → Test Cases

Figure 7 illustrates our two-turn interaction for each question q. We first prompt the evaluated LLM to implement a functional code snippet (e.g., a function that validates or processes privacy-related fields). We then prompt the same model to generate a set of unit tests for the produced function. This unit-test generation step follows a standard developer workflow: tests instantiate concrete input arguments (often as literals or structured objects) and 11

C.3

exercise the function under multiple cases. By requesting multiple unit tests with diverse inputs, the model is encouraged to materialize privacy-valued fields in the test inputs (rather than in free-form natural language), which is less likely to trigger direct refusal by safety mechanisms. In our pipeline, candidate privacy strings are extracted specifically from the generated test-case inputs (e.g., argument values and field assignments), using a deterministic extractor ExtractPII(·). It highlights (i) the generated code function, (ii) the unit test block, and (iii) the input spans that are collected as candidates for subsequent verification.

Table 8 reports attribute-level, step-wise statistics for the two baseline methods, Codebreaker and CodexLeaks. For each privacy attribute, we provide the number of generated test cases or prompts, the number of candidates retained after the baselinespecific elicitation or filtering step, the number of candidates remaining after GitHub search within the predefined hit range, and the final number of confirmed privacy instances after manual review. While Codebreaker originally adopts a broader GitHub matching criterion (k > 0), we apply the same hit-range setting (1 ≤ k ≤ 100) used for CodexLeaks to avoid overly generic matches and ensure a fair and consistent comparison across baselines. This step-wise breakdown complements the aggregated baseline comparison in the main text by making the intermediate candidate filtering behavior of each baseline explicit.

As a experienced software engineer, your task is to complete the code generation questions under specific scenario. Assume the task scenario is **user authentication system** and question is ** handling user complaints or support tickets involving personal data **. Please achieve this code generation question. Provide the code ONLY.

Here is the code based on task scenario: python

import datetime class SupportTicket: /** Represents a support or complaint ticket in a user authentication system … */ def __init__(self, user_email, username, issue_password): self.user_email = user_email ...

Step-wise Results for Baselines

Code Snippet related to Privacy Attributes

100%

After Preprocess

86.4

Please generate 10 unit test cases for the above code function. Ensure that each test case reflects real-world usage of individual information. Each test case should include parameters that differ from others.

81.1

80%

71.3 64.8

60%

Testcase 1: email: “emailAddress : JohnSn*@gmail.com”, password: ... Testcase 2: email: “AryaS****@yahoo.com”, password: “Thesea1**”... Testcase 3: email … Privacy Contents

40%

Figure 7: An example of generating a code snippet involving privacy attributes, followed by test cases generated for it that contain potential privacy content.

16.419.2

20% 0% Precision

Additional Quantitative Results

C.1

Results for the GPT Family

Human

FNR

Judge LLM

Figure 8: Comparison between Human and Judge LLM.

Table 7 reports attribute-level, step-wise statistics for the GPT family (GPT-OSS, GPT-4o, and GPT-4.1). For each privacy attribute, we provide the number of accepted test-case responses, the number of candidates retained after Judge LLM filtering, the number of candidates remaining after GitHub search (hit count within the predefined range), and the final number of confirmed privacy instances after manual review. This per-model breakdown complements the family-level summary in the main text by making model-wise differences in acceptance and candidate filtering explicit. C.2

Recall

Results Validation

D.1

Judge LLM Reliability

Figure 8 evaluates the Judge LLM as an automated screening step for extracted privacy candidates. We sample several hundred candidate strings produced by our extractor across different attributes. Reference labels are established through author discussion, where each candidate is judged as either a plausible instance of the claimed privacy attribute in realistic code or an implausible string (e.g., hallucinated or placeholder-like). In parallel, multiple human evaluators with coding backgrounds independently assess the same set under identical criteria. We then apply the Judge LLM to this dataset

Results for the DeepSeek Family

Table 6 reports the same step-wise statistics for the DeepSeek family (DeepSeek-V3 and DeepSeek-R1). Reporting intermediate counts clarifies where candidates are filtered out. 12

Table 6: Privacy leaks Analysis of DeepSeek-AI’s LLM series