DP RIV B ENCH : B ENCHMARKING LLM S ’ R EASONING FOR D IFFERENTIAL P RIVACY Erchi Wang1† Pengrun Huang2† Eli Chien3 Om Thakkar4 Kamalika Chaudhuri2 Yu-Xiang Wang1 Ruihan Wu4†,∗ 1
Halıcıoğlu Data Science Institute, UC San Diego Department of Computer Science and Engineering, UC San Diego 3 Department of Electrical Engineering, National Taiwan University 4 OpenAI † Denotes core contribution, ∗ Work performed at UC San Diego Correspondence to: [email protected], [email protected]
arXiv:2604.15851v1 [cs.LG] 17 Apr 2026
2
A BSTRACT Differential privacy (DP) has a wide range of applications for protecting data privacy, but designing and verifying DP algorithms requires expert-level reasoning, creating a high barrier for non-expert practitioners. Prior works either rely on specialized verification languages that demand substantial domain expertise or remain semi-automated and require human-in-the-loop guidance. In this work, we investigate whether large language models (LLMs) can automate DP reasoning. We introduce DPrivBench, a benchmark in which each instance asks whether a function or algorithm satisfies a stated DP guarantee under specified assumptions. The benchmark is carefully designed to cover a broad range of DP topics, span diverse difficulty levels, and resist shortcut reasoning through trivial pattern matching. Experiments show that while the strongest models handle textbook mechanisms well, all models struggle with advanced algorithms, revealing substantial gaps in current DP reasoning capabilities. Through further analytic study and failure-mode analysis, we identify several promising directions for improving automated DP reasoning. Our benchmark provides a solid foundation for developing and evaluating such methods, and complements existing benchmarks for mathematical reasoning. Resources: § Code
1
|
Dataset
Introduction
Differential privacy (DP) (Dwork et al., 2006) has emerged as the gold standard for data privacy, offering rigorous mathematical guarantees that protect individual information while still enabling meaningful statistical and machine learning analyses. Its impact spans a wide range of applications from national statistics released by government agencies (Abowd, 2018; Garfinkel, 2020) to the handling of user data by major technology companies (Google Developers Blog, 2021; Figas, 2025; Rogers, 2021; Apple Blog, 2025). Despite its broad applicability, developing and deploying DP mechanisms for specific use cases often requires substantial expertise. Designing an algorithm with a target privacy budget involves careful reasoning with specialized knowledge in DP, an error-prone task even for DP researchers (see, e.g., Lyu et al., 2017). This high barrier prevents non-experts from utilizing DP in their application despite the potential privacy need. Towards the long-term goal of developing and deploying DP algorithms automatically, the literature has primarily advanced along two directions. Programmatic DP verification (Reed & Pierce, 2010; Barthe et al., 2014, 2016; Albarghouthi & Hsu, 2017; Sato et al., 2019) formally verify DP guarantees by checking symbolic proofs or synthesizing mechanisms from formal algorithm specifications. While these systems provide strong soundness guarantees, they typically require substantial domain expertise to encode algorithms in specialized verification languages, which limits their accessibility to non-expert users. Another complementary line of work can be viewed as semi-automated DP,
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
pioneered by DPCheatSheet (Chu et al., 2025), in which LLMs are used to help non-experts design and implement DP algorithms interactively. With the rapid progress of large language models (LLMs), especially their strong performance on general mathematical reasoning tasks (Huang & Yang, 2025; OpenAI, 2025; Google DeepMind, 2025), it is natural to ask whether they can assist with reasoning about differential privacy (DP), for example, by identifying flaws in DP proofs or verifying the privacy guarantees of stated DP algorithms. Unlike approaches based on formal verification languages or intensive human-in-the-loop guidance, this direction treats the LLM as the primary reasoning agent. Given an algorithm description in natural language or LATEX, as is often the case in practice, the LLM is asked to reason directly about whether the algorithm satisfies a claimed DP guarantee. However, to the best of our knowledge, this problem has not been systematically studied in prior work. This paper focuses on a fundamental question within this emerging direction: Can LLM reason about the DP guarantees of algorithms? To facilitate the study of this question, we introduce a benchmark, DPrivBench. The benchmark consists of carefully curated instances, each describing an algorithm together with explicit assumptions (if any), and tasks LLMs with determining whether the stated differential privacy guarantee holds. The benchmark is designed according to three guiding principles: broad topic coverage, diverse difficulty levels, and resistance to shortcut reasoning through trivial pattern matching. The three principle enables a meaningful and reliable evaluation for the automated DP through LLM reasoning. The benchmark consists of two complementary categories. Category 1 focuses on foundational sensitivity-based DP mechanisms at the textbook level. More challengingly, Category 2 covers a broader range of research topics and evaluates advanced DP algorithms that require substantially more sophisticated reasoning.
Overview of DPrivBench Category 1
Category 2
Text-book DP Mechanisms
Mechanism
Function Bank
+
Function
Templates Reasoning …
Yes
Positive and Negative Instace
DP Literature
Algorithm with DP Guarantee
Perturb it in three aspects 1. algorithm perturb 2. assumption perturb 3. conclusion perturb
Negative Instance
Collect from experts
Positive Instance
Common questions from DP community
Figure 1: Overview of DPrivBench. The left panel illustrates a representative reasoning instance posed to an LLM. The right panel summarizes the benchmark construction, consisting of Category 1 (mechanism-level instance with a function bank) and Category 2 (algorithm-level instance from the DP literature). In total, DPrivBench contains 588 instances in Category 1 and 132 instances in Category 2 (720 instances overall). By evaluating a diverse set of state-of-the-art language models on DPrivBench, we obtain several key observations. For foundational sensitivity-based DP mechanisms, the strongest closed-source models (GPT-5-High and Gemini-3-Pro) equipped with an enhanced reasoning mode achieve high accuracy, while all other models exhibit non-negligible error rates. For more advanced differential privacy algorithms with nontrivial analysis, no evaluated model demonstrates consistently strong performance. These results suggest that while current models are largely sufficient for textbook-level DP reasoning and may serve as useful aids for beginners, a gap remains for reliably analyzing modern DP algorithms. We conduct further study, aiming to guide the future study for better LLM reasoning towards automated DP. First, we evaluate whether providing explicit references that mimic information retrieval from external sources improves accuracy. In this case, performance increases noticeably, pointing to a promising direction for future tools that integrate LLM reasoning with curated DP knowledge bases. second, we conduct targeted case studies to identify and characterize common failure modes in model behavior, which sheds light on what the highlighted aspects should be when improving the reasoning trajectory. Our benchmark serves as a cornerstone for advancing the automation of DP reasoning with LLMs. Beyond its practical value for privacy research, it also serves as a new and challenging testbed for mathematical reasoning. Since DP is typically taught as a graduate-level topic in applied mathematics and theoretical computer science, our benchmark complements existing math reasoning datasets (e.g., GSM8K (Cobbe et al., 2021), GPQA (Rein et al., 2024), MATH-Perturb (Huang et al., 2025)). 2
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
2
Problem Set-Up
2.1
Preliminary
Differential privacy (DP) is a formal framework for quantifying privacy guarantees in randomized algorithms. It ensures that the algorithm’s output distribution changes only minimally when a single individual’s data is modified, providing robustness against arbitrary auxiliary information. We next present the standard definition of (ε, δ)-differential privacy. Definition 2.1 ((ε, δ)-Differential Privacy (Dwork et al., 2006)). A randomized mechanism M : D → R with domain D and range R satisfies (ε, δ)-differential privacy if for any neigbouring datasets pair D, D′ ∈ D, and for any measurable subset of outputs S ⊆ R, it holds that Pr[M(D) ∈ S] ≤ eε · Pr[M(D′ ) ∈ S] + δ. 2.2
Problem Statement
In this paper, we study a central question in automated DP: Can LLM reason about the DP guarantees of algorithms? Specifically, given a concrete description of a function or an algorithm together with a claimed privacy guarantee, an LLM is tasked with determining whether the guarantee holds and returning a binary decision. To support this study, we construct a benchmark with the following design principles: • Broad topic coverage. The benchmark spans core topics in DP, making the evaluation broadly representative and relevant to the DP community. • Diverse difficulty. It ranges from textbook mechanisms to advanced mathematical reasoning of research-level DP algorithms, enabling fine-grained assessment across difficulty levels. • Resistance to shortcut reasoning. Instances are designed to require genuine reasoning, preventing correct answers from being obtained via recall of public training data. Within these principles, benchmark performance provides a reliable signal of DP reasoning ability and a solid foundation for developing improved DP reasoning methods. The scope of LLM reasoning for differential privacy. We present an initial benchmark for evaluating LLMs’ ability to determine whether an algorithm satisfies a stated DP guarantee, a core capability underpinning future end-to-end systems for designing, validating, and deploying differentially private algorithms. A natural next step towards such an end-to-end system is DP algorithm generation: given the description of a nonprivate function or algorithm, can an LLM automatically generate a differentially private variant that satisfies a target privacy budget? Success on this task would enable LLMs to act as accessible design assistants for non-expert users, provide strong baselines for DP researchers, and potentially inspire improved algorithmic designs through human–LLM collaboration. The task studied in this paper is a critical building block for such systems: a reliable DP checker can be used to evaluate candidate designs, guide iterative refinement, or serve as a reward signal for training LLMs that generate increasingly valid DP algorithms. A further important topic is implementation-level automation. Even when a DP algorithm is theoretically sound, subtle coding errors—such as incorrect noise calibration or flawed randomness handling—can completely invalidate its privacy guarantees. An open question is how effectively LLMs can detect such implementation-level privacy violations, as well as generate correct and faithful implementations of DP algorithms. More promisingly, how can the LLMs organically leverage the existing verified DP implementations (which we discuss in detail in Section 2.3)? Progress along this dimension would substantially lower the barrier to deploying DP in real-world systems, enabling practitioners without deep DP expertise to use privacy-preserving methods while also streamlining the development process for expert users. 2.3
Related Work
We now discuss how the emerging direction of LLM-based reasoning for differential privacy connects to other lines of work in differential privacy. Overall, we view this emerging direction as complementary to existing directions: in some cases, they address different stages or challenges, while in others, they pursue similar goals from different angles. In both settings, combining these approaches can lead to stronger, more accessible, and reliable privacy-preserving systems. DP Auditing. DP auditing is an important line of work for detecting implementation-level violations of differential privacy. Most existing approaches adopt a black-box framework (Bichsel et al., 2021; Ding et al., 2018): by carefully 3
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
designing neighboring input datasets and empirically evaluating the outputs of a target implementation, auditors estimate a lower bound on the privacy loss. If this empirical lower bound exceeds the claimed theoretical guarantee, it indicates a likely privacy violation due to implementation bugs. Recently, several works (Steinke et al., 2023; Mahloujifar et al., 2024; Xiang et al., 2025) have focused on improving the efficiency of such audits, reducing the number of required executions to mitigate the substantial computational overhead, especially in settings involving large-scale deep learning models. For a comprehensive survey of DP auditing, we refer readers to Annamalai et al. (2025). In contrast to implementation auditing, our work targets the correctness of algorithms themselves, as specified in natural language and mathematical notation. These two directions address complementary but equally essential stages of the DP pipeline. Implementation auditing serves as a final safeguard prior to deployment, whereas algorithm-level checking operates earlier in the design process, verifying whether a proposed algorithmic description satisfies differential privacy in the first place. Programmatic DP verification. A line of work (Reed & Pierce, 2010; Barthe et al., 2014, 2016; Albarghouthi & Hsu, 2017; Zhang & Kifer, 2017; Sato et al., 2019; Near & Abuah, 2021) investigates DP verification through program-language and formal-methods approaches, which encode algorithms in specialized languages and establish privacy guarantees using symbolic proofs such as type systems, relational Hoare logic, or coupling arguments. Our work pursues the same goal of determining whether an algorithm satisfies a DP guarantee, but through LLM-based reasoning. While prior systems offer strong formal soundness, they require substantial expertise and are often limited in expressiveness. In contrast, LLMs can reason directly over natural-language and mathematical algorithm descriptions, lowering the barrier to use and enabling analysis of more expressive settings. Verified DP implementations. Besides DP auditing and programmatic verification, there are also software libraries for verified DP implementations and privacy accounting, such as TensorFlow Privacy (Tensorflow Privacy Contributors, 2019), Opacus (Yousefpour et al., 2021), AutoDP (Autodp Contributors., 2023), and OpenDP (Shoemate et al.). These tools provide verified building blocks for implementing DP mechanisms and support privacy accounting to improve implementation correctness. Under the broader objective of automating differential privacy, LLM-based reasoning is better suited to supporting algorithm design and first-pass expert-like checking, while verified DP libraries remain essential for deployment and serve as an additional verification safeguard. LLM benchmarks on mathematical reasoning. Mathematical reasoning with large language models has been extensively studied in recent years, leading to the development of a wide range of evaluation benchmarks. Representative pre-college–level benchmarks include MATH (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), and AIME (Mathematical Association of America, 2025), which primarily assess problem-solving skills in algebra, geometry, and arithmetic. Recent closed-source models with enhanced reasoning capabilities achieve near-perfect performance on these benchmarks, while open-source models continue to exhibit a noticeable performance gap. More challenging benchmarks target college-level and graduate-level mathematics, including MathBench (Liu et al., 2024), Ghost (Frieder et al., 2023), GPQA (Rein et al., 2024), HARDMath (Fan et al., 2024), and MATH-Perturb (Huang et al., 2025). These datasets require deeper conceptual understanding and multi-step reasoning. Even the strongest existing models still show substantial room for improvement. Our proposed benchmark, DPrivBench, also contributes to this line of work by focusing on differential privacy reasoning, a core topic in graduate-level applied mathematics and theoretical computer science.
3
Dataset Construction
In this section, we construct DPrivBench, a benchmark designed to probe LLM reasoning towards DP guarantees under a unified question format. Each question presents a concrete function or algorithm together with a claimed privacy guarantee, and asks whether the algorithm satisfies the stated definition. The privacy guarantee may be expressed in terms of standard (ε, δ)-differential privacy, Rényi differential privacy, or related notions, with privacy parameters specified either as fixed constants or as functions of algorithmic parameters. Please check the left panel in Figure 1 for an example of the question. Guided by the principles of broad topic coverage and diverse difficulty levels, we design two complementary categories in our benchmark. Category 1 focuses on foundational sensitivity-based mechanisms, requiring basic reasoning about function sensitivity and mechanism application at the textbook level. As a complement, Category 2 targets advanced differential privacy algorithms drawn from a wide range of research topics, requiring nontrivial, algorithm-specific 4
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
reasoning beyond standard textbook guarantees and thus posing a substantially greater challenge than Category 1. The right panel of Figure 1 provides an overview of the benchmark structure. 3.1
Category 1: Mechanism-Level Instances with a Function Bank
This category evaluates models’ ability to reason about foundational, textbook-level DP mechanisms. To this end, we instantiate each mechanism with multiple concrete functions and their explicitly global sensitivities for constructing the benchmark instances. Foundational DP mechanisms. In the first category, we consider six sensitivity-based DP mechanisms: the Laplace mechanism (Dwork et al., 2006), the Gaussian mechanism under zero-concentrated DP (zCDP) (Bun & Steinke, 2016) or Gaussian DP (GDP) (Dong et al., 2022), Report-Noisy-Max with Gumbel noise (known as Exponential mechanism (McSherry & Talwar, 2007)), Report-Noisy-Max with Laplace noise (Dwork et al., 2014), Report-Noisy-Max with Exponential noise (i.e. Permute-and-Flip mechanism (McKenna & Sheldon, 2020; Ding et al., 2021)). These mechanisms are all sensitivity-based, where the sensitivity ∆f is associated with a function f . Specifically, let f : D → Rd be a query function, where D denotes the space of datasets and d is the output dimension. The (global) ℓ1 -sensitivity of f is defined as ∆f =
max
D,D ′ ∈D:D and D’ differ at one data
∥f (D) − f (D′ )∥1 .
As a concrete example of sensitivity-based mechanisms, the Laplace mechanism is stated as below: Formal definitions of all six mechanisms are provided in the Appendix A.1. Theorem 3.1. (Dwork et al., 2006) A(D) = f (D) + Lap(∆f /ε)d , where Lap(b) denotes the Laplace distribution 1 with density p(x) = 2b exp(−|x|/b), satisfies ε-DP. Function bank construction. To systematically evaluate sensitivity-based differential privacy mechanisms, we construct a curated function bank designed to span a wide range of sensitivity reasoning difficulty. The bank contains 49 functions, initially generated using GPT-5 and then subsequently filtered by the authors, where each function maps an n-dimensional vector x ∈ [0, 1]n to a real value. For each function f , we manually compute its tight global sensitivity ∆f =
max
x∼x′ ∈[0,1]n
|f (x) − f (x′ )|
which serves as the ground truth for constructing later positive or negative test instances. Pn At the easy end of the spectrum, we include functions with immediately obvious sensitivity, such as f (x) = i=1 xi , which has ∆f = 1. At the more challenging end, we incorporate functions whose Pn sensitivity requires careful Pnreasoning about global extrema and coupling between coordinates, such as f (x) = i=1 |xi − x̄|, where x̄ = n1 j=1 xj and the tight sensitivity is ∆f = 2(n−1) . n Instance construction with templates. Given a DP mechanism (from the six listed above) and a function-sensitivity pair (g, ∆g ) from the function bank, we generate a question with ground-truth answer yes by instantiating the corresponding mechanism with the correct sensitivity. Below, we show the template used for the Laplace mechanism: Template for Laplace Mechanism Suppose X = (x1 , . . . , xn ) ∈ [0, 1]n . Define f (X) = g(X) and define f˜(X) = f (X) + Lap(∆g · 1ε ). The neighboring datasets X and X ′ differ in at most one element. Does f˜ satisfy ε-differential privacy?
To construct negative examples, we intentionally under-calibrate the noise multiplier. For instance, in the template ∆g ∆g above, we replace Lap ε with Lap 2ε , which yields a 2ε-DP guarantee rather than the claimed ε-DP guarantee in the positive example. Resistance to shortcut reasoning. To ensure that LLMs are evaluated not only on theorem memorization but also on genuine reasoning, we adopt two design choices: (1) instances are instantiated with explicit mechanisms and concrete functions, rather than directly querying the name of the mechanism; and (2) positive and negative instances are constructed in pairs that differ only by a single term in the noise scaling. Together, these choices prevent correct answers from being obtained through surface-level pattern matching or memorization. 5
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
Topic Privacy Accounting Privacy Amplification via Subsampling Private Selection DP Gradient Descent DP Linear Regression DP Hyperparameter Optimization Private Aggregation of Teacher Ensembles (PATE) Propose–Test–Release (PTR) Parallel Composition Sparse Vector Technique Sequential/Adaptive Composition Smooth Sensitivity DP Adam DP Stochastic Gradient Descent Mean Estimation Objective Perturbation Output Perturbation Quantile Estimation
Yes
No
Count
5 2 2 1 1 1 1 2 6 3 3 2 1 3 5 1 1 2
7 3 3 12 5 5 2 2 8 5 4 1 2 6 10 3 2 3
12 5 5 13 6 6 3 4 14 8 7 3 3 9 15 4 3 5
Total 42 83 125 Table 1: Distribution of positive and negative questions across topics in the benchmark dataset.
Evaluation scope. Besides assessing overall model accuracy, this category enables two forms of fine-grained analysis. First, the included mechanisms span two core topics (additive noise mechanisms and private selection) as well as three variants of DP notions, allowing us to examine model performance across both mechanism topics and DP notions. Second, this category supports detailed analysis of model failure modes. In particular, the use of a shared function bank across multiple mechanisms enables us to separate errors due to incorrect function-level sensitivity reasoning from incorrectly selecting or applying of the DP mechanism. 3.2
Category 2: Algorithm-Level Instances from the Research Literature
Beyond textbook and well-established differential privacy mechanisms, we further evaluate models on a collection of more advanced DP algorithms. Instances in this category are derived from algorithms proposed in the research literature, together with systematically constructed perturbations of these algorithms. We attach all Category 2 questions in the supplementary materials. DP algorithm selection from the literature. We organize the algorithm selection around four major research directions in differential privacy: DP accounting, DP statistics, DP for machine learning (DP-ML), Data adaptive mechanism. Under these directions, we identify 16 topics that are commonly studied in practice and pick the representative algorithms under those topics. Table 1 summarizes the list of topics and the number of instances associated with each topic. We note that the Sparse Vector Technique has been extensively studied in prior work on DP verification (Zhang et al., 2016; Lyu et al., 2017), and we include it as a standalone topic under the Privacy Accounting category. Structured instance construction. Starting from each picked algorithm, we will construct the questions with the ground truty yes and no respectively. We first note that algorithms extracted directly from the literature naturally correspond to positive (i.e., yes) instances, since they satisfy the stated DP guarantees under their original assumptions. To meaningfully evaluate whether a model truly understands DP reasoning—rather than merely memorizing canonical results—we systematically construct negative instances that violate DP correctness. We further perturb the positive question to construct negative questions. Notice that each correct example is decomposed into three components: (i) the algorithm description, (ii) the underlying assumptions or conditions, and (iii) the claimed DP guarantee. We provide a colored box on the following page for illustration. We then generate negative instances by perturbing exactly one of these components at a time: (1) algorithm perturbations, such as altering the necessary algorithmic steps; (2) assumption perturbations, such as removing essential conditions (e.g., convexity of the loss function); and (3) guarantee perturbations, such as claiming a strictly stronger privacy guarantee than what the algorithm can support. Concrete examples of these perturbations are illustrated in Appendix E. We also include a more fine-grained taxonomy of error patterns in the Appendix B. 6
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
Category 2 Positive Example Suppose that for all z ∈ Z, ℓ(·, z) is twice-differentiable, and the rank of its Hessian ∇2 ℓ(w, z) at any w ∈ W is at most 1. Also assume that the smoothness parameter satisfies β ≤ εnλ. Is the following algorithm (ε, δ)-differentially private? Input: Private dataset S = (z1 , . . . , zn ) ∈ Z n , L-Lipschitz, β-smooth, convex loss function ℓ, convex set W ⊆ Rd , privacy parameters ε ≤ 1, δ ≤ 1/n2 , regularization parameter λ. 10L2 log(1/δ) ; ε2 1 Pn ⟨G,w⟩ 2 ℓ(w, zi ); 2: return ŵ = arg min L̂(w; S) + n + λ∥w∥ , where L̂(w; S) ≜ w∈W n i=1
1: Sample G ∼ N (0, σ 2 Id ), where σ 2 =
Algorithm 1: AObjP : Objective Perturbation
Resistance to shortcut reasoning. Because negative instances are constructed via targeted perturbations, models cannot succeed by merely memorizing publicly available results: such memorization would cause them to incorrectly label all perturbed (negative) instances as correct due to over-reliance on superficial pattern matching. Evaluation scope. This category is substantially more challenging, as correctness cannot be established by directly applying textbook DP mechanisms. Instead, it requires reasoning about advanced, algorithm-specific analyses and nontrivial design choices, some of which are challenging even for human experts. Moreover, because each question is drawn from one of the topics listed in Table 1, this category enables fine-grained evaluation of model performance across different research topics. Auxiliary metadata for future research. In addition to questions and labels, we provide auxiliary metadata to support future research on automated DP reasoning. Specifically, each instance includes a reference link that justifies the correct conclusion, as well as explanatory comments for negative instances detailing why the stated DP guarantee does not hold. We believe this metadata is valuable for developing improved methods, for example, by serving as supervision for knowledge retrieval or providing dense reasoning signals for approaches such as reinforcement learning. Table 2: Model accuracy on the Category 1 benchmark (reported as mean ± 1.96×standard error). All standard deviations are computed over five random seeds. Model-Avg reports the mean ± standard deviation of accuracies after averaging across the six tasks for each model. Task-Avg reports the accuracy after averaging across the eleven models for each task. Task-Avg
Laplace
Gaussian (GDP)
Gaussian (zCDP)
ExpoMech
LaplaceRNM
PermuteFlip
GPT-5-High Gemini-3-Pro Gemini-2.5-Flash Claude-Sonnet-4.5 GPT-5-Minimal Claude-Opus-4.5
0.995 0.923 0.778 0.738 0.736 0.728
1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000 0.949 ± 0.017 0.963 ± 0.022 0.955 ± 0.023
0.976 ± 0.014 0.996 ± 0.005 0.743 ± 0.052 0.814 ± 0.015 0.733 ± 0.046 0.753 ± 0.038
1.000 ± 0.000 1.000 ± 0.000 0.996 ± 0.005 0.931 ± 0.022 0.835 ± 0.032 0.908 ± 0.017
1.000 ± 0.000 0.959 ± 0.022 0.749 ± 0.045 0.588 ± 0.023 0.782 ± 0.016 0.502 ± 0.031
0.998 ± 0.004 0.645 ± 0.019 0.514 ± 0.026 0.549 ± 0.038 0.573 ± 0.019 0.590 ± 0.026
0.996 ± 0.005 0.939 ± 0.006 0.665 ± 0.015 0.600 ± 0.038 0.533 ± 0.037 0.657 ± 0.035
DeepSeek-V3.1-chat DeepSeek-R1 Qwen3-30-Instruct Goedel-Prover-V2 Qwen3-30-Think
0.841 0.741 0.671 0.628 0.565
0.992 ± 0.007 0.933 ± 0.030 0.992 ± 0.007 0.873 ± 0.033 0.973 ± 0.015
0.951 ± 0.016 0.929 ± 0.020 0.567 ± 0.038 0.596 ± 0.041 0.545 ± 0.010
0.994 ± 0.005 0.943 ± 0.020 0.918 ± 0.006 0.822 ± 0.027 0.569 ± 0.013
0.818 ± 0.026 0.763 ± 0.043 0.520 ± 0.033 0.500 ± 0.035 0.418 ± 0.023
0.643 ± 0.033 0.359 ± 0.040 0.508 ± 0.032 0.469 ± 0.022 0.416 ± 0.037
0.651 ± 0.038 0.522 ± 0.029 0.520 ± 0.054 0.506 ± 0.030 0.465 ± 0.042
-
0.966
0.782
0.901
0.691
0.570
0.641
Model-Avg
4
Experimental Setup and Results Overview
Rubric-guided binary evaluation. To enable reliable and automated metric computation, we adopt a standardized prompt template that instructs the LLM to produce a binary “yes” or “no” decision in a fixed, machine-parsable format. If a model fails to follow this instruction and does not output an explicit binary verdict, we apply a secondary judging step using GPT-4o to map the response to a “yes” or “no” label. Each question is evaluated using the binary correctness metric. For the main results, we run five trials with different random seeds and report accuracy aggregated across seeds. Benchmarking LLMs. We evaluate 11 LLMs, covering the closed-source models (GPT-5 with minimal reasoning effort, GPT-5 with highest reasoning effort, Gemini-3, Gemini-2.5-flash, Claude-Sonnet, Claude-Opus), the open-source models (Qwen3-30-Think, Qwen3-30-Instruct, DeepSeek-R1, DeepSeek-V3.1-chat, Goedel-Prover-V2). We report 7
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
Table 3: Model performance on the DPrivBench, Category 2 (mean ± 1.96×standard error). Model
F1 Score
Precision
Recall
Gemini-3-Pro GPT-5-High Gemini-2.5-Flash GPT-5-Minimal Claude-Sonnet-4.5 Claude-Opus-4.5
0.748 ± 0.019 0.742 ± 0.032 0.602 ± 0.028 0.548 ± 0.036 0.518 ± 0.046 0.487 ± 0.026
0.699 ± 0.018 0.716 ± 0.044 0.637 ± 0.038 0.497 ± 0.031 0.499 ± 0.040 0.516 ± 0.028
0.805 ± 0.034 0.771 ± 0.024 0.571 ± 0.026 0.614 ± 0.052 0.538 ± 0.054 0.462 ± 0.028
DeepSeek-R1 DeepSeek-V3.1-chat Qwen3-30-Instruct Qwen3-30-Think Goedel-Prover-V2
0.614 ± 0.050 0.603 ± 0.032 0.460 ± 0.019 0.407 ± 0.046 0.362 ± 0.045
0.657 ± 0.045 0.576 ± 0.029 0.526 ± 0.024 0.509 ± 0.070 0.509 ± 0.063
0.576 ± 0.054 0.633 ± 0.041 0.410 ± 0.027 0.343 ± 0.052 0.281 ± 0.037
Table 4: Topic-wise performance of GPT-5-High and Gemini-3-Pro on Category 2. Topics are sorted by the averaged accuracy of the two models, and we report the top four and bottom five topics. The complete results are provided in Table 10. Topic DP Adam Quantile Estimation Privacy Accounting DP Gradient Descent ... DP Hyperparameter Optimization Propose–Test–Release (PTR) Private Aggregation of Teacher Ensembles (PATE) Output Perturbation Smooth Sensitivity
Gemini-3-Pro
GPT-5-High
Avg Accuracy
1.00 1.00 0.92 0.92
1.00 1.00 0.95 0.94
1.00 1.00 0.93 0.93
0.60 0.70 0.67 0.33 0.33
0.77 0.65 0.33 0.60 0.33
0.68 0.68 0.50 0.47 0.33
model details, including version and release date, in Table 8. Throughout all experiments, we disabled the tool use for a fair comparison. In Section 5, we will present the results for the 11 LLMs on our benchmark DPrivBench. In Section 6, we will conduct further analysis and a case study to better understand the model performance and gain more intuition of how to improve the performance in future work.
5
Main Results
In this section, we present the performance of 11 LLMs on our benchmark DPrivBench. For Category 1, we will have two observations. First, the strongest closed-source models GPT-5 and Gemini-3 have near-perfect performance, while all other models have a certain gap to improve. Second, most models, including open-source models, have near-perfect performance for the subset of questions about the most common Laplace Mechanism, but the accuracy significantly drops for the questions about the other five mechanisms. For Category 2, we will have two observations. First, consistent with Category 1, GPT-5-High and Gemini-3-Pro achieve the strongest performance; however, both models still exhibit substantial room for improvement in this more challenging setting. Second, we analyze per-topic accuracy for GPT-5-High and Gemini-3-Pro. By ranking topics according to accuracy, we identify which research topics are easiest and most challenging for LLMs to reason about. 5.1
Category 1
Overall model performance. Table 2 summarizes overall performance on Category 1. Among all evaluated models, GPT-5-High is the only model to achieve near-perfect accuracy (0.995). The strongest open-source model, DeepSeekV3.1-chat, attains an accuracy of 0.841, leaving a substantial performance gap relative to the best closed-source models. Overall, these results indicate that DP reasoning remains a significant challenge for current open-source models on controlled foundational sensitivity-based tasks. 8
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
Fine-grained results across mechanisms. We further report mechanism-level accuracy for the six evaluated DP mechanisms in Table 2. Nearly all models (except Goedel-Prover-V2) achieve near-perfect performance ≥ 0.94 on the Laplace mechanism. In contrast, accuracy drops substantially for the remaining five mechanisms. Importantly, questions across all mechanisms are constructed from the same underlying function bank, so sensitivity computation is shared; the only difference lies in how the computed sensitivity must be instantiated within different mechanism-specific privacy guarantees. This suggests that many models can correctly compute sensitivity but struggle to apply it within less familiar DP mechanisms. One plausible explanation is that the Laplace mechanism appears most frequently in training data, whereas other mechanisms are less understood. A particularly striking case is the Report-Noisy-Max mechanism with Laplace noise. GPT-5-High is the only model that achieves perfect accuracy, while all other models, including Gemini-3-Pro, perform near or below the random-guess baseline of 0.5. Manual inspection reveals a common failure mode: many models incorrectly reuse the noise scale of the standard Laplace mechanism, adding Lap(∆f /ε) rather than the required Lap(2∆f /ε) to each candidate value when considering Report-Noisy-Max. As a consequence, these models tend to answer “yes" for all questions, resulting in a baseline accuracy of 0.5. This systematic confusion between closely related mechanisms leads to consistently incorrect conclusions and highlights a deeper limitation in mechanism-specific DP reasoning.
5.2
Category 2
Overall model performance. In Table 3, because the dataset on Category 2 is imbalanced, we report the F1 score, precision, and recall for each LLM. We observe that GPT-5-High and Gemini-3-Pro achieve comparable stateof-the-art performance, with F1 scores of 0.742 and 0.748, respectively; GPT-5-High exhibits higher precision, while Gemini-3-Pro attains higher recall. Among open-source models, the DeepSeek series performs best, reaching an accuracy of 0.614, which is competitively higher than GPT-5-Minimal (0.548) and Gemini-2.5-Flash (0.602). Nevertheless, other models perform close to or below that of a naive strategy (F1 ≈ 0.503)1 , indicating that advanced algorithm-level DP reasoning remains challenging for most current LLMs. Fine-grained results across topics. We further analyze per-topic accuracy for the two best-performing models, GPT-5-High and Gemini-3-Pro. By ranking topics according to their mean accuracy across the two models, we identify which topics are easiest and most challenging for LLMs to reason about. As shown in Table 4, Quantile, DP-Adam, Accounting, and DP-GD are the easiest topics, with both models achieving accuracy above 0.9. In contrast, Smooth Sensitivity, PATE, Output Perturbation, PTR, and Hyperparameter Tuning emerge as the most challenging topics, with accuracies below 0.7. Notably, neither model achieves perfect accuracy on SVT, indicating that analyses known to be challenging even for human experts remain difficult for state-of-the-art LLMs.
6
More Analytic Results and Case Study
6.1
Theorem Augmentation and Retrieval-Based Assistance
We analyze 18 questions from Category 2 that have the lowest accuracies across all eleven models. For each question, we augment the LLM prompt with additional helpful information drawn from three sources: (1) a correct implementation of related algorithms from prior work, (2) a key theorem needed for a critical step in the proof, and (3) relevant definitional details. We consider two ways of providing this information to the LLM: (1) directly including it in the prompt, and (2) supplying a database of reference papers containing the relevant information and using RAG to retrieve it. As shown in Figure 2, directly providing the relevant theorem yields the largest performance improvement across all four model settings we evaluate. RAG also generally improves performance, except for Gemini-3-Pro, although the gains are smaller than those from directly injecting the theorem into the prompt. This ordering is intuitive: exact theorem augmentation provides the most precise supporting context, whereas retrieval over a theorem database is a noisier but more realistic intermediate setting.
1
This baseline corresponds to always predicting “yes” for all Category 2 instances, yielding Recall = 1.00, Precision =
42 ≈ 0.336, and F1 ≈ 0.503. 42+83
9
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
0.650 0.581
0.6
0.523*
0.521
0.5
F1 Score
w. theorem RAG zero-shot
0.625
0.467 0.403
0.4
0.463
0.435
0.353
0.315
0.3
0.258
0.2 0.1 0.0
Gemini-3-Pro
GPT-5-High
Gemini-2.5-Flash
GPT-5-Low
Figure 2: Performance on the 18 hardest Category 2 questions under varying levels of helpful information: theoremaugmented prompting (with theorems), restricted retrieval augmentation (RAG), and zero-shot prompting. Scores are averaged over five trials. 2
6.2
Will In-context Learning improve the performance of DP reasoning?
Previous work has shown that in-context learning, especially few-shot chain-of-thought prompting, can substantially improve model performance on mathematical reasoning tasks (Wei et al., 2022). We focus on the Private Selection with Laplace Noise task in Category 1 LaplaceRNM, which is challenging for most LLMs, with most models achieving relatively low accuracy (Table 2). Because instances in this task family are relatively homogeneous and admit similar proof structures, in-context learning is particularly natural in this setting. We therefore conduct a one-shot experiment with GPT-5-minimal using a single question–answer exemplar for the Laplace Report Noisy Max mechanism. The proof template (Appendix C.4.2) is adapted from Theorem A.8. Under this one-shot prompt, both models we tested have improved performance, as shown in Table 5.
Table 5: Accuracy for In-context learning (Mean±1.96×standard error) Model GPT-5-Minimal Gemini-2.5-Flash
6.3
Zero-Shot
One-shot
0.573 ± 0.019 0.514 ± 0.026
0.737 ± 0.065 0.661 ± 0.073
Failure Mode
We analyze two common failure modes exhibited by the top two performing models, GPT-5-High and Gemini-3-Pro. By identifying these error patterns, we aim to provide insights that can inform future method development – for example, by using these failure modes as highlighted aspects when assessing the correctness of a reasoning trajectory. Failure to identify subtle but semantically significant changes. We find that LLMs can confuse questions with a similar structure, leading to systematic errors. In the following example (Q26 in the dataset), we modify the assumption required for parallel composition on the data partition from pairwise disjointness (i.e. Xi ∩ Xj = ∅) to only sequential disjointness (i.e. Xi ∩ Xi+1 = ∅), and ask whether the resulting algorithm satisfies parallel composition. Both GPT-5-High and Gemini-3-Pro answer this question incorrectly across all five random seeds. In Appendix F, we provided one more example. 2
The asterisk(*) on the Gemini-3-Pro RAG bar indicates that this result was obtained using Gemini-3.1-Pro, the Gemini-3 checkpoint became unavailable during this test.
10
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
Question 26 (Label: No) n Consider the following procedure: S M := {Mi (Xi )}i=1 , where Xi ∩ Xi+1 = ∅ for all i ∈ {1, . . . , n − 1}, and their union covers the entire dataset, i.e., n Furthermore, the partition is data-independent. Suppose each mechanism X = X. i i=1 Mi satisfies (εi , δi )-differential privacy. Does the combined mechanism M satisfy (maxi∈[n] εi , maxi∈[n] δi )-differential privacy?
Hallucination of assumptions learned from training data. LLMs may also hallucinate assumptions based on their pretraining data. For example, in a question on output perturbation, the original formulation appears as Lowy & Razaviyayn (2021, Proposition 2.1), where the sensitivity of the objective function is defined with respect to the minimizer. In our question, however, we do not state this assumption and only specify that an L2 sensitivity is considered in general, which should typically be interpreted as the sensitivity of the function value f . This interpretation is insufficient to guarantee the correctness of the output perturbation algorithm. Nevertheless, the LLMs verify the statement as correct and assume that the sensitivity is taken with respect to the minimizer, as evidenced in their outputs. On this question, Gemini-3-Pro achieves zero accuracy, while GPT-5-High succeeds in only three trials. Q 57 (Label: No) Let ε > 0, δ ∈ [0, 12 ). Let F : Rd × X n → R be a function with L2 sensitivity ∆F and unique minimizer w∗ (X) = arg minw∈Rd F (w, X) for all X ∈ X n . implementation of output perturbation, abbreviated) .
7
Conclusion and Future Work
Conclusion. We study whether large language models can automate differential privacy reasoning and introduce DPrivBench, the first benchmark designed for this purpose. Covering both foundational DP mechanisms and more advanced DP algorithms, DPrivBench enables fine-grained evaluation across varying reasoning complexity. Our results show that while leading models perform well on textbook DP mechanisms, they consistently struggle with algorithmspecific analyses requiring careful accounting and assumption validation, revealing a substantial gap to expert-level DP reasoning. We also identify promising directions for improving LLMs’ DP reasoning ability, including structured DP knowledge retrieval and improved domain-specific reasoning. Overall, DPrivBench serves as a meaningful and reliable testbed for DP reasoning through LLMs and paves the way toward an end-to-end automated DP system. Future work. This work is a first step toward understanding how LLMs assess whether a given mechanisms satisfy stated DP guarantees. Our results show that advanced DP reasoning remains challenging even for the strongest models, motivating algorithmic improvements to LLM-based DP reasoning guided by DPrivBench. A natural next step is therefore to algorithmically improve LLM-based reasoning for differential privacy guided by our benchmark DPrivBench. Building on this capability, two further directions discussed in Section 2 are particularly promising for enabling end-to-end automated DP systems: automated DP algorithm generation, which aims to synthesize privacypreserving algorithms from non-private specifications, and implementation-level automation, which focuses on detecting and preventing privacy violations arising from flawed code implementations.
8
Acknowledgment
This work was supported in part by the ONR under grants N000142412304 and N00014-25-1-2116, by the NSF under grants CNS 2048091, CIF-2402817 and CNS-2241100, and by the ARO-MURI under grant W911NF2110317. We acknowledge an OpenAI security research grant for providing the necessary credits and API access to their models.
11
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
References Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. 2016. Abowd, J. M. The us census bureau adopts differential privacy. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2867–2867, 2018. Albarghouthi, A. and Hsu, J. Synthesizing coupling proofs of differential privacy. Proceedings of the ACM on Programming Languages, 2(POPL):1–30, 2017. Annamalai, M. S. M. S., Balle, B., Hayes, J., Kaissis, G., and De Cristofaro, E. The hitchhiker’s guide to efficient, end-to-end, and tight dp auditing. arXiv preprint arXiv:2506.16666, 2025. Apple Blog. Understanding aggregate trends for apple intelligence using differential privacy, 2025. URL https: //machinelearning.apple.com/research/differential-privacy-aggregate-trends. Autodp Contributors. autodp: Automating differential privacy computation. https://github.com/yuxiangw/ autodp, 2023. GitHub repository, version 0.2.3.1, accessed 2026-04-16. Balle, B., Barthe, G., and Gaboardi, M. Privacy amplification by subsampling: Tight analyses via couplings and divergences. arXiv preprint arXiv:1807.01647, 2018. Barthe, G., Gaboardi, M., Arias, E. J. G., Hsu, J., Kunz, C., and Strub, P.-Y. Proving differential privacy in hoare logic. In 2014 IEEE 27th Computer Security Foundations Symposium, pp. 411–424. IEEE, 2014. Barthe, G., Gaboardi, M., Grégoire, B., Hsu, J., and Strub, P.-Y. Proving differential privacy via probabilistic couplings. In Proceedings of the 31st Annual ACM/IEEE Symposium on Logic in Computer Science, pp. 749–758, 2016. Bassily, R., Smith, A., and Thakurta, A. Differentially private empirical risk minimization: Efficient algorithms and tight error bounds. arXiv preprint arXiv:1405.7085, 2014. Bassily, R., Feldman, V., Talwar, K., and Thakurta, A. Private stochastic convex optimization with optimal rates. arXiv preprint arXiv:1908.09970, 2019. Bichsel, B., Steffen, S., Bogunovic, I., and Vechev, M. Dp-sniper: Black-box discovery of differential privacy violations using classifiers. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 391–409. IEEE, 2021. Bun, M. and Steinke, T. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of cryptography conference, pp. 635–658. Springer, 2016. Bun, M. and Steinke, T. Average-case averages: Private algorithms for smooth sensitivity and mean estimation. arXiv preprint arXiv:1906.02830, 2019. Canonne, C. L., Kamath, G., and Steinke, T. The discrete gaussian for differential privacy. arXiv preprint arXiv:2004.00010, 2020. Cesar, M. and Rogers, R. Bounding, concentrating, and truncating: Unifying privacy loss composition for data analytics. arXiv preprint arXiv:2004.07223, 2020. Chaudhuri, K., Monteleoni, C., and Sarwate, A. D. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3), 2011. Chu, S.-Y., Tian, Y., Wang, Y.-X., and Jin, H. Dpcheatsheet: Using worked and erroneous llm-usage examples to scaffold differential privacy implementation. arXiv preprint arXiv:2509.12590, 2025. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. De, S., Berrada, L., Hayes, J., Smith, S. L., and Balle, B. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650, 2022. Ding, Z., Wang, Y., Wang, G., Zhang, D., and Kifer, D. Detecting violations of differential privacy. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 475–489, 2018. Ding, Z., Kifer, D., Steinke, T., Wang, Y., Xiao, Y., Zhang, D., et al. The permute-and-flip mechanism is identical to report-noisy-max with exponential noise. arXiv preprint arXiv:2105.07260, 2021. Dong, J., Durfee, D., and Rogers, R. Optimal differential privacy composition for exponential mechanisms and the cost of adaptivity. arXiv preprint arXiv:1909.13830, 2019. Dong, J., Durfee, D., and Rogers, R. Optimal differential privacy composition for exponential mechanisms. 2020. Dong, J., Roth, A., and Su, W. J. Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):3–37, 2022. 12
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
Durfee, D. and Rogers, R. Practical differentially private top-k selection with pay-what-you-get composition. 2019. Dwork, C. and Lei, J. Differential privacy and robust statistics. 2009. Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265–284. Springer, 2006. Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and trends® in theoretical computer science, 9(3–4):211–407, 2014. Fan, J., Martinson, S., Wang, E. Y., Hausknecht, K., Brenner, J., Liu, D., Peng, N., Wang, C., and Brenner, M. P. Hardmath: A benchmark dataset for challenging problems in applied mathematics. arXiv preprint arXiv:2410.09988, 2024. Figas, N. How meta uses privacy-enhancing technologies in advertising and analytics, 2025. URL https://www. avenga.com/magazine/how-meta-uses-privacy-enhancing-technologies-pets-in-adtech/. Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., and Berner, J. Mathematical capabilities of chatGPT. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=xJ7YWXQOrg. Garfinkel, S. How we’re helping developers with differential privacy, Jan 2020. URL https://csrc.nist.gov/ presentations/2020/stppa1-census. Google DeepMind. Gemini 3 pro model card, 2025. URL https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Pro-Model-Card.pdf. Google Developers Blog. How we’re helping developers with differential privacy, Jan 2021. URL https: //developers.googleblog.com/how-were-helping-developers-with-differential-privacy/. Harrison, C. and Manurangsi, P. Exact zcdp characterizations for fundamental differentially private mechanisms. arXiv preprint arXiv:2510.25746, 2025. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. Huang, K., Guo, J., Li, Z., Ji, X., Ge, J., Li, W., Guo, Y., Cai, T., Yuan, H., Wang, R., Wu, Y., Yin, M., Tang, S., Huang, Y., Jin, C., Chen, X., Zhang, C., and Wang, M. MATH-perturb: Benchmarking LLMs’ math reasoning abilities against hard perturbations. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=OZy70UggXr. Huang, Y. and Yang, L. F. Winning gold at imo 2025 with a model-agnostic verification-and-refinement pipeline. arXiv preprint arXiv:2507.15855, 2025. Kaplan, H., Schnapp, S., and Stemmer, U. Differentially private approximate quantiles. 2022. Kifer, D., Smith, A., and Thakurta, A. Private convex empirical risk minimization and high-dimensional regression. 2012. Kulesza, A., Suresh, A. T., and Wang, Y. Mean estimation in the add-remove model of differential privacy. arXiv preprint arXiv:2312.06658, 2023. Li, X., Tramèr, F., Liang, P., and Hashimoto, T. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679, 2021. Liu, H., Zheng, Z., Qiao, Y., Duan, H., Fei, Z., Zhou, F., Zhang, W., Zhang, S., Lin, D., and Chen, K. Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark. arXiv preprint arXiv:2405.12209, 2024. Liu, Y., Sun, K., Jiang, B., and Kong, L. Identification, amplification and measurement: A bridge to gaussian differential privacy. Advances in Neural Information Processing Systems, 35:11410–11422, 2022. Lowy, A. and Razaviyayn, M. Output perturbation for differentially private convex optimization: Faster and more general. arXiv preprint arXiv:2102.04704, 2021. Lyu, M., Su, D., and Li, N. Understanding the sparse vector technique for differential privacy. Proceedings of the VLDB Endowment, 10(6):637–648, 2017. Mahloujifar, S., Melis, L., and Chaudhuri, K. Auditing f -differential privacy in one run. arXiv preprint arXiv:2410.22235, 2024. Mathematical Association of America. American invitational mathematics examination. https://maa.org/ maa-invitational-competitions/, 2025. Accessed: 2026-01-23. 13
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
McKenna, R. and Sheldon, D. R. Permute-and-flip: A new mechanism for differentially private selection. Advances in Neural Information Processing Systems, 33:193–203, 2020. McSherry, F. Privacy integrated queries: an extensible platform for privacy-preserving data analysis. Communications of the ACM, 2010. McSherry, F. and Talwar, K. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS’07), pp. 94–103. IEEE, 2007. Mironov, I. Renyi differential privacy. arXiv preprint arXiv:1702.07476, 2017. Near, J. P. and Abuah, C. Programming differential privacy. URL: https://uvm, 2021. OpenAI. Introducing GPT-5, 2025. URL https://openai.com/index/introducing-gpt-5/. Papernot, N. and Steinke, T. Hyperparameter tuning with renyi differential privacy. arXiv preprint arXiv:2110.03620, 2021. Papernot, N., Song, S., Mironov, I., Raghunathan, A., Talwar, K., and Úlfar Erlingsson. Scalable private learning with pate. arXiv preprint arXiv:1802.08908, 2018. Redberg, R., Zhu, Y., and Wang, Y.-X. Generalized ptr: User-friendly recipes for data-adaptive algorithms with differential privacy. arXiv preprint arXiv:2301.00301, 2022. Reed, J. and Pierce, B. C. Distance makes the types grow stronger: a calculus for differential privacy. In Proceedings of the 15th ACM SIGPLAN international conference on Functional programming, pp. 157–168, 2010. Rehn, A., Zhao, L., Heikkilä, M. A., and Honkela, A. On optimal hyperparameters for differentially private deep transfer learning. arXiv preprint arXiv:2510.20616, 2025. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https: //openreview.net/forum?id=Ti67584b98. Rogers, R. Deploying differential privacy in industry: Progress and learnings, 2021. URL https://icml.cc/ virtual/2021/11631. Sato, T., Barthe, G., Gaboardi, M., Hsu, J., and Katsumata, S.-y. Approximate span liftings: Compositional semantics for relaxations of differential privacy. In 2019 34th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS), pp. 1–14. IEEE, 2019. Sheffet, O. Differentially private ordinary least squares. 2017. Shoemate, M., Vyrros, A., McCallum, C., Prasad, R., Durbin, P., Casacuberta Puig, S., Cowan, E., Xu, V., Ratliff, Z., Berrios, N., Whitworth, A., Eliot, M., Lebeda, C., Renard, O., and McKay Bowen, C. OpenDP Library. URL https://github.com/opendp/opendp. Steinke, T., Nasr, M., and Jagielski, M. Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems, 36:49268–49280, 2023. Tensorflow Privacy Contributors. TensorFlow Privacy: Library for training machine learning models with privacy for training data. https://github.com/tensorflow/privacy, 2019. GitHub repository, accessed 2026-04-16. Vadhan, S. The complexity of differential privacy. In Tutorials on the Foundations of Cryptography: Dedicated to Oded Goldreich, pp. 347–450. Springer, 2017. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Xiang, Z., Wang, T., and Wang, D. Privacy audit as bits transmission:(im) possibilities for audit by one run. In USENIX Security, 2025. Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., Zhao, J., et al. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint arXiv:2109.12298, 2021. Zhang, D. and Kifer, D. Lightdp: Towards automating differential privacy proofs. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, pp. 888–901, 2017. Zhang, J., Xiao, X., and Xie, X. Privtree: A differentially private algorithm for hierarchical decompositions. In Proceedings of the 2016 international conference on management of data, pp. 155–170, 2016.
14
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
A
Privacy Guarantee and References
A.1
Privacy guarantee for Category 1
In this section, we state formal proof for all six mechanisms use in Category 1. Through the question statement in Category 1, we use replace-one neighbouring relationship. Theorem A.1 (Laplace Mechanism (Dwork et al., 2006)). Let f : X n → Rd be a function with ℓ1 -sensitivity ∆1 (f ) := max′ ∥f (X) − f (X ′ )∥1 . The Laplace mechanism defined as follow satisfies ε-DP: X∼X ∆1 (f ) i.i.d. M(X) = f (X) + Z, Zi ∼ Lap . ε Theorem A.2 (zCDP guarantee of Gaussian Mechanism (Bun & Steinke, 2016)). Let f : X n → Rd be a function with ℓ2 -sensitivity ∆2 (f ) := max′ ∥f (X) − f (X ′ )∥2 . The Gaussian mechanism defined as follow satisfies ρ-zero X∼X
Concentrated Differential Privacy (zCDP): ∆2 (f )2 M(X) = f (X) + Z, Zi ∼ N 0, . 2ρ Theorem A.3 (GDP guarantee of Gaussian Mechanism (Dong et al., 2022)). Let f : X n → Rd be a function with ℓ2 -sensitivity ∆2 (f ) := max′ ∥f (X) − f (X ′ )∥2 . The Gaussian mechanism defined as follow satisfies µ-Gaussian i.i.d.
X∼X
Differential Privacy (GDP): M(X) = f (X) + Z,
∆2 (f )2 Zi ∼ N 0, µ2 i.i.d.
.
Another type of question in Category 1 is report noisy max(Dwork et al., 2014) for private selection, we provide a general template in Algorithm 2. In particular, when instantiated with exponential noise, the algorithm is called the permute and flip (McKenna & Sheldon, 2020; Ding et al., 2021). When instantiated with Gumbel noise, the algorithm is the famous exponential algorithm (McSherry & Talwar, 2007). Input :Dataset X; score functions {uj (·)}m i=1 with sensitivity ∆ (w.r.t. same neighboring relation); noise distribution P Output :Index î ∈ {1, . . . , m} of the selected item. for i = 1 to m do si ← ui (X) Sample ηi ∼ P s̃i ← si + ηi end î ← arg max s̃i i∈{1,...,m}
return î Algorithm 2: R EPORT N OISY M AX Before stating the privacy guarantee of private selection mechanisms used in Category 1, we introduce definition of noises: Definition A.4 (Exponential Distribution). A random variable X is said to follow an exponential distribution with parameter λ > 0, denoted by X ∼ Exp(λ), if it has probability density function 1 −x/λ e , x ≥ 0, fX (x) = λ 0, x < 0. Definition A.5 (Gumbel Distribution). A random variable X is said to follow Gumbel(α) distribution if it has probability density function x x 1 fX (x) = exp − − exp − , x ∈ R. α α α Theorem A.6 (Privacy guarantee of report noisy max with Laplace noise (Dwork et al., 2014)). Suppose noise distribution P follows Laplace 2∆ ε , then Algorithm 2 satisfies ε-DP Theorem A.7 (Privacy guarantee of report noisy max with Exponential noise (Ding et al., 2021)). Suppose noise distribution P follows Exp 2∆ ε , then Algorithm 2 satisfies ε-DP. Theorem A.8 (Privacy guarantee of report noisy max with Gumbel noise (Ding et al., 2021)). Suppose noise distribution P follows Gumbel 2∆ ε , then Algorithm 2 satisfies ε-DP. 15
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
A.2
References for Category 2 questions
In addition to DP textbook references (Dwork et al., 2014; Vadhan, 2017), we include in Table 6 the research papers used to construct and justify the Category 2 questions.
Subject
Topic
Reference
Accounting
(Mironov, 2017; Dong et al., 2022; Cesar & Rogers, 2020; Ding et al., 2021; Liu et al., 2022; Harrison & Manurangsi, 2025; Dong et al., 2020; Durfee & Rogers, 2019) (Balle et al., 2018) (McSherry, 2010) (Lyu et al., 2017) (Bun & Steinke, 2016; Mironov, 2017; Canonne et al., 2020; Cesar & Rogers, 2020)
Accounting and Composition Amplification by Subsampling Parallel Composition SVT Sequential or Adaptive Composition
DP-ML
DP-GD DP-OLS Hyperparameter-tuning PATE dp-adam dp-sgd objective perturbation output perturbation
(Bassily et al., 2014; Dong et al., 2022) (Sheffet, 2017) (Papernot & Steinke, 2021) (Papernot et al., 2018) (Li et al., 2021) (Abadi et al., 2016; De et al., 2022; Rehn et al., 2025) (Bassily et al., 2019; Kifer et al., 2012; Chaudhuri et al., 2011) (Lowy & Razaviyayn, 2021)
DP-statistics
DP selection: expoMech mean estimation quantile
(Dong et al., 2019, 2020) (Kulesza et al., 2023) (Kaplan et al., 2022)
Data-Adaptive
PTR Smooth_sensitivity
(Redberg et al., 2022; Dwork & Lei, 2009) (Bun & Steinke, 2019)
Table 6: Reference for Category 2 questions (grouped by subject and topic).
B
Taxonomy of error pattern in Category 2 benchmark design
Table 7: Error taxonomy and empirical distribution in Category 2 Error pattern
Core characterization
Stronger-than-valid claim / overclaiming guarantee
Starts from a valid result but claims a strictly stronger privacy guarantee than what the theorem or mechanism actually supports.
Broken implementation step
The overall algorithm looks standard, but a key implementation detail is altered in a way that breaks privacy.
Noise / sensitivity miscalibration
Noise is added, but not at the level required by the true sensitivity or correct accounting.
Assumption mismatch / neighboringrelation change
The claim changes the assumptions, adjacency relation, norm bound, or sampling model while keeping the statement superficially similar.
Unprivatized data-dependent component
A control-flow, hyperparameter, initialization, or stopping rule depends on private data without being privatized.
Public/private boundary mistake
Quantities such as dataset size n or N are treated as public when they are private under the stated neighboring relation.
Conceptual impossibility / wrong privacy notion
The claim is false at the conceptual level, such as invalid conversions between DP notions or privacy guarantees that are impossible in general.
Natural mistake
The error originates in the literature
16
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
C
Experiment Details
C.1
Model Details
We provide version information of models in the following table:
Table 8: Version information of Models Model
C.2
Provider
Version/Link/Release-date
GPT-5-High GPT-5-Minimal Gemini-3-Pro Gemini-2.5-Flash Claude-Sonnet-4.5 Claude-Opus-4.5
OpenAI OpenAI Google DeepMind Google DeepMind Anthropic Anthropic
2025-08-07 2025-08-07 November 18, 2025 June 17, 2025 September 29, 2025 November 24, 2025
DeepSeek-R1 DeepSeek-V3.1-chat Qwen3-30-Instruct Qwen3-30-Think Goedel-Prover-V2
Open-sourced Open-sourced Open-sourced Open-sourced Open-sourced
https://huggingface.co/deepseek-ai/DeepSeek-R1 https://huggingface.co/deepseek-ai/DeepSeek-V3.1 https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507 https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507 https://huggingface.co/Goedel-LM/Goedel-Prover-V2-32B
Experiment details for evaluation Category 1 and Category 2
For each question, we repeat the experiment with five random seeds. The prompt we use is as follows: QA prompt for testing Category 1 and Category 2 {Question} Please provide some explanations or deriviations first and then provide your final answer ‘yes‘ or ‘no‘ inside a Latex boxed format ‘\\boxed{}‘
C.3
Experiment details for paraphrasing
In this section we evaluate: Do LLMs rely on memorization for positive instances? Standard DP textbooks and research papers may have appeared in model pretraining data, and many positive instances in our benchmark are faithful re-statements of results with the same notations established in the literature. This raises the concern that LLMs might rely on superficial pattern matching – simply answering “yes” when a question resembles previously seen material. To probe this possibility, we conduct an ablation study in which all positive instances in Category 2 are paraphrased and re-evaluated using the two best-performing models, GPT-5-High and Gemini-3-Pro. As shown in Table 9, accuracy changes only marginally, suggesting that performance is not driven solely by memorization but also the understanding.
Table 9: Average accuracy on paraphrased positive questions, results are reported in format Mean Accuracy± standard deviation
C.3.1
Model
After Paraphrase
Before Paraphrase
Gemini-3-pro GPT-5-High
0.810 ± 0.024 0.771 ± 0.055
0.805 ± 0.039 0.771 ± 0.027
Paraphrasing Set-up
For each positive question in Category 2, we use GPT-5 to generate a paraphrased version. We then let a human expert verify that the paraphrase preserves the original question’s meaning and fixes any LaTeX compilation errors when necessary. The system prompt used for paraphrasing is provided below. 17
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
System prompt for paraphrasing question You are given a technical description of a differential privacy mechanism. Task: - paraphrasing, change notation and algorithm style, but don’t change name of algorithm - Change the structure of the algorithm (change the order of the steps, represent the operations in a different way) Constraints: - Preserve exact mathematical meaning. - Change all variable names, symbols, and function identifiers. - Rewrite the algorithm using a different presentation style. - Do NOT change the algorithm name. - ALL mathematical symbols, formulas, and equations MUST be written in valid LaTeX. - Do NOT introduce non-LaTeX math notation. - Do NOT add explanations, commentary, or assumptions. - Output ONLY the paraphrased LaTeX content.
We identified some interesting patterns in the paraphrased questions. In some cases, the paraphrased versions exhibit only symbolic differences or superficial changes in structure, as shown in Figure 3, panels (a) and (a). However, the structure of the paraphrased algorithm can also change in a more substantive way. As shown in Figure 3 (c) and (d), the high-level description of the exponential mechanism (Lines 2-3 of Figure 3c) has been paraphrased to inverse CDF sampling (Lines 3-4 of Figure 3d).
(a) Question 107: Original
(b) Question 107: Paraphrased
(c) Question 62: Original
(d) Question 62: Paraphrased
Figure 3: Examples of paraphrased questions. 18
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
C.4
Experiment details for Augmented QA prompt
C.4.1
QA prompt augmented with relevant theorems
QA prompt augmented with relevant theorems {Question description} You may use the following theorems or definitions if applicable. [THEOREM] {Theorem} [/THEOREM] Please provide explanations or derivations first. If you use any theorem or definition, cite it by its ID in the reasoning. The very last line of your response must be exactly ‘\\boxed{yes}‘ or ‘\\boxed{no}‘. Do not include citations or any other text on the last line. C.4.2
QA prompt with one-shot proof template
QA template for one-shot reasoning on LaplaceRNM task Below is an example of the question. Example Question: For each item j ∈ F , define a score sj (D) with sensitivity 1. Consider the mechanism that samples independent noise Zj Laplace(2/ϵ) for each j and outputs argmaxj (sj (D) + Zj ). Claim: This mechanism satisfies ϵ-differential privacy. Now answer the following question. {Question description} Please provide some explanations or deriviations first and then provide your final answer ‘yes‘ or ‘no‘ inside a Latex boxed format ‘\\boxed{}‘.
D
Additional Experimental Results for Category 2 Topic DP Adam Quantile Estimation Privacy Accounting DP Gradient Descent Privacy Amplification via Subsampling Private Selection Sparse Vector Technique Mean Estimation DP Linear Regression Sequential/Adaptive Composition Objective Perturbation DP Stochastic Gradient Descent Parallel Composition DP Hyperparameter Optimization Propose–Test–Release (PTR) Private Aggregation of Teacher Ensembles (PATE) Output Perturbation Smooth Sensitivity
Gemini-3-Pro
GPT-5-High
Avg Accuracy
1.00 1.00 0.92 0.92 1.00 0.80 0.92 0.87 0.80 0.91 0.80 0.78 0.71 0.60 0.70 0.67 0.33 0.33
1.00 1.00 0.95 0.94 0.76 0.96 0.80 0.85 0.87 0.71 0.80 0.80 0.81 0.77 0.65 0.33 0.60 0.33
1.00 1.00 0.93 0.93 0.88 0.88 0.86 0.86 0.83 0.81 0.80 0.79 0.76 0.68 0.68 0.50 0.47 0.33
Table 10: The performance of GPT-5-High and Gemini-3-Pro on Category 2 by topics. We sort the topics by the mean accuracy of two models.
E
Benchmark Examples
Example of the positive question in category 2: Suppose that for all z ∈ Z, ℓ(·, z) is twice-differentiable, and the rank of its Hessian ∇2 ℓ(w, z) at any w ∈ W is at most 1. Also assume that the smoothness parameter satisfies β ≤ εnλ. Is the following algorithm(ε, δ)-differentially private? 19
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
Input: Private dataset S = (z1 , . . . , zn ) ∈ Z n , L-Lipschitz, β-smooth, convex loss function ℓ, convex set W ⊆ Rd , privacy parameters ε ≤ 1, δ ≤ 1/n2 , regularization parameter λ. 10L2 log(1/δ) ; ε2 ⟨G, w⟩ 1 Pn ℓ(w, zi ); 2: return ŵ = arg minw∈W L̂(w; S) + + λ∥w∥2 , where L̂(w; S) ≜ n n i=1 Algorithm 3: AObjP : Objective Perturbation
1: Sample G ∼ N (0, σ 2 Id ), where σ 2 =
Example of the negative question (assumption perturbation) in category 2: Suppose that for all z ∈ Z, ℓ(·, z) is twice-differentiable, and the rank of its Hessian ∇2 ℓ(w, z) at any w ∈ W is at most 1. Also assume that the smoothness parameter satisfies β ≤ εnλ. Is the following algorithm(ε, δ)-differentially private? loss function ℓ, privacy parameters convex Input: Private dataset S = (z1 , . . . , zn ) ∈ Z n , L-Lipschitz, β-smooth, 2 ε ≤ 1, δ ≤ 1/n , regularization parameter λ. 10L2 log(1/δ) ; ε2 ⟨G, w⟩ 1 Pn 2: return ŵ = arg minw∈W L̂(w; S) + + λ∥w∥2 , where L̂(w; S) ≜ ℓ(w, zi ); n n i=1 Algorithm 4: AObjP : Objective Perturbation
1: Sample G ∼ N (0, σ 2 Id ), where σ 2 =
Example of the negative question (algorithm perturbation) in category 2: Suppose that for all z ∈ Z, ℓ(·, z) is twice-differentiable, and the rank of its Hessian ∇2 ℓ(w, z) at any w ∈ W is at most 1. Also assume that the smoothness parameter satisfies β ≤ εnλ. Is the following algorithm(ε, δ)-differentially private? Input: Private dataset S = (z1 , . . . , zn ) ∈ Z n , L-Lipschitz, β-smooth, convex loss function ℓ, convex set W ⊆ Rd , privacy parameters ε ≤ 1, δ ≤ 1/n2 , regularization parameter λ. 10L2 log(1/δ) ; ε2 ⟨G, w⟩ λ 1 Pn 2: return ŵ = arg minw∈W L̂(w; S) + + 2 ∥w∥2 , where L̂(w; S) ≜ ℓ(w, zi ); n n i=1 Algorithm 5: AObjP : Objective Perturbation
1: Sample G ∼ N (0, σ 2 Id ), where σ 2 =
Example of the negative question (conclusion perturbation) in category 2: Suppose that for all z ∈ Z, ℓ(·, z) is twice-differentiable, and the rank of its Hessian ∇2 ℓ(w, z) at any w ∈ W is at most 1. Also assume that the smoothness parameter satisfies β ≤ εnλ. Is the following algorithm(ε/2, δ)-differentially private? Input: Private dataset S = (z1 , . . . , zn ) ∈ Z n , L-Lipschitz, β-smooth, convex loss function ℓ, convex set W ⊆ Rd , privacy parameters ε ≤ 1, δ ≤ 1/n2 , regularization parameter λ. 10L2 log(1/δ) ; ε2 ⟨G, w⟩ 1 Pn 2: return ŵ = arg minw∈W L̂(w; S) + + λ∥w∥2 , where L̂(w; S) ≜ ℓ(w, zi ); n n i=1 Algorithm 6: AObjP : Objective Perturbation
1: Sample G ∼ N (0, σ 2 Id ), where σ 2 =
F
Additional failure mode examples
Another example of “Failure to identify subtle but semantically significant changes" arises in mean estimation, as shown below. Although the expression appears correct at first glance, the summation index ranges from 1 to n − 1 rather than the standard 1 to n, which can double the sensitivity. The LLMs fail to detect this discrepancy and instead assume the summation still ranges from 1 to n in their derivations. For this question, both GPT-5-High and Gemini-3-Pro succeed in only one trial. 20
DPrivBench: Benchmarking LLMs’ Reasoning for Differential Privacy
Q7 (Label is No) Does the following algorithm satisfy ε-differential privacy under the add/remove neighboring relation? Input :Input scalar dataset D := {x1 , . . . , xn }, privacy budget ε, clipping threshold C Sample ξ1 ∼ Lap(2C/ε) Sample ξ2P∼ Lap(2/ε) Let θ̂ ←
return θ̂
n−1 i=1 min{max{−C,xi },C}+ξ1
n+ξ2
Algorithm 7: Mean estimation
21