Differentially Private Mixing of Public Datasets Improves Private Learning

Differentially Private Mixing of Public Datasets Improves Private Learning

arXiv:2610.06636v1 [cs.LG] 5 Oct 2026

Yufei Chen1 , Tejumade Afonja2 , Anvith Thudi1 , and Nicolas Papernot1 1

2

University of Toronto and Vector Institute, Toronto, Canada CISPA Helmholtz Center for Information Security, Saarbrucken, Germany

Abstract Many machine learning applications involve sensitive data and therefore require training under differential privacy (DP). However, DP training often degrades model utility. In some cases, first pre-training the model on "public" data before finetuning with DP on the sensitive data can reduce the drop in utility. However, the success of this depends on how relevant the selected public dataset is to the sensitive data. We introduce the first pipeline that privately learns the mixture of several public datasets to pretrain on for a given sensitive downstream task. Our key insight is that we can privately find the best mixture of multiple public datasets by privately learning a low-dimensional linear model. We tested our method on the NIH dataset for X-ray classification and the ENRON email dataset for language modeling. Applying our method to find tailored mixtures of X-ray datasets to pretrain on for diseases in the NIH ChestX-ray14 dataset, we improved macro AUC by up to 0.037 across privacy budgets compared to the baselines, with gains as large as +22.8% relative AUC on Cardiomegaly at ϵ = 1. For DP training on the ENRON dataset, pre-training on our mixture of The Common Pile (a collection of public-domain text datasets) decreased test perplexity by 16% relative to the baseline mixtures.

1

Introduction

For many machine learning tasks, the only data we have comes from sensitive sources, such as medical records [21] or conversations between individuals [26]. Models trained on such data, even if the data is anonymized, are known to leak information about the individuals who contributed to the training dataset [43, 7]. Training with differential privacy (DP) is the de-facto framework for addressing this concern [12, 11]; DP guarantees that the models trained using an individual’s data are indistinguishable to the models trained without their data. However, training with DP often comes with a large drop in model utility [1, 9]. Past work has observed that first pre-training a model on public data (data that can be treated as non-sensitive, hence not contributing to privacy leakage) before DP finetuning the model on the sensitive data can sometimes reduce the drop in model utility [9, 50]. Importantly, the effectiveness of this pre-training depends on how related the selected public data is to the target (sensitive) data [5, 19, 40, 10, 14, 46]. To the best of our knowledge, pre-training on all the public datasets uniformly is common practice and state-of-the art for privately mixing the public dataset: we improve this. In this paper, we study how to privately choose the public pre-training mixture that gives the best downstream performance on a given sensitive dataset. Finding the best mixture privately can 1

D1

Dk

Stage 1

Learn the mixture privately

Stage 2

Pretrain, then DP fine-tune

Proxy models

DP mixture learning

Pretraining

DP fine-tuning

Di sampled w.p. λi

DP-SGD

f̂1

f̂k

DP-SGD minλ L(f̂λ (x), y ) clip + noise

priv

Dλ

ρ of the data

Mixture λ

Pretrained model

λ1

λ2

λk

Sensitive dataset

split into disjoint parts

clip + noise

Private model (ϵ, δ )-DP

priv

Dft

the rest, 1 − ρ

Figure 1: Overview of DP-MixMin. Proxy models trained on public data are combined with mixture weights λ learned with DP-SGD on a fraction ρ of the sensitive data (split A). We then pretrain on public data according to λ and DP fine-tune on the rest (split B). generically be approached with private hyperparameter selection techniques [30, 37], where one trains on several different mixtures and then releases the best found mixture. However, this comes with significant computational costs: the space of mixtures which we need to trial (i.e., train on) grows exponentially with the number of public datasets. Compounding this, every candidate mixture we evaluate touches the sensitive data, so the more mixtures we try, the higher the privacy cost [37]. We show that it is possible to significantly improve downstream utility over standard approaches to public data mixing. Our key insight is that we can privately learn the weighting of public datasets by privately optimizing a convex objective. This builds off MixMin [45], a non-private data mixing method which showed that as models approach Bayes optimal (the best a model can do with enough compute and data), finding the best weighting of pre-training datasets for a downstream task reduces to minimizing a convex objective. We build onto this insight by realizing this objective is amenable to private learning with little utility degradation. Formally, if k is the number of public datasets, by using √ existing DP algorithms [3, 13, 2, 4] we have the privacy cost for a fixed error grows at worst O( k). In contrast, for certain data mixing problems, private hyperparameter search would need to try exponentially many mixtures, which translates to a privacy cost that grows linearly in k [37]. However, a key limitation of this convex objective is that it requires having models near Bayes optimal on each of the individual pre-training sources; we hence explore empirical approaches. Our pipeline, DP-MixMin, empirically instantiates the DP convex optimization for data mixing alongside further DP finetuning (Figure 1); our pipeline provides one end-to-end privacy guarantee. We first split the sensitive data into two disjoint parts: one for mixture learning with a variant of DP-SGD [1], and the other for DP fine-tuning the model pre-trained on the mixture. Concretely, we empirically learn the mixture by replacing the Bayes optimal models for each source with a proxy model trained on each individual public dataset (following past work [45]); these models do not see the private data, so leak no privacy. We then optimize the convex objective for data mixing with DP-SGD, providing a DP bound for the mixture. In the final stage we pre-train on this mixture, and then apply DP finetuning on the held out set. Because the two DP stages (mixing and fine-tuning) use disjoint datasets, parallel composition [11] bounds the pipeline’s overall privacy cost by the worse of the two steps, not their sum. So learning the mixture costs data, not privacy. DP-MixMin fine-tunes on less data than our baseline of finetuning a model trained uniformly over all public datasets, but we show 2

1

2 Privacy budget

5

0.75 0.73 0.71 0.69 0.67 0.65

better

Uniform

MIMIC-CXR

Random Search

NIH

Test perplexity

better

DP-MixMin

NIH Macro AUC

Macro AUC

0.75 0.73 0.71 0.69 0.67 0.65

CheXpert

1

2 Privacy budget

5

21 20 19 18 17 16 15

Enron (user-level DP)

better 1

2 Privacy budget

5

Figure 2: DP-MixMin beats Uniform and Random Search at every ε. Macro AUC on CheXpert/MIMIC-CXR→NIH (↑) and Enron test perplexity at ρ = 0.3 (↓). that better mixtures more than make up for it. We empirically tested DP-MixMin across medical imaging and language tasks, and found across varying DP guarantees it consistently improved baselines (Figure 2). For medical imaging, we used the 14 disease classification tasks in the NIH chest X-ray dataset [47] as sensitive datasets, and CheXpert [18] and MIMIC-CXR [21] as public sources. These sources also contain classification tasks over chest X-rays, but differ in the diseases and how the images were labelled. DP-MixMin consistently outperformed the baselines, with an average AUC gain of ≈ 0.03, including a 22.8% relative gain on the Cardiomegaly class. For language modeling, we used Enron [26] as the sensitive dataset, targeting user-level DP; protecting all of a user’s emails, a strictly stronger and costlier guarantee than protecting single emails [23, 28, 8]. Pre-training on mixtures of The Common Pile [24] decreased test perplexity by up to 16%, with similar gains under example-level DP on OpenWebMath [38]. We also found that DP-MixMin recovered mixtures close to the non-private solution in most settings, despite using a fraction of the sensitive data. To conclude, our contributions are: 1. DP-MixMin: The first end-to-end privatized pretraining mixture-learning pipeline. 2. Theoretically showing that when we train models near Bayes optimal, our procedure asymptotically outperforms the baseline of private hyperparameter search. 3. Empirically showing that across medical imaging and language modeling tasks, DP-MixMin improves the performance of public-pretraining over past mixture finding approaches.

2

Related Work

We now discuss past work on DP finetuning and non-private mixing we build on, and the limitations of previous public data selection for DP learning: they could not improve over a uniform mixture. DP public pretraining and non-private data mixing. Public pretraining followed by private finetuning drives state-of-the-art DP results across language [29, 50] and vision [9, 34], and its effectiveness depends heavily on which public data is used [14, 46]. Non-privately, optimizing mixtures over many sources has shown to provide large gains in pre-training performance [15, 48, 31, 17, 45]; among these, MixMin [45], which we build on, showed that the mixture-learning objective is (asymptotically) equivalent to fitting a linear model over per-source proxy model predictions. 3

Prior work on private selection of public datasets. Gu et al. [16] propose GSD, which scores a candidate public dataset’s relevance by the distance between its gradients’ top-k subspace and the private task’s, and picks the single source with the smallest distance to the private task. However, GSD breaks down by construction when the public sources share the same input space X : their scoring uses a single model’s gradient on randomly labeled examples from each source, so two sources with identical inputs become indistinguishable regardless of their actual label distributions.1 Gu et al. [16] also present no end-to-end private experiments—the privatized version of the selection step is proposed but never empirically validated. Our method, DP-MixMin, can operate when input distributions are shared and provides DP guarantees. Yu et al. [51] train a DP domain classifier to rank examples within a single pre-chosen public corpus and pretrain on the top-ranked subset. While their method improves over the unfiltered baseline at smaller model scales, the benefit is no longer apparent at GPT2-Medium (355M parameters) 2 . In contrast, DP-MixMin continues to deliver substantial gains over unweighted pretraining at GPT2-Medium scale.

3

Preliminaries

In this section, we review differential privacy (DP), data mixing, and relevant past results. Differential privacy. A randomized mechanism M : D → R satisfies (ϵ, δ)-differential privacy [12] if, for every pair of adjacent datasets D, D′ ∈ D (differing in a single record) and every measurable set S ⊆ R, Pr[M(D) ∈ S] ≤ eϵ Pr[M(D′ ) ∈ S] + δ, where ϵ ≥ 0 is the privacy budget and δ ∈ [0, 1] bounds the probability of failure. We will use DP-SGD, which privatizes the SGD algorithm by clipping individual gradients and adding noise [1], and the post-processing property of DP which states any function (independent of the private dataset) of a DP algorithm is also a DP mechanism with the same guarantees [11]. We also leverage parallel composition, which states that if two DP mechanisms operate on disjoint parts of the private dataset, then the overall DP guarantee of releasing both mechanisms is at most the worst DP guarantee of the two mechanisms [33]. User-level differential privacy. The definition above protects individual records (record-level or example-level DP). However, a single individual may contribute multiple records, and record-level guarantees can degrade substantially for such individuals. User-level DP [32, 28] aims to resolve this. S Suppose each dataset D is partitioned by users D = u∈U Du , where Du is the (possibly multi-record) contribution of user u. Two datasets D, D′ are user-adjacent if they only differ by the Du of a single user. A mechanism M satisfies (ϵ, δ) user-level DP if the inequality above holds for every pair of user-adjacent datasets and every measurable S ⊆ R. Note that while Record-level DP implies user-level DP through group privacy [11], this bound is often loose. User-level guarantees are typically obtained directly without relying on record-level DP methods, e.g., by running DP-SGD [1] with sampling, clipping, and noise applied per user rather than per example [32]. User-level DP satisfies both post-processing and parallel composition, provided the private dataset is split along user boundaries. 1We also implemented GSD [16], and confirmed we could not obtain meaningful results in our setting; implementation details and results are provided in Appendix A.4. 2 Figure 2 in Yu et al. [51] shows selective pretraining on 15% of OpenWebText and full pretraining on 100% of OpenWebText reaches the same perplexity.

4

Data mixing setup. We consider learning from a collection of k public datasets {Dipub }ki=1 in order to improve performance on a private dataset. Each public dataset Dipub is assumed to be drawn from an underlying source distribution di over a shared input–output domain X × Y. The private downstream dataset Dpriv is drawn from a target distribution dt , which is also over X × Y. Let H denote the hypothesis space for our models. We represent models as functions f : X → O, where O is some output space (e.g., label probabilities), and we evaluate performance using a loss function L : O × Y → R+ . We require the loss function to be either cross-entropy (CE) or mean squared error (MSE), as required for the MixMin algorithm which we adapt [45]. Our goal is to learn sampling weights λ ∈ ∆k over the public datasets. Operationally, we sample an index i ∼ λ and then a data Pk point from Dipub , giving samples from dλ = i=1 λi di . To simplify data mixing formulations, we assume no covariate shift among public sources: di (x) = dj (x) for all i, j. This assumption is used only to simplify how we frame the mixture problem, and it is also trivially satisfied in our experiments (i.e., generative tasks have no "covariates", and disease classification is done over the same distribution of images). In general, if there is covariate shift, one should use a more general data mixing objective that optimizes the joint distribution over the inputs and outputs to be the same as the target distribution (see Section 6.1 in Thudi et al. [45]). Data mixing as bi-level optimization. The data mixing problem asks which λ ∈ ∆k yields a model trained on dλ that performs best on dt . This is a bi-level optimization,     min E(x,y)∼dt L(fλ (x), y) where fλ ∈ arg min E(x,y)∼dλ L(f (x), y) , (1) f ∈H

λ∈∆k

that is generally hard to solve: the inner minimization over H typically has to be run for many candidates λ. Beyond computational complexity, we note that private hyperparameter search (which optimizes this bi-level problem with random search) also has worse privacy-utility trade-offs. MixMin: a convex reduction. Thudi et al. [45] showed that under the assumptions above, as our model fλ approaches Bayes optimal for dλ (i.e., large compute and data limit), eq. (1) reduces to a single convex problem in λ. Letting fi be the Bayes optimal model for each di , the optimal λ solves: min E(x,y)∼dt L

λ∈∆k

k X

λi fi (x), y .

(2)

i=1

In practice, it was found that running this optimization with empirically trained proxy models fˆi still outperformed other mixture-finding methods [45].

4

Differentially Private Data Mixing

In this section we first highlight how privately optimizing the reduced data mixing objective (eq. (2)) leads to better utility-privacy trade-offs than private hyperparameter search. We then describe our empirical method for privately optimizing the reduced objective, which we call DP-MixMin.

4.1

Improved Utility over Private Hyperparameter Search Asymptotically

Privately optimizing the MixMin reduction leads to asymptotically better privacy-utility trade-offs than private hyperparameter search using Poisson sampling [37] as we increase the number of public datasets 5

k. We prove that there are cases where private hyperparameter search requires the DP parameter ϵ to grow at least linearly with k, for a fixed error. In contrast, existing DP stochastic convex optimization √ (SCO) algorithms applied to the MixMin reduction only require the DP guarantee to grow with k, meaning we can achieve better privacy at a given utility. We state this formally in Lemma 4.1, where we suppress constants that do not depend on k. We quantify utility by the excess error of the recovered mixture λ̂ relative to the optimal mixture: h P i   k α := E F (λ̂) − F (λ⋆ ), F (λ) = E(x,y)∼dt L , (3) i=1 λi fi (x), y where F is the MixMin objective of eq. (2), λ⋆ = arg minλ∈∆k F (λ) is its minimizer. Lemma 4.1. Suppose the conditions for the MixMin convex data mixing reduction hold (eq. (2)). There exists a Lipschitz data mixing objective such that private hyperparameter search with Poisson sampling requires ϵ = Ω(k) to achieve a fixed (sufficiently small) excess error α with constant probability.√In contrast, there exist DP-SCO algorithms that, for any Lipschitz data mixing objective, have ϵ = O( k) to achieve excess error α, for mixture-learning data size n = Ω(1/α2 ). The key idea of the proof is to consider a specific Lipschitz data mixing problem where finding a good mixture is equivalent to finding a mixture within an ℓ2 ball around the best mixture. We then show the probability of uniformly sampling a mixture λ ∈ ∆k that is also in this ball vanishes exponentially with k, which implies private hyperparameter search needs to try many random mixtures and hence use larger privacy budget ϵ. In contrast, we are able to apply existing DP-SCO results to any reduced Lipschitz data mixing objectives to show we can achieve much less error at lower privacy budgets. The full proof is in Appendix A.1. In Appendix A.6 we empirically show our proposed DP √ data mixing algorithm (Algorithm 1) follows an O( k) excess error rate. While Lemma 4.1 shows optimizing eq. (2) should lead to better utility, this assumes we can train models close to Bayes optimal. We turn to how we empirically approximate eq. (2).

4.2

Implementing DP-MixMin

We now explain how we empirically optimize the MixMin objective (eq. (2)), train the proxy models fˆi which we use instead of the Bayes optimal fi , and how we allocate the privacy budget for private mixture finding and finetuning. The overall pipeline is summarized in Figure 1. Making MixMin Differentially Private. To empirically optimize eq. (2) with DP, we replace the Bayes optimal models fi with models fˆi trained empirically on Dipub , and the expectation over dt with the empirical expectation over a dataset Dλpriv ∼ dt . To now optimize this we run a post-processing of the DP-SGD [1] update: for each iteration, per-example gradients w.r.t λ are computed, clipped to a fixed norm, and perturbed with Gaussian noise before being aggregated. The resulting noisy gradient is then passed to the entropic-descent update. The overall procedure satisfies (ϵ, δ)-DP by the DP guarantees of the sampled Gaussian mechanism [36], post-processing [11], and composition theorems to accumulate DP guarantees across steps [35] 3 . See Algorithm 1, with blue text describing changes to the non-private optimization described by Thudi et al. [45]. See Algorithm 2 in the Appendix for user-level DP. 3 In particular, we use the DP accountant for DP-SGD provided by Opacus [49]

6

Algorithm 1 DP-MixMin Require: Target privacy guarantees (ϵ,δ), step size η, number of steps n, loss function L (CE or MSE), private mixture-learning split Dλpriv ⊆ X × Y, proxy models {fˆi : i = 1, . . . , k}, clipping norm C 1 1: Initialize λi ← k for all i = 1, . . . , k priv ˆ 2: Pre-compute {fi (x) : x ∈ Dλ , i = 1, . . . , k} 3: for j = 1 to n do Pk 4: fˆλ (x) ← i=1 λi fˆi (x) 5: ℓ(x, y) ← L(fˆλ (x), y) for all (x, y) ∈ Dλpriv 6: g(x, y) ← ∇λ ℓ(x, y) 7: g̃(x, y) ← PClip(g(x, y), C) {clip per-example gradients} 8: gsum ← (x,y)∈Dpriv g̃(x, y) λ 9: z ∼ N (0, σ 2 I) {noise calibrated to (ϵ, δ)} 1 (g + z) 10: ḡ ← |Dpriv | sum λ

−ηḡi for all i = 1, . . . , k λi ← λi eP k 12: λ ← λ / i=1 λi {project onto ∆k } 13: end for 14: Return λ

11:

Reducing Proxy Model Costs. Past work on empirically optimizing eq. (2) required training a separate proxy model on each public dataset [45]. We reduced costs for classification tasks by training a single multi-class proxy model to jointly predict the labels for each public dataset: fˆ : X → Ok . To do so, we concatenated the label spaces of all k public datasets (representing labels as one-hot or multi-hot vectors depending on the datasets), and trained the proxy model to output a probability vector over this combined label space. The proxy model for the i-th dataset is then obtained as fˆi (x) = fˆ(x)[i], i.e., the i-th coordinate of the output. If learned to Bayes optimality, this proxy would provide the same output probabilities as a model trained only on Dipub . We found this significantly reduced training and storage costs when scaling the number of public datasets to mix. For generative language modeling, we followed the standard procedure of training an individual model on each public dataset. Further details are provided in Section 5.2. Privacy budget allocation. We split the private target dataset Dpriv into two disjoint subsets: Dλpriv , which is used to compute the mixture weights (Algorithm 1), and Dfpriv t , which is used to fine-tune the pre-trained model. By parallel composition [11], if the mechanism used for mixture learning is (ϵ1 , δ1 )-DP and the mechanism used for finetuning is (ϵ2 , δ2 )-DP, then the overall guarantee
is: (ϵ, δ) = max(ϵ1 , ϵ2 ), max(δ1 , δ2 ) . Note that the baseline of pre-training on the uniform mixture uses all of Dpriv for finetuning; we found that DP-MixMin improves over uniform mixtures, showing that we can use less data for finetuning if we optimize the mixture.

5

Experiments

We compared the full DP-MixMin pipeline—private mixture learning, mixture-weighted public pretraining, and DP finetuning—against three baselines: 1. Random Initialization: privately fine-tune with no public pretraining, isolating the impact of public pretraining. 7

2. Uniform Pretraining: pretrain with equal weight on every public source, isolating the value of learning a target-specific public-data mixture. 3. Random Search (non-private): select the best mixture from a random sample of candidate mixtures [6, 37] by training a proxy model on each one and evaluating performance on the downstream task. The selection of the best mixture from the random sample was done nonprivately, so this over-estimates the performance of private hyperparameter search. Full details are in Appendix A.10. We evaluated these methods’ downstream utility on the private dataset, across privacy budgets, sources of public data, and modalities. We also compare DP-MixMin to non-private MixMin.

5.1

Image Classification

We evaluated DP-MixMin for chest X-ray disease classification. Prior work applied DP-SGD to train models from scratch on these datasets [44, 16], which corresponds to our random initialization baseline. We found mixtures of other chest X-ray datasets (for different diseases) so that pretraining on them improves performance. 5.1.1

Experimental Setup

Datasets and Models. We considered two different collections of pretraining datasets: (1) CheXpert [18] and (2) MIMIC-CXR [22], both standard public sources for chest X-ray transfer learning [41, 16]. Note each of these datasets also consist of 14 chest X-ray disease classification tasks, but differ in the diseases and how the images were labelled. We used a ResNet-50 as the proxy model and a randomly initialized ResNet-18 for downstream pretraining and finetuning. The smaller downstream model reduced the cost of pretraining, allowing us to run more experiments within our compute budget. Data preprocessing and computational usage For all three chest X-ray datasets, CheXpert, MIMIC-CXR, and NIH, uncertain labels (−1) were mapped to 0 (negative); 0 and 1 were kept as-is. The total compute used was ∼ 20,000 GPU hours over a cluster of L40 GPUs. Methodology. For each of the 14 downstream tasks, 80% of the NIH data was used for training and 20% was split between validation and test. DP-MixMin and Random Search accessed the private task only through parallel composition, using 10% of the training split for mixture learning and the rest for finetuning. Uniform pretraining used all of the training split for DP finetuning. We repeated each method 5 times with a different random partition of the NIH data into mixture-finding, finetuning, validation, and test sets, as well as a different random seed for model initialization and DP noise. See Appendix A.3.1 for details on training hyperparameters. We write ρ = |Dλpriv |/|Dpriv | for the fraction used for private mixture-learning: see Appendix A.2 for guidance on choosing it. 5.1.2

Image Classification Results

Improvements were consistent across downstream tasks. Across the 14 NIH tasks, we found DP MixMin always improved or was on-par (within one standard deviation) with the baselines of uniform pretraining and random search (Figure 3). This was the case whether CheXpert or MIMIC-CXR was used for the public datasets. In the case of Cardiomegaly, DP-MixMin’s gain over Random Init was roughly 5× Uniform’s at ϵ = 1. 8

DP-MixMin

Uniform

Random Search

(a) CheXpert ! NIH

(b) MIMIC-CXR ! NIH

Relative AUC gain over Random Init (%)

30 20 10 0

sis

ta

lec

e At

Co

. a a x y n n is le ia ck al io ra os sio em em on du hi at eg ho br fu m ys tr Ed No m ot lT Fi Ef eu ph fil io a m n r n d I r P eu eu Em Ca Pl Pn n

lid

o ns

io at

Macro: DP-MixMin +11.48%

Uniform +5.42%

s

as

M

ia

rn

He

sis

ta

c ele At

Co

n

io

at

lid

o ns

ilt

f In

n

tio

ra

P

ot um ne

x

ra

ho

Ed

a

em

Em

a

em

ys

ph

is

os

br

Fi

. y le ia al ick on du eg m Th No m eu io al n r d r P eu Ca Pl n

sio

fu

Ef

Macro: DP-MixMin +12.29%

RS +4.74%

Uniform +6.87%

s

as

M

ia

rn

He

RS +5.22%

Figure 3: Per-task relative AUC gain over
Random Init on NIH Chest X-ray at ϵ = 1. Bars show AUC(method) − AUC(Random Init) /AUC(Random Init) as a percentage, with propagated standard deviation over 3–5 seeds. The macro AUC gain (unweighted mean across 14 tasks) is reported next to each subplot title; DP-MixMin roughly doubles, and in most cases more than doubles, the gain of both Uniform and Random Search on each public pretraining source. Full raw AUCs are reported in Tables 7 and 8. Gains were consistent across privacy budgets DP-MixMin also outperformed the baselines at every privacy budget we tested: +0.037, +0.032, +0.029 absolute macro AUC gain over uniform at ϵ = 1, 2, 5 for CheXpert pretraining(Figure 2), and +0.033, +0.028, +0.024 for MIMIC-CXR. Tables 7 and 8 in the Appendix report absolute AUC across settings.

5.2

Language Models

We evaluated DP-MixMin on two downstream tasks: Enron [25] email completion with user-level DP and OpenWebMath [38] with example-level DP. For Enron we compared to uniform pretraining and random search, and for OpenWebMath we only compared to uniform as it had often outperformed the other baseline (see Figure 3 and Table 1). 5.2.1

Experimental Setup

Datasets and Models. We used 16 domains from The Common Pile [24] as candidate public pretraining sources, selected from the 18 domains containing at least 10GB of data 4 . For each domain we trained a GPT-2 Small (124M parameters) proxy on documents with at least 1024 GPT-2 tokens until roughly 45M tokens had been seen. The downstream model was GPT-2 Medium (355M parameters). DP finetuning was only done on the last two layers and output head of the model, due to memory allocation constraints on our GPUs. 4 Two were excluded due to streaming limitations.

9

Table 1: Enron test perplexity (mean ± std) after DP finetuning at δ=10−5 . DP-MixMin beats both Uniform and Random Search at every privacy budget. At ϵ=1, the mixture quality (and stability) improves monotonically with the mixture weight finding fraction ρ. Method

ρ

ϵ=1

ϵ=2

ϵ=5

DP-MixMin DP-MixMin DP-MixMin

0.3 0.2 0.1

16.78 ± 0.55 17.52 ± 1.55 17.97 ± 2.72

16.44 ± 0.95 — —

15.71 ± 0.77 — —

Uniform Random Search

— 0.3

19.88 ± 1.10 20.46 ± 0.70

19.18 ± 1.01 19.72 ± 0.66

18.24 ± 0.89 18.72 ± 0.62

Table 2: OpenWebMath test perplexity (mean ± std) after DP finetuning at δ=10−5 . DPMixMin lowers test perplexity by roughly 13% over uniform Common Pile pretraining at ϵ=1. The uniform baseline is essentially flat in ϵ on this domain, showing DP-MixMin at ϵ = 1 is still better. Method

ρ

ϵ=1

ϵ=2

ϵ=5

DP-MixMin

0.1

12.22 ± 0.17

—

—

Uniform

—

14.05 ± 0.17

14.04 ± 0.17

14.02 ± 0.17

Data preprocessing and computational usage. For Enron, we restricted to the _sent_mails and sent_mails folders, yielding 148 users and approximately 80,000 emails. This avoided multi-author records such as received emails, forwarded messages, and duplicated threads, which require more complex multi-attribution privacy treatment [39]. For OpenWebMath, we streamed 50,000 samples to form the dataset. The total compute was ≈ 5,000 L40 GPU-hours. Methodology For Enron, we ablated ρ = |Dλpriv |/|Dpriv | between {10%, 20%, 30%}. For OpenWebMath, DP-MixMin used 10%. See Appendix A.3.2 for further details. 5.2.2

Language Results

DP-MixMin’s utility advantage grew under tighter ϵ : As shown in Table 1, the relative perplexity reduction of DP-MixMin over the baselines widened as ϵ shrank on Enron: 13.9% at ϵ=5, 14.3% at ϵ=2, and 15.6% at ϵ=1. On OpenWebMath, we found the performance of uniform mixture weights was stable across ϵ = 1, 2, 5, and DP-MixMin improved over it at ϵ = 1 (Table 2). Public pretraining mixture mattered more than amount of finetuning data. While each of the mixtures at ratios ρ ∈ 0.1, 0.2, 0.3 outperformed uniform pretraining, allocating private data toward mixture estimation mattered more than reserving it for finetuning for Enron. The proxy-fraction ablation in Table 1 exhibited monotonic improvement with ρ in both mean perplexity (17.97 → 17.52 → 16.78) and cross-seed standard deviation (2.72 → 1.55 → 0.45).

5.3

Comparing DP-MixMin to Non-Private MixMin

Preventing privacy leakage. We audited whether the data mixture λ leaked private information in practice, following the canary-based auditing framework of Jagielski et al. [20] using both natural 10

and worst-case (max-clip) canaries. Across all 14 NIH tasks and both pretraining sources, the DP parameter measured by the audit, ϵLB , stayed far below the claimed ϵ = 1, with the highest being 0.131. Removing the DP noise from the same audit recovered ϵLB as high as 5.6, confirming that this step is genuinely privacy-sensitive and that DP is necessary (Tables 16, 17; further details in Appendix A.8). DP-MixMin algorithm often recovered non-private MixMin weights. To understand the impact of DP on mixture finding, we compared the DP-MixMin weights λDP to the non-private MixMin λ⋆ (which used the entire Dpriv training split for mixture finding). Across modalities, DPMixMin recovered mixtures close to λ⋆ even with small privacy budgets and limited private data (Tables 11, 14, 15 in the Appendix). By ϵ = 1, DP-MixMin achieved cosine similarity of 0.97 and 0.93 for CheXpert→NIH and MIMIC-CXR→NIH respectively. On Enron, DP-MixMin reached cosine similarity 0.92 having seen only 12 users at ϵ = 2 user-level DP. On OpenWebMath, using only 2.8M tokens (ρ = 0.1), DP-MixMin achieved cosine similarity above 0.9 at ϵ = 0.2. DP-MixMin’s downstream performance was on-par with non-private MixMin. Given the similarity in the weights, we further checked whether it translated to matched downstream utility. We compared the full DP-MixMin pipeline at ε = 1 to a non-private MixMin baseline that paid no privacy for mixture finding, and also reused the entire private dataset (both the mixing and finetuning splits) for weight-finding and DP finetuning steps. Across both pretraining sources, DP-MixMin was comparable to non-private MixMin: the macro AUC difference between DP-MixMin and non-private MixMin was only 0.717 vs. 0.723 for MIMIC-CXR and 0.711 vs. 0.712 for CheXpert. Most per-class differences lay within one standard deviation (Tables 19, 18).

Conclusion In this paper, we initiated the study of privately optimizing public-data pretraining mixtures for a target sensitive dataset. Across medical imaging and language modeling, DP-MixMin improved private downstream utility over uniform pretraining and random mixture search, with large gains under low-prevalence labels, tight privacy budgets, and user-level privacy. Extending DP-MixMin to tabular and other domain settings remains important future work. In particular, public tabular sources often differ in their input features, violating the no-covariate-shift assumption our reduction relies on (Section 3). Future work could also look into how to improve our current DP-MixMin pipeline. Initial results showed our approach was robust to less proxy models training (Appendix A.5) and whether parallel or sequential composition was used (Appendix A.7). However, the theoretical results of Bassily et al. [4] suggest using DP Franke-Wolfe instead of DP mirror descent.

Acknowledgements We acknowledge the following sponsors, who support our research with financial and in-kind contributions: CIFAR through the Canada CIFAR AI Chair, NSERC through the Discovery Grant and an Alliance Grant with CSE and wirh ServiceNow and DRDC, the Schmidt Sciences foundation through the AI2050 Early Career Fellow program. Anvith Thudi is supported by a Vanier Fellowship from the Natural Sciences and Engineering Research Council of Canada. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We thank the computing team at the University of

11

Toronto’s Computer Science Department for administering and procuring the compute infrastructure used for the experiments in this paper.

AI use statement We only used generative AI tools for formatting the calculations presented in Appendix A.2, debugging code, formatting the figures and tables in this paper, and checking if there was literature on optimizing public pretraining data we missed. All code and references were verified by the authors. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

References [1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016. [2] H. Asi, V. Feldman, T. Koren, and K. Talwar. Private stochastic convex optimization: Optimal rates in l1 geometry. In International Conference on Machine Learning, pages 393–403. PMLR, 2021. [3] R. Bassily, V. Feldman, K. Talwar, and A. Guha Thakurta. Private stochastic convex optimization with optimal rates. Advances in neural information processing systems, 32, 2019. [4] R. Bassily, C. Guzmán, and M. Menart. Differentially private stochastic optimization: New results in convex and non-convex settings. Advances in Neural Information Processing Systems, 34:9317–9329, 2021. [5] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine learning, 79(1):151–175, 2010. [6] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012. [7] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650, 2021. [8] L. Chua, B. Ghazi, Y. Huang, P. Kamath, R. Kumar, D. Liu, P. Manurangsi, A. Sinha, and C. Zhang. Mind the privacy unit! user-level differential privacy for language model fine-tuning. arXiv preprint arXiv:2406.14322, 2024. [9] S. De, L. Berrada, J. Hayes, S. L. Smith, and B. Balle. Unlocking high-accuracy differentially private image classification through scale. arXiv preprint arXiv:2204.13650, 2022. [10] R. Dominguez-Olmedo, F. E. Dorner, and M. Hardt. Training on the test task confounds evaluation and emergence. arXiv preprint arXiv:2407.07890, 2024. [11] C. Dwork and A. Roth. The algorithmic foundations of differential privacy. Foundations and trends® in theoretical computer science, 9(3-4):211–487, 2014. 12

[12] C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265–284. Springer, 2006. [13] V. Feldman, T. Koren, and K. Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 439–449, 2020. [14] A. Ganesh, M. Haghifam, M. Nasr, S. Oh, T. Steinke, O. Thakkar, A. Thakurta, and L. Wang. Why is public pretraining necessary for private model training? arXiv preprint arXiv:2302.09483, 2023. [15] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027. [16] X. Gu, G. Kamath, and Z. S. Wu. Choosing public datasets for private machine learning via gradient subspace distance, 2025. URL https://arxiv.org/abs/2303.01256. [17] S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. [18] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, J. Seekins, D. A. Mong, S. S. Halabi, J. K. Sandberg, R. Jones, D. B. Larson, C. P. Langlotz, B. N. Patel, M. P. Lungren, and A. Y. Ng. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison, 2019. URL https://arxiv.org/abs/1901.07031. [19] B. Isik, N. Ponomareva, H. Hazimeh, D. Paparas, S. Vassilvitskii, and S. Koyejo. Scaling laws for downstream task performance of large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. [20] M. Jagielski, J. Ullman, and A. Oprea. Auditing differentially private machine learning: How private is private sgd?, 2020. URL https://arxiv.org/abs/2006.07709. [21] A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1):317, 2019. [22] A. E. W. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C. ying Deng, Y. Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs, 2019. URL https://arxiv.org/abs/1901.07042. [23] N. Kandpal, K. Pillutla, A. Oprea, P. Kairouz, C. A. Choquette-Choo, and Z. Xu. User inference attacks on large language models. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 18238–18265, 2024. [24] N. Kandpal, B. Lester, C. Raffel, S. Majstorovic, S. Biderman, B. Abbasi, L. Soldaini, E. Shippole, A. F. Cooper, A. Skowron, J. Kirchenbauer, S. Longpre, L. Sutawika, A. Albalak, Z. Xu, G. Penedo, L. B. Allal, E. Bakouch, J. D. Pressman, H. Fan, D. Stander, G. Song, A. Gokaslan, T. Goldstein, B. R. Bartoldson, B. Kailkhura, and T. Murray. The common pile v0.1: An 8tb dataset of public domain and openly licensed text, 2025. URL https://arxiv.org/abs/2506.05209. 13

[25] B. Klimt and Y. Yang. The enron corpus: A new dataset for email classification research. In J.-F. Boulicaut, F. Esposito, F. Giannotti, and D. Pedreschi, editors, Machine Learning: ECML 2004, pages 217–226, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg. ISBN 978-3-540-30115-8. [26] B. Klimt and Y. Yang. Introducing the enron corpus. In CEAS, volume 45, pages 92–96, 2004. [27] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. [28] D. Levy, Z. Sun, K. Amin, S. Kale, A. Kulesza, M. Mohri, and A. T. Suresh. Learning with user-level privacy. Advances in Neural Information Processing Systems, 34:12466–12479, 2021. [29] X. Li, F. Tramer, P. Liang, and T. Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations, 2022. [30] J. Liu and K. Talwar. Private selection from private candidates. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, 2019. [31] Q. Liu, X. Zheng, N. Muennighoff, G. Zeng, L. Dou, T. Pang, J. Jiang, and M. Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024. [32] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations (ICLR), 2018. URL https://openreview.net/forum?id=BJ0hF1Z0b. [33] F. D. McSherry. Privacy integrated queries: an extensible platform for privacy-preserving data analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, pages 19–30, 2009. [34] H. Mehta, A. Thakurta, A. Kurakin, and A. Cutkosky. Large scale transfer learning for differentially private image classification. arXiv preprint arXiv:2205.02973, 2022. [35] I. Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263–275. IEEE, 2017. [36] I. Mironov, K. Talwar, and L. Zhang. R\’enyi differential privacy of the sampled gaussian mechanism. arXiv preprint arXiv:1908.10530, 2019. [37] N. Papernot and T. Steinke. Hyperparameter tuning with renyi differential privacy. In International Conference on Learning Representations, 2022. [38] K. Paster, M. D. Santos, Z. Azerbayev, and J. Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023. URL https://arxiv.org/abs/2310.06786. [39] N. Ponomareva, Z. Xu, H. B. McMahan, P. Kairouz, L. Rosenblatt, V. Cohen-Addad, C. Guzmán, R. McKenna, G. Andrew, A. Bie, D. Yu, A. Kurakin, M. Zadimoghaddam, S. Vassilvitskii, and A. Terzis. How to dp-fy your data: A practical guide to generating synthetic data with differential privacy, 2025. URL https://arxiv.org/abs/2512.03238. [40] A. Pouget, L. Beyer, E. Bugliarello, X. Wang, A. P. Steiner, X. Zhai, and I. Alabdulmohsin. No filter: Cultural and socioeconomic diversity in contrastive vision-language models. Advances in Neural Information Processing Systems, 37:106474–106496, 2024.

14

[41] P. Rajpurkar, J. Irvin, K. Zhu, B. Yang, H. Mehta, T. Duan, D. Ding, A. Bagul, C. Langlotz, K. Shpanskaya, M. P. Lungren, and A. Y. Ng. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning, 2017. URL https://arxiv.org/abs/1711.05225. [42] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015. doi: 10.1007/s11263-015-0816-y. [43] R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017. [44] S. Tayebi Arasteh, M. Lotfinia, T. Nolte, M.-J. Sähn, P. Isfort, C. Kuhl, S. Nebelung, G. Kaissis, and D. Truhn. Securing collaborative medical AI by using differential privacy: Domain transfer for classification of chest radiographs. Radiology: Artificial Intelligence, 6(1):e230212, 2024. doi: 10.1148/ryai.230212. [45] A. Thudi, E. Rovers, Y. Ruan, T. Thrush, and C. J. Maddison. Mixmin: Finding data mixtures via convex minimization, 2025. URL https://arxiv.org/abs/2502.10510. [46] F. Tramèr, G. Kamath, and N. Carlini. Considerations for differentially private learning with large-scale public pretraining. arXiv preprint arXiv:2212.06470, 2022. [47] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), page 3462–3471. IEEE, 2017. doi: 10.1109/cvpr.2017.369. URL http://dx.doi.org/10.1109/ CVPR.2017.369. [48] S. M. Xie, H. Pham, X. Dong, N. Du, H. Liu, Y. Lu, P. Liang, Q. V. Le, T. Ma, and A. W. Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305.10429, 2023. [49] A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao, G. Cormode, and I. Mironov. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv:2109.12298, 2021. [50] D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz, et al. Differentially private fine-tuning of language models. In International Conference on Learning Representations, 2022. [51] D. Yu, S. Gopi, J. Kulkarni, Z. Lin, S. Naik, T. L. Religa, J. Yin, and H. Zhang. Selective pre-training for private fine-tuning, 2024. URL https://arxiv.org/abs/2305.13865.

A

Appendix

A.1

Proof of Lemma 4.1

Proof. We first provide a specific data mixing problem (under the lemma assumptions) that requires private hyperparameter search to have ϵ = Ω(k) for fixed excess error with constant probability. We √ then show, under the lemma assumptions, the reduced MixMin objective always has ϵ = O( k). 15

To show worst-case scaling for private hyperparameter search, we need only find one "hard" example. Consider the case where the data mixture objective is ∥λ − λ∗ ∥22 for λ∗ in the relative interior of the simplex (i.e., λ∗i = ̸ 0 for any i). This corresponds to taking L to be MSE, fi (x) = ei for all x, and y = λ∗ always (which √ hence defines the target distribution). Then for random search to achieve α error, we need ∥λ − λ∗ ∥2 ≤ α. Note the probability density function for uniform sampling on the simplex is (k − 1)! (where k is the number of public datasets we are mixing). Thus the probability of a random √ π (k−1)/2 (k−1)/2 mixture being in Sα = {λ : ∥λ − λ∗ ∥2 ≤ α, λ ∈ ∆k } is (k − 1)! V ol(Sα ) = (k−1)! for α Γ((k−1)/2+1) α sufficiently small. This follows from noting that in a sufficiently small ball around any point in the interior of the simplex, the simplex is a hyperplane defined by ⟨1, λ⟩ = 1, and the intersection of a hyperplane around a k-dim sphere centered at a point on it is yet another sphere of dimension k − 1 and the same radius (by rotational symmetry). Now recall Poisson hyperparameter search for mixture finding first samples a number r following the Poisson distribution, and then trains on r random mixtures and picks the best. As α → 0, we have the probability that the closest of the r random mixtures is in Sα , where r is also a random variable,  (k−1)! π (k−1)/2 (k−1)/2 is Θ Er Γ((k−1)/2+1) α (see Appendix A.2.1 in Papernot and Steinke [37] for the first-order approximation 
which  becomes tight as α → 0). Hence for the probability to not vanish with k, we need (k−1)/2 Er = Ω α1 . Plugging this into Theorem 6 in Papernot and Steinke [37] shows the ϵ guarantee for (ϵ, δ) grows ∝ log(Er) = Ω(k). In contrast, the MixMin reduction presents a Lipschitz DP-SCO√objective. It is known that an k ln(1/δ)

achievable expected excess error vs. privacy trade-off is α = O( √1n + ) [3, 13, 2] 5 . We hence ϵn √ √ √ need ϵ = O( k/(nα)) = O( k · α) = O( k), noting n = Ω(1/α2 ) by assumption and α ≤ 1. Note that by Lipschitzness and the fact the simplex has ℓ2 diameter bounded by 1, the loss of the mixture is always bounded by a constant independent of k, and so always √ has at least constant probability around the expectation (independent of k), and we can conclude O( k) is sufficient for excess error less than α with constant probability.

A.2

Guidance on choosing the mixture-selection budget ρ

DP-MixMin sets aside a fraction ρ of the private data to estimate the mixture weights. In the main text we validate a choice of ρ a posteriori, by checking that the privately recovered mixture λDP has high cosine similarity to the non-private solution λ⋆ (Tables 11–13). Here we give a complementary a-priori guideline, derived directly from our error bound (Lemma 4.1), that tells a practitioner how large ρ must be before running anything. Setup. We measure quality by the excess error α of eq. (3), here instantiated with the empirically trained proxies fˆi in place of the Bayes-optimal fi . Write Tλ = ρN for the number of private examples used for mixing, with N the private set size and k the number of public sources. 5 In fact, in the specific case of generalized linear losses, this can be far better in dimension dependence [4]

16

√ An a-priori bound on ρ. The DP-SCO trade-off for this objective (Lemma 4.1) is α = O 1/ Tλ + p
k ln(1/δ)/(ϵ Tλ ) . Requiring each term to contribute at most α/2 yields a sufficient budget: ! p k ln(1/δ) 1 , (4) α=O √ + ϵ Tλ Tλ 1 α 4 Requiring each term ≤ α/2 : √ ≤ ⇒ Tλ ≥ 2 , 2 α Tλ p p k ln(1/δ) 2 k ln(1/δ) α ≤ ⇒ Tλ ≥ , ϵTλ 2 ϵα ! p Tλ 4 2 k ln(1/δ) , , ρ= . ∴ Tλ ≥ max 2 α ϵα N

(5) (6) (7)

This is a guideline rather than a single universal constant. The required budget is governed by two 2 regimes: a statistics-limited regime where Tλ ≳ 4/α p (the cost of estimating the objective from finite data), and a privacy-limited regime where T ≳ 2 k ln(1/δ)/(ϵα) (the cost of the DP noise). The two λ p α ⋆ ⋆ cross over at ϵ = 2 k ln(1/δ): for ϵ > ϵ the statistical term dominates and ρ is independent of the privacy budget; for ϵ < ϵ⋆ the required ρ grows as 1/ϵ. Reading the guideline at the budgets we use. For ϵ ≥ 1 we operate in the statistics-limited regime (e.g. √ for our vision setting k = 14, δ = 10−5 give ϵ⋆ ≈ 0.63 < 1), so the achievable error is of order α ≈ 1/ Tλ . Evaluated at ρ = 0.1, this single relation reproduces both of our empirical operating points: √ • Vision (per-example DP). ρ = 0.1 gives Tλ ≈ 8,970 examples, so α ≈ 1/ Tλ ≈ 0.01, consistent with the strong recovery we observe (Table 11: cosine similarity 0.97 at ϵ = 1 across the 14 NIH tasks). √ • Language (example-level DP). ρ = 0.1 gives ≈ 2.8M OpenWebMath tokens, so α ≈ 1/ Tλ ≈ 0.001, consistent with the near-exact recovery we observe (Table 15: cosine similarity 1.00 at ϵ = 1). The achievable α therefore tracks data availability: with millions of tokens, language reaches a far tighter α than vision, whose mixture-learning splits contain only thousands of examples. In both cases ρ = 0.1 clears the threshold of eq. (7), which is why it is a robust default; a practitioner facing a new dataset can invert the same relation to size ρ for a target α.

A.3

Extended Experimental Details

A.3.1

Image Classification

DP-MixMin Implementation. Proxy models were trained for 4 and 10 epochs respectively for CheXpert MIMIC-CXR at learning rate 10−4 and batch size 256 (the largest batch size that fit on a single L40 GPU). DP-MixMin used clipping norm 13.0, selected from {1.0, 2.0, 5.0, 10.0, 13.0, 15.0, 20.0} with a batch size of 1000 selected from {500, 1000, 1500}, and learning rate chosen from {0.5, 1.0, 2.0}. Training details. Pretraining and private fine-tuning on the ResNet18s both used batch size 200 (the largest batch that fit on an L40). Given mixture weights λ ∈ ∆14 , we pretrained the ResNet-18 by minimizing the binary cross-entropy loss between its scalar output and the mixture-weighted target 17

P14

c=1 λc yc , where yc was the per-disease binary label.

We evaluated the pretrained ResNet-18 at 3, 6, and 9 epochs and selected the checkpoint with the highest downstream validation AUC before fine-tuning. Fine-tuning ran for 36,000 steps, with learning rate 10−4 and DP-SGD clipping norm 2.0. All hyperparameters were tuned once on a single pilot configuration—Atelectasis classification as the NIH downstream task with CheXpert pretraining at ε = 1—and then held fixed across all 14 tasks, both pretraining sources, all seeds, and all three privacy budgets ε ∈ {1, 2, 5} at δ = 10−5 . A.3.2

6

Language Modeling

DP-MixMin Implementation. For Enron, DP-MixMin was run for 200 steps with per-sample clipping norm 0.01 and step size 5.0. For OpenWebMath, DP-MixMin was run for 200 steps with clipping norm 0.2 and step size 5.0. Training details. Pretraining uses GPT-2 Medium from random initialization for 7.1×109 tokens at context length 1024, effective batch size 64, and learning rate 5 · 10−3 . Enron fine-tuning uses 5 users per gradient step with 20 accumulation steps (effective batch of 100 users), G_ULS of 15, context length 512, and learning rate 5 · 10−4 (chosen as the largest stable rate from {5 · 10−5 , 5 · 10−4 , 10−3 , 5 · 10−3 }). Per-device batch sizes were set to the largest power of 2 that fit in available GPU memory, with gradient accumulation used to reach the target effective batch. Private Enron fine-tuning uses clipping norm 0.5 and a token budget of 25M. Private OpenWebMath fine-tuning used batch size 500 at learning rate 5 · 10−3 for pretraining and a 50M-token fine-tuning budget. All results are averaged over 3 seeds.

A.4

On the infeasibility of GSD in our setting

Gu et al. [16] propose Gradient Subspace Distance (GSD), which scores a candidate public source by training a model on randomly labeled examples from both the public source and the private task, then measuring the distance between their gradient subspaces. The lowest-distance source is selected. GSD is inapplicable to our setting in Section 5.1. We binarize each of CheXpert and MIMIC-CXR along their 14 disease axes (Section 5.1), so all 14 candidate sources within a corpus share the same image set X and differ only in per-image labels. With random labels, gradient subspaces depend only on X , so any two sources sharing inputs receive a GSD of zero, and the candidates DP-MixMin must rank are precisely such sources. GSD is, by construction, only informative when candidates have distinct input distributions, for example disjoint vision benchmarks of the kind on which Gu et al. [16] evaluate it. The coarsest GSD variant with signal in our setting selects between whole corpora: CheXpert versus MIMIC-CXR. Either choice is dominated by DP-MixMin, which improves over uniform pretraining on each corpus independently (Table 10). Even granting GSD an oracle picker over the two corpora, the resulting pipeline still falls short of DP-MixMin.

A.5

Robustness to Proxy Model Quality

Lemma 4.1 assumes proxy models near Bayes optimal. To check how much this matters in practice, we varied the number of epochs used to train the CheXpert proxy (1 to 6, versus 4 in our main experiments) and ran DP-MixMin for NIH Atelectasis at ϵ = 1. Results were similar across all proxy budgets (Table 3), and all remained well above uniform pretraining (0.707 AUC), suggesting small changes in proxy quality have little effect on DP-MixMin’s downstream performance. 6 Note that this is the importance weighting defined by the mixture, and it reduces variance relative to sampling y c according to λc .

18

Table 3: DP-MixMin is insensitive to proxy training budget. NIH Atelectasis AUC (mean ± std over 3 seeds) with CheXpert pretraining at ϵ = 1. Proxy epochs

Atelectasis AUC 0.744 ± 0.005 0.745 ± 0.005 0.742 ± 0.006 0.749 ± 0.003

1 2 4 (main experiments) 6

A.6

Empirically Validating Lemma 4.1

To show that Lemma 4.1 holds empirically under its assumptions, we analyze the excess error as a function of the number of public sources being mixed, k, and trace its trajectory as k increases. Setup. We train ResNet-50 proxies on k ImageNet-1k classes [42] (k ∈ {2, 4, 8, 16, 64, 128, 256, 512, 1000}, the k classes being the k sources) and run DP-MixMin on the first 10 binarized CIFAR-100 classes [27] at ε=1, δ=10−5 over all N =50,000 images. ∝ √ k reference

Excess error α (×10−3, log scale)

DP-MixMin (avg)

±1 s.d. 17.7

20

10.7 8.8

10 4.9

4.9 3.8

5

3.4

2.4

2

8

16

32

64

128

256

512

1000

k (# public sources, log scale)

√ Figure 4: Excess error scales roughly as k for large k. Average excess error of DP-MixMin over the 10 CIFAR-100 tasks as the number of public sources k increases (both axes log scale). √ Error bars and the shaded band show ±1 standard deviation across tasks. The √ dashed red line is a k reference; the fitted slope for k ≥ 32 is 0.49, consistent with the predicted k rate.

Metric. We report the excess error α = L(λDP ) − L(λ⋆ ) (eq. (3)) against the non-private optimum λ⋆ , averaged over 5 seeds and the first 10 binarized CIFAR100 tasks Result. Excess √ error grows sub-linearly in k (Table 4). A log-log fit over k ≥ 32 gives slope 0.49, matching the k prediction (0.5) and well below the linear cost of private hyperparameter search.

19

Table 4: DP-MixMin excess error α = L(λDP ) − L(λ⋆ ) across the number of public sources k (ImageNet-1k → CIFAR-100, ε=1, δ=10−5 ), for all 10 CIFAR-100 one-vs-rest tasks. Mean ± std over 5 seeds, in units of 10−3 . The last column (Avg) averages over all 5 seeds and 10 tasks. k

class 0

class 1

class 2

class 3

class 4

class 5

class 6

class 7

class 8

class 9

2 0.1±0.1 0.1±0.1 0.1±0.2 0.1±0.2 0.1±0.1 0.1±0.2 0.1±0.1 0.1±0.2 0.1±0.1 0.1±0.2 4 0.1±0.0 0.0±0.0 0.0±0.0 0.1±0.0 0.0±0.0 0.0±0.0 0.0±0.0 0.0±0.0 0.0±0.0 0.1±0.0 8 1.1±0.1 1.4±0.2 1.3±0.2 3.7±0.2 3.3±0.2 2.1±0.1 2.3±0.2 2.8±0.1 4.1±0.2 2.1±0.1 16 5.7±0.4 7.0±0.2 3.3±0.4 2.9±0.3 2.4±0.4 7.0±0.6 4.2±0.3 2.8±0.4 5.8±0.5 7.8±0.5 32 2.8±0.2 6.7±0.5 3.4±0.3 2.0±0.1 1.9±0.1 3.4±0.2 3.7±0.2 2.7±0.3 5.8±0.3 5.3±0.3 64 3.8±0.2 2.1±0.3 3.7±0.2 3.4±0.3 2.6±0.3 4.4±0.3 4.4±0.2 5.0±0.5 1.8±0.1 2.3±0.3 128 5.9±0.5 1.6±0.3 5.7±0.3 5.2±0.4 6.4±0.4 2.7±0.2 7.4±0.2 6.8±1.0 3.8±0.4 3.6±0.7 256 12.2±0.7 2.7±0.3 9.2±0.3 10.6±0.3 11.5±0.3 4.6±0.4 10.9±0.2 10.5±0.7 7.9±0.6 8.3±0.9 512 8.6±0.4 6.7±0.7 4.5±0.5 16.2±0.3 10.1±0.6 10.3±0.4 13.7±0.4 15.7±0.9 13.9±0.5 7.0±1.0 1000 18.6±1.0 14.9±0.9 16.3±0.7 23.4±0.3 14.9±0.7 9.6±0.4 21.9±0.3 22.3±0.5 20.6±0.5 14.8±1.3

A.7

Avg 0.1±0.1 0.0±0.0 2.4±1.0 4.9±2.0 3.8±1.6 3.4±1.1 4.9±1.9 8.8±3.0 10.7±3.9 17.7±4.3

Sequential vs. Parallel Composition

Instead of splitting Dpriv into disjoint Dλpriv and Dfpriv (parallel composition), one can use all of Dpriv t for both steps and split the privacy budget, giving (ϵ1 + ϵ2 , δ1 + δ2 )-DP by sequential composition [11]. We compared the two on NIH with CheXpert pretraining at total ϵ = 1: sequential used ϵ1 = 0.1 and ϵ2 = 0.9, and parallel used our main setting (ρ = 0.1, ϵ1 = ϵ2 = 1). Neither was consistently better, and both were close to non-private MixMin (Table 6). This is expected: DP noise in each step scales roughly with 1/(ϵ · n) (Appendix A.2), and with ϵ1 = ρ ϵ and ϵ2 = (1 − ρ) ϵ, sequential composition matches parallel composition’s ϵ · n in both steps, differing only in using somewhat more data per step. Table 5: Learned mixture is stable across proxy training budget. Cosine similarity of the DP-MixMin mixture (early-stop, ε=1) at proxy epochs {1, 2, 6} to the main E=4 mixture, per NIH class (mean ± std, 3 seeds). E=4 is the self-reference (1.000). Class Atelectasis Consolidation Infiltration Pneumothorax Edema Emphysema Fibrosis Effusion Pneumonia Pleural Thickening Cardiomegaly Nodule Mass Hernia

E1

E2

E6

0.868 ± 0.015 0.886 ± 0.004 0.895 ± 0.018 0.952 ± 0.019 0.978 ± 0.003 0.986 ± 0.006 0.993 ± 0.005 0.870 ± 0.020 0.975 ± 0.003 0.982 ± 0.011 0.904 ± 0.004 0.967 ± 0.010 0.934 ± 0.021 0.983 ± 0.001

0.986 ± 0.003 0.941 ± 0.035 0.873 ± 0.046 0.976 ± 0.003 0.965 ± 0.003 0.988 ± 0.002 0.989 ± 0.002 0.884 ± 0.006 0.984 ± 0.004 0.988 ± 0.001 0.975 ± 0.007 0.972 ± 0.012 0.982 ± 0.004 0.998 ± 0.000

0.972 ± 0.014 0.944 ± 0.012 0.848 ± 0.023 0.975 ± 0.003 0.887 ± 0.029 0.970 ± 0.009 0.916 ± 0.024 0.938 ± 0.006 0.889 ± 0.008 0.931 ± 0.013 0.726 ± 0.067 0.932 ± 0.031 0.974 ± 0.009 0.914 ± 0.007

20

Table 6: Sequential and parallel composition perform similarly. NIH AUC (mean ± std over 3 seeds), CheXpert pretraining, total ϵ = 1.

A.8

Class

Sequential

Parallel

Non-private

Atelectasis

0.740 ± 0.010

0.747 ± 0.005

0.746 ± 0.006

Auditing DP-MixMin’s Privacy Guarantee

Threat model. We consider example-level (ϵ, δ)-differential privacy, where neighboring private datasets differ by the addition or removal of one example. We partition the private dataset as Dpriv = Dmix ⊔ Dft , where Dmix is used to privately learn the mixture weights λ, and Dft is used for DP-SGD fine-tuning. Because the two mechanisms operate on disjoint subsets, parallel composition gives an end-to-end guarantee of (ϵ, δ)-DP. In our experiments, this is (ϵ, δ) = (1, 10−5 ). We assume a deliberately strong adversary who observes everything released by the pipeline, including all public data, the algorithm and its hyperparameters, the learned mixture weights λ, and the final model. The adversary additionally knows every private example except the target example whose membership it seeks to infer. Audit scope and methodology. We audit the component novel to DP-MixMin, Mmix : Dmix → λ. For an example in Dmix , all subsequent model outputs depend on that example only through λ and are therefore post-processing of λ. We consequently audit the released mixture weights directly. Following the empirical privacy-auditing framework and algorithms of Jagielski et al. [20], we repeatedly run the mechanism on neighboring datasets and estimate an empirical lower bound ϵLB . Each audit configuration is evaluated over 2000 independent runs. We use a mixture-finding split ratio of ρ = 0.1. Public sources are the 14 CheXpert classes (proxy predictions). We run DP-MixMin once per NIH target class, auditing 14 mixture-finding instances in total. We evaluate two types of canaries. The natural canary is a real, unmodified private training example whose membership the adversary tries to infer. This captures realistic leakage of a genuine data point. The max-clip canary is a strong construction. Its per-example gradient attains the clipping norm, the maximum influence any single example can have under clipping, in a fixed direction aligned with the mixture’s most-weighted sources, since the released mixture is most sensitive to perturbations of those sources. Results. At the claimed privacy level ϵ = 1, the natural-canary audit gives ϵLB = 0 for all 14 NIH tasks. The stronger max-clipped-gradient canary gives maxtask ϵLB = 0.131. Thus, all measured private lower bounds satisfy ϵLB ≤ 0.131 < 1. When privacy noise is removed, the same audit obtains values as high as ϵLB = 5.60, demonstrating that it can detect substantial leakage when present. Overall, we find no empirical evidence of a privacy violation. Full per-class results for both CheXpert and MIMIC-CXR pretraining are reported in Tables 16 and 17.

A.9

User-Level DP-Mixmin

User-level DP protects the contribution of an entire user rather than a single example: two neighboring datasets may differ in all examples contributed by one user [28]. For Enron, we therefore partition the FN private mixture-learning split by user, Dλpriv = u=1 Du , where Du contains the examples attributed 21

to user u. At each DP-MixMin step, we sample a batch Uj of B users, compute one gradient per user by averaging the loss over that user’s sampled examples, clip this per-user gradient, and add Gaussian noise before updating the mixture weights. This differs from example-level DP-MixMin, where clipping and noise are applied to individual examples. The parameter GULS is the user-level subsample size. For each sampled user u, we use ku = min(GULS , |Du |) examples, so users with many emails cannot dominate a single update, while users with fewer than GULS emails contribute all available examples. Because the gradient is averaged within each user before clipping, the sensitivity is controlled at the user level: the clipped object is the user’s aggregate contribution, not an individual email. Algorithm 2 User-Level DP-MixMin Require: Target privacy guarantees (ϵ, δ), step size η, number of steps n, loss function L (CE or FN MSE), private mixture-learning split Dλpriv ⊆ X × Y partitioned by user as Dλpriv = u=1 Du , proxy models {fˆi : i = 1, . . . , k}, user batch size B, user-level subsample size GULS , clipping norm C 1 1: Initialize λi ← k for all i = 1, . . . , k priv 2: Pre-compute {fˆi (x) : x ∈ Dλ , i = 1, . . . , k} 3: for j = 1 to n do 4: Sample a batch Uj of B users from {1, . . . , N } Pk 5: fˆλ (x) ← i=1 λi fˆi (x) 6: for u ∈ Uj do 7: ku ← min(GULS , |Du |) (j) (j) 8: Sample Du ⊆ Du with |Du | = ku P 1 9: ℓu ← ku (x,y)∈D(j) L(fˆλ (x), y) u 10: gu ← ∇λ ℓu 11: g̃u ← Clip(gu , C) {clip per-user gradients} 12: end forP 13: gsum ← u∈Uj g̃u 14: z ∼ N (0, σ 2 I) {noise calibrated to (ϵ, δ)} 15: ḡ ← B1 (gsum + z) −ηḡi 16: λi ← λi eP for all i = 1, . . . , k k 17: λ ← λ / i=1 λi 18: end for 19: Return λ

A.10

Random Search

Random Search is the standard hyperparameter optimization baseline across fields [6, 37, 45]: rather than searching the space of configurations in a structured way, one simply draws a fixed number of candidates independently at random, evaluates each on a held-out validation set, and returns the best performer. Despite its simplicity, Random Search is known to be competitive with, and often outperform, grid search and many model-based optimizers, particularly in moderate-to-high-dimensional spaces where only a few hyperparameters meaningfully affect performance [6]. Note that private hyperparameter search algorithms also do random search, but randomize the number of candidates to test [37]. We adapted this baseline to mixture selection as follows. We sampled m candidate mixture weights λ1 , . . . , λm independently from the probability simplex ∆k , ran mixture-weighted public pretraining 22

Table 7: Per-class final AUC on NIH ChestX-ray14 after transfer from CheXpert. Each entry is the mean ± standard deviation of AUC over 3–5 seeds at δ = 10−5 . Bold indicates the best method per (class, ϵ) cell. DP-MixMin achieves the best macro AUC across all three privacy budgets, with absolute gains of +0.029 to +0.037 over uniform pretraining. DP-MixMin ϵ=1

ϵ=2

Atelectasis Consolidation Infiltration Pneumothorax Edema Emphysema Fibrosis Effusion Pneumonia Pleural Thick. Cardiomegaly Nodule Mass Hernia

0.740±0.012 0.742±0.014 0.671±0.004 0.780±0.007 0.829±0.017 0.683±0.028 0.627±0.039 0.848±0.007 0.626±0.035 0.655±0.008 0.872±0.008 0.574±0.009 0.674±0.009 0.583±0.054

0.747±0.005 0.745±0.017 0.674±0.005 0.788±0.002 0.836±0.014 0.715±0.022 0.660±0.016 0.850±0.002 0.626±0.025 0.668±0.009 0.878±0.010 0.566±0.017 0.688±0.009 0.624±0.054

Macro AUC ∆ vs. Uniform

0.707 +0.037

0.719 +0.032

Class

Uniform ϵ=5

ϵ=1

ϵ=2

Random Search ϵ=5

ϵ=1

ϵ=2

No Pretraining ϵ=5

ϵ=1

ϵ=2

ϵ=5

0.747±0.007 0.707±0.013 0.724±0.009 0.736±0.008 0.724±0.009 0.733±0.011 0.739±0.010 0.679±0.006 0.696±0.008 0.715±0.006 0.748±0.014 0.745±0.015 0.749±0.014 0.752±0.013 0.734±0.008 0.740±0.011 0.742±0.010 0.710±0.006 0.716±0.010 0.725±0.010 0.678±0.002 0.668±0.004 0.670±0.003 0.672±0.003 0.664±0.004 0.670±0.003 0.675±0.003 0.663±0.004 0.672±0.003 0.680±0.002 0.798±0.004 0.697±0.008 0.728±0.008 0.753±0.009 0.674±0.005 0.701±0.009 0.726±0.007 0.650±0.009 0.684±0.005 0.723±0.005 0.845±0.013 0.829±0.006 0.835±0.007 0.838±0.008 0.844±0.008 0.850±0.008 0.857±0.009 0.804±0.012 0.816±0.010 0.829±0.010 0.725±0.016 0.632±0.015 0.652±0.010 0.688±0.016 0.608±0.008 0.637±0.006 0.664±0.009 0.566±0.037 0.592±0.027 0.631±0.013 0.668±0.021 0.609±0.022 0.641±0.029 0.658±0.024 0.591±0.019 0.618±0.013 0.632±0.012 0.578±0.038 0.612±0.023 0.653±0.013 0.852±0.006 0.819±0.006 0.830±0.009 0.833±0.007 0.823±0.009 0.835±0.003 0.839±0.003 0.759±0.004 0.784±0.004 0.807±0.002 0.650±0.022 0.649±0.023 0.660±0.032 0.661±0.030 0.648±0.025 0.652±0.039 0.663±0.045 0.613±0.021 0.634±0.019 0.642±0.019 0.676±0.027 0.639±0.027 0.652±0.022 0.670±0.017 0.604±0.009 0.629±0.009 0.643±0.009 0.582±0.014 0.593±0.010 0.610±0.013 0.883±0.010 0.710±0.009 0.733±0.006 0.748±0.010 0.762±0.004 0.792±0.002 0.820±0.004 0.669±0.004 0.679±0.010 0.691±0.010 0.584±0.015 0.550±0.009 0.564±0.007 0.582±0.004 0.546±0.022 0.564±0.018 0.583±0.017 0.563±0.017 0.578±0.013 0.584±0.015 0.700±0.012 0.631±0.010 0.648±0.009 0.666±0.005 0.619±0.009 0.634±0.006 0.652±0.007 0.550±0.023 0.571±0.018 0.593±0.007 0.655±0.044 0.499±0.043 0.531±0.042 0.551±0.048 0.498±0.036 0.523±0.027 0.538±0.045 0.517±0.022 0.551±0.050 0.561±0.047 0.729 +0.029

0.670 —

0.687 —

0.701 —

0.667 −0.003

0.684 −0.003

0.698 −0.003

0.636 −0.034

0.656 −0.031

0.675 −0.026

Table 8: Per-class final AUC on NIH ChestX-ray14 after transfer from MIMIC-CXR. Each entry is the mean ± standard deviation of AUC over 3–5 seeds at δ = 10−5 . Bold indicates the best method per (class, ϵ) cell. DP-MixMin achieves the best macro AUC across all three privacy budgets, with absolute gains of +0.024 to +0.033 over uniform pretraining. DP-MixMin

Uniform

ϵ=1

ϵ=2

ϵ=5

ϵ=1

Atelectasis Consolidation Infiltration Pneumothorax Edema Emphysema Fibrosis Effusion Pneumonia Pleural Thick. Cardiomegaly Nodule Mass Hernia

0.751±0.010 0.756±0.012 0.674±0.008 0.754±0.013 0.845±0.003 0.710±0.023 0.641±0.019 0.849±0.003 0.681±0.023 0.683±0.013 0.804±0.025 0.589±0.012 0.690±0.008 0.546±0.048

0.749±0.010 0.758±0.011 0.679±0.004 0.770±0.004 0.849±0.004 0.728±0.012 0.653±0.019 0.849±0.005 0.677±0.027 0.688±0.017 0.816±0.016 0.591±0.016 0.707±0.005 0.620±0.003

0.737±0.000 0.759±0.015 0.682±0.005 0.782±0.005 0.853±0.005 0.742±0.012 0.680±0.028 0.851±0.005 0.674±0.020 0.704±0.022 0.835±0.009 0.598±0.008 0.719±0.009 0.604±0.058

0.728±0.009 0.748±0.010 0.667±0.004 0.719±0.012 0.839±0.007 0.650±0.013 0.614±0.024 0.824±0.007 0.666±0.029 0.664±0.014 0.711±0.007 0.552±0.013 0.635±0.006 0.495±0.024

Macro AUC ∆ vs. Uniform

0.712 +0.033

0.724 +0.028

0.730 +0.024

0.679 —

Class

ϵ=2

Random Search ϵ=5

ϵ=1

ϵ=2

No Pretraining ϵ=5

ϵ=1

ϵ=2

ϵ=5

0.734±0.008 0.740±0.008 0.742±0.006 0.749±0.006 0.755±0.005 0.679±0.006 0.696±0.008 0.715±0.006 0.751±0.011 0.753±0.011 0.743±0.012 0.746±0.012 0.749±0.012 0.710±0.006 0.716±0.010 0.725±0.010 0.673±0.005 0.677±0.006 0.673±0.006 0.676±0.006 0.680±0.003 0.663±0.004 0.672±0.003 0.680±0.002 0.741±0.015 0.754±0.015 0.679±0.015 0.707±0.016 0.732±0.016 0.650±0.009 0.684±0.005 0.723±0.005 0.841±0.008 0.845±0.007 0.826±0.011 0.834±0.008 0.841±0.011 0.804±0.012 0.816±0.010 0.829±0.010 0.674±0.010 0.691±0.011 0.633±0.033 0.645±0.022 0.693±0.034 0.566±0.037 0.592±0.027 0.631±0.013 0.638±0.026 0.637±0.018 0.587±0.006 0.620±0.004 0.644±0.010 0.578±0.038 0.612±0.023 0.653±0.013 0.830±0.005 0.833±0.006 0.813±0.003 0.822±0.004 0.829±0.006 0.759±0.004 0.784±0.004 0.807±0.002 0.664±0.041 0.682±0.031 0.645±0.029 0.655±0.029 0.667±0.029 0.613±0.021 0.634±0.019 0.642±0.019 0.676±0.011 0.684±0.010 0.621±0.013 0.641±0.005 0.663±0.012 0.582±0.014 0.593±0.010 0.610±0.013 0.765±0.026 0.782±0.027 0.749±0.012 0.770±0.019 0.801±0.017 0.669±0.004 0.679±0.010 0.691±0.010 0.569±0.012 0.585±0.009 0.539±0.008 0.548±0.009 0.559±0.010 0.563±0.017 0.578±0.013 0.584±0.015 0.654±0.004 0.667±0.007 0.597±0.014 0.628±0.017 0.653±0.017 0.550±0.023 0.571±0.018 0.593±0.007 0.536±0.053 0.551±0.033 0.527±0.056 0.525±0.040 0.580±0.053 0.517±0.022 0.551±0.050 0.561±0.047 0.696 —

0.706 —

0.670 −0.010

0.683 −0.013

0.703 −0.003

0.636 −0.044

0.656 −0.041

0.675 −0.031

followed by downstream DP fine-tuning for each λi , and selected the best-performing candidate on the same held-out set Dλpriv that DP-MixMin used for its mixture-selection step. We controlled compute by choosing m so that the total cost of training and evaluating all m mixtures matched or exceeded the end-to-end compute of DP-MixMin, ensuring any gains over Random Search could not be attributed to a larger compute budget. The selection step itself was also implemented non-privately: our implementation of Random Search read Dλpriv directly when picking the winning candidate, paying no privacy cost for the selection step. Random Search was therefore a strong baseline along two axes: equal or greater compute, and no utility degradation from private selection.

A.11

B

Tables and Figures

Dataset Access and Asset Licenses

All restricted or credentialed datasets were accessed under their applicable data-use terms. For PhysioNet credentialed-access datasets, the authors completed the required credentialing process and signed the applicable Data Use Agreement before accessing the data. We do not redistribute restricted datasets. 23

22 21 Perplexity

20 19 18 17 16 15 14

²=1

²=2

²=5

█ DP-MIXMIN █ UNIFORM █ RANDOM SEARCH

Figure 5: DP-MixMin beats both uniform-mixture and random-search Common Pile pretraining from scratch on Enron email completion across all three privacy budgets, with the largest gain at the tightest budget. Test-set perplexity after DP fine-tuning on Enron (δ=10−5 ) at ϵ ∈ {1, 2, 5}. All three methods pretrain GPT-2 medium from random initialization on Common Pile and are then DP-finetuned on Enron under identical settings. DP-MixMin reduces test perplexity by roughly 16%/14%/14% over uniform and 18%/17%/16% over random-mixture pretraining at ϵ ∈ {1, 2, 5}. NIH Chest X-rays. Accessed under the CCO: public domain license which gives free-use. The dataset is available at https://www.kaggle.com/datasets/nih-chest-xrays/data CheXpert. Accessed under the Stanford CheXpert Dataset Research Use Agreement, which permits non-commercial research use. The dataset is available at https://stanfordmlgroup.github.io/ competitions/chexpert. MIMIC-CXR. Accessed under the PhysioNet Credentialed Health Data License 1.5.0. Access requires completion of a credentialing process and signing a Data Use Agreement through PhysioNet [21]. Enron Email Dataset. The Enron Corpus was released into the public domain following federal litigation and is freely available for research use [26]. The Common Pile. Released under the Open Data Commons Attribution License (ODC-By-1.0). Individual constituent sources carry their own licenses; we refer readers to the dataset documentation for a full provenance breakdown [24].

24

22 21 Perplexity

20 19 18 17 16 15 14

½ = 0:1

½ = 0:2

½ = 0:3

█ DP-MIXMIN █ UNIFORM

Figure 6: DP-MixMin consistently outperforms uniform Common Pile pretraining on Enron at ϵ=1 across all proxy-split fractions, with larger fractions yielding both lower perplexity and tighter cross-seed variance. Test-set perplexity after DP fine-tuning on Enron (δ=10−5 ) at ϵ=1, varying the proxy fraction ρ ∈ {0.1, 0.2, 0.3} used by MixMin to estimate domain weights. The uniform baseline is independent of ρ and is shown for reference at all three values. DP-MixMin reduces test perplexity by 10%/12%/16% over uniform at ρ ∈ {0.1, 0.2, 0.3}, with ρ=0.3 giving the most stable mixture (cross-seed σ drops from 2.72 → 0.45).

Table 9: NIH ChestX-ray14 class distribution and correspondence to CheXpert/MIMICCXR labels. The Match column indicates whether each NIH class has a direct (Yes), Partial, or no (None) equivalent in CheXpert/MIMIC-CXR. NIH Idx 0 1 2 3 4 5 6 7 8 9 10 11 12 13

Class Name

# Pos.

% Pos.

Match

CheXpert/MIMIC-CXR Label

Atelectasis Consolidation Infiltration Pneumothorax Edema Emphysema Fibrosis Effusion Pneumonia Pleural Thickening Cardiomegaly Nodule Mass Hernia

11,559 4,667 19,894 5,302 2,303 2,516 1,686 13,317 1,431 3,385 2,776 6,331 5,782 227

10.31% 4.16% 17.75% 4.73% 2.05% 2.24% 1.50% 11.88% 1.28% 3.02% 2.48% 5.65% 5.16% 0.20%

Yes Yes Partial Yes Yes None None Yes Yes Partial Yes Partial Partial None

Atelectasis Consolidation Lung Opacity Pneumothorax Edema — — Pleural Effusion Pneumonia Pleural Other Cardiomegaly Lung Lesion Lung Lesion —

25

Table 10: Average AUC gains of DP-MixMin over Uniform pretraining, grouped by class type. Group A: well-represented classes with a direct CheXpert match (Atelectasis, Cardiomegaly, Effusion, Pneumonia, Pneumothorax, Consolidation, Edema). Group B: rare or unmatched classes (Infiltration, Mass, Nodule, Emphysema, Fibrosis, Pleural Thickening, Hernia); see Table 9. Each cell is the within-group mean of per-class absolute (abs) and relative (rel) AUC gain of DP-MixMin over Uniform pretraining at δ = 10−5 . ϵ=1 abs

ϵ=2 rel

ϵ=5

abs

rel

abs

rel

CheXpert → NIH ChestX-ray14 Group A (n = 7) +0.040 +5.6% Group B (n = 7) +0.034 +6.0%

+0.030 +0.034

+4.0% +5.7%

+0.029 +0.028

+3.8% +4.7%

MIMIC-CXR → NIH ChestX-ray14 Group A (n = 7) +0.029 +4.0% Group B (n = 7) +0.037 +6.2%

+0.020 +0.035

+2.7% +5.8%

+0.015 +0.034

+1.8% +5.3%

Table 11: Weight reconstruction of DP-MixMin across 14 NIH ChestX-ray14 tasks. Mean ± std over n=70 paired comparisons (14 tasks × 5 seeds). We compare the private weights λDP to the non-private MixMin solution λ⋆ using cosine similarity cos⋆ and L1 distance. Uniform references are CheXpert: cosu =0.75, L1,u =0.68; MIMIC-CXR: cosu =0.80, L1,u =0.60. The DP-MixMin selection budget ρ = 0.1. Bold values improve over the corresponding uniform reference. ϵ = 0.2 src

n

cos⋆

ϵ = 0.5 L1

cos⋆

ϵ=1 L1

cos⋆

ϵ=2 L1

cos⋆

ϵ=5 L1

cos⋆

L1

CheXpert → NIH ChestX-ray14 CheXpert 70 0.70±.08 0.81±.14 0.93±.03 0.37±.09 0.97±.01 0.24±.06 0.98±.01 0.19±.05 0.99±.01 0.16±.04 MIMIC-CXR → NIH ChestX-ray14 MIMIC-CXR 70 0.58±.17 1.04±.30 0.82±.09 0.57±.20 0.93±.04 0.33±.10 0.97±.01 0.22±.05 0.98±.01 0.16±.03

Table 12: Weight reconstruction of DP-MixMin for NIH Hernia (0.20% prevalence in Dpriv , 227 total positives). Mean ± std over 5 paired comparisons (5 seeds). We compare the private weights λDP to the non-private MixMin solution λ⋆ using cosine similarity cos⋆ and L1 distance. Uniform references: CheXpert cosu =0.64, L1,u =0.96; MIMIC-CXR cosu =0.73, L1,u =0.82. Bold values improve over the corresponding uniform reference. ρ=0.05 n≈4,485, #pos≈9

ρ=0.1 n≈8,970, #pos≈18

ρ=0.2 n≈17,939, #pos≈36

Pretrain

ε

cos⋆

L1

cos⋆

L1

cos⋆

L1

CheXpert

0.2 0.5 1.0 2.0 5.0

0.79±0.01 0.96±0.01 0.99±0.00 0.99±0.00 1.00±0.00

0.64±0.02 0.27±0.03 0.16±0.03 0.11±0.02 0.09±0.02

0.94±0.01 0.99±0.00 0.99±0.00 1.00±0.00 1.00±0.00

0.32±0.01 0.15±0.01 0.10±0.01 0.08±0.01 0.08±0.02

0.98±0.00 0.99±0.00 1.00±0.00 1.00±0.00 1.00±0.00

0.17±0.01 0.10±0.01 0.08±0.01 0.08±0.01 0.08±0.01

MIMIC-CXR

0.2 0.5 1.0 2.0 5.0

0.62±0.00 0.83±0.00 0.94±0.00 0.98±0.00 0.99±0.00

1.08±0.00 0.57±0.00 0.31±0.00 0.18±0.00 0.10±0.00

0.79±0.00 0.94±0.00 0.98±0.00 0.99±0.00 1.00±0.00

0.67±0.00 0.30±0.00 0.17±0.00 0.11±0.00 0.07±0.00

0.92±0.00 0.98±0.00 0.99±0.00 1.00±0.00 1.00±0.00

0.35±0.00 0.17±0.00 0.10±0.00 0.08±0.00 0.06±0.00

26

Table 13: Weight reconstruction of DP-MixMin for NIH Fibrosis (1.50% prevalence in Dpriv , 1,686 total positives). Mean ± std over 5 paired comparisons (5 seeds). We compare the private weights λDP to the non-private MixMin solution λ⋆ using cosine similarity cos⋆ and L1 distance. Uniform references: CheXpert cosu =0.73, L1,u =0.77; MIMIC-CXR cosu =0.83, L1,u =0.60. Bold values improve over the corresponding uniform reference. ρ=0.05 n≈4,485, #pos≈57

ρ=0.1 n≈8,970, #pos≈114

ρ=0.2 n≈17,939, #pos≈229

Pretrain

ε

cos⋆

L1

cos⋆

L1

cos⋆

L1

CheXpert

0.2 0.5 1.0 2.0 5.0

0.78±0.02 0.95±0.01 0.98±0.01 0.99±0.00 0.99±0.00

0.70±0.03 0.33±0.04 0.22±0.03 0.18±0.03 0.17±0.02

0.94±0.01 0.98±0.00 0.99±0.00 0.99±0.00 0.99±0.00

0.38±0.02 0.20±0.02 0.16±0.01 0.16±0.01 0.15±0.01

0.98±0.00 0.99±0.00 0.99±0.00 0.99±0.00 0.99±0.00

0.23±0.01 0.16±0.01 0.16±0.01 0.15±0.01 0.15±0.01

MIMIC-CXR

0.2 0.5 1.0 2.0 5.0

0.59±0.00 0.82±0.00 0.93±0.00 0.97±0.00 0.99±0.00

1.10±0.01 0.59±0.01 0.34±0.01 0.21±0.01 0.15±0.01

0.78±0.00 0.94±0.00 0.98±0.00 0.99±0.00 0.99±0.00

0.69±0.01 0.32±0.01 0.21±0.00 0.16±0.00 0.14±0.01

0.92±0.00 0.98±0.00 0.99±0.00 0.99±0.00 1.00±0.00

0.37±0.01 0.19±0.00 0.15±0.01 0.13±0.01 0.10±0.01

Table 14: Enron weight recovery under user-level DP with K=16 Common Pile proxies. Mean ± std over 3 paired comparisons (3 seeds). We compare the private mixture weights λDP to the non-private MixMin solution λ⋆ using cosine similarity cos⋆ and L1 distance. Columns vary the DP-MixMin selection budget ρ, the fraction of the private mixture-learning split used to estimate the mixture weights; the second header row reports the corresponding number of Enron users. The uniform reference is cosu =0.40 and L1,u =1.47. Bold values improve over the uniform reference on that metric. ρ=0.05 (6 users)

ρ=0.1 (12 users)

ρ=0.15 (18 users)

ρ=0.2 (24 users)

ρ=0.3 (35 users)

ρ=0.5 (59 users)

ε

cos⋆

L1

cos⋆

L1

cos⋆

L1

cos⋆

L1

cos⋆

L1

cos⋆

L1

0.2 0.5 1 2 5 10

— 0.12±0.15 0.12±0.15 0.16±0.13 0.93±0.04 0.98±0.01

— 1.85±0.19 1.83±0.19 1.68±0.15 0.48±0.14 0.28±0.05

0.08±0.11 0.15±0.20 0.61±0.26 0.92±0.05 0.98±0.00 0.99±0.00

1.89±0.13 1.75±0.30 1.08±0.38 0.65±0.22 0.41±0.10 0.17±0.04

0.01±0.00 0.07±0.04 0.61±0.26 0.93±0.05 0.99±0.00 1.00±0.00

1.98±0.01 1.75±0.13 1.01±0.27 0.45±0.14 0.17±0.05 0.09±0.02

0.34±0.44 0.70±0.30 0.97±0.03 0.99±0.00 1.00±0.00 1.00±0.00

1.41±0.72 0.80±0.46 0.36±0.15 0.28±0.06 0.28±0.02 0.29±0.01

0.19±0.16 0.86±0.13 0.97±0.03 0.99±0.01 0.99±0.00 0.99±0.00

1.65±0.16 0.70±0.13 0.46±0.07 0.37±0.03 0.33±0.01 0.32±0.01

0.44±0.24 0.95±0.01 0.99±0.00 0.99±0.00 0.99±0.00 0.99±0.00

1.25±0.28 0.54±0.08 0.40±0.03 0.36±0.01 0.34±0.01 0.33±0.01

27

Table 15: OpenWebMath weight recovery under example-level DP with K=16 Common Pile proxies. Mean ± std over 3 paired comparisons (3 seeds). We compare the private mixture weights λDP to the non-private MixMin solution λ⋆ using cosine similarity cos⋆ and L1 distance. Columns vary the DP-MixMin selection budget ρ, the fraction of the private mixture-learning split used to estimate the mixture weights; the second header row reports the corresponding number of OpenWebMath tokens. The uniform reference is cosu =0.40 and L1,u =1.28. Bold values improve over the uniform reference on that metric. ρ=0.01 (275K tokens)

ρ=0.02 (552K tokens)

ρ=0.05 (1.4M tokens)

ρ=0.1 (2.8M tokens)

ε

cos⋆

L1

cos⋆

L1

cos⋆

L1

cos⋆

L1

0.2 0.4 0.6 0.8 1 2 5

0.17±0.08 0.32±0.14 0.49±0.15 0.64±0.12 0.73±0.09 0.90±0.01 0.94±0.01

1.66±0.14 1.39±0.20 1.19±0.20 1.02±0.15 0.92±0.12 0.74±0.04 0.67±0.01

0.34±0.17 0.68±0.13 0.82±0.07 0.88±0.04 0.91±0.02 0.94±0.00 0.94±0.01

1.38±0.23 0.99±0.16 0.84±0.10 0.77±0.06 0.74±0.04 0.68±0.02 0.67±0.02

0.78±0.10 0.92±0.02 0.94±0.01 0.94±0.01 0.94±0.00 0.95±0.00 0.95±0.00

0.88±0.14 0.72±0.04 0.69±0.02 0.68±0.01 0.67±0.00 0.67±0.01 0.67±0.01

0.92±0.02 0.94±0.00 0.95±0.00 0.95±0.00 1.00±0.00 1.00±0.00 1.00±0.00

0.72±0.04 0.67±0.01 0.67±0.01 0.67±0.01 0.05±0.03 0.04±0.01 0.03±0.01

Table 16: Empirical privacy audit of DP-MixMin’s mixture-learning step, CheXpert → NIH ChestX-ray14, at ε = 1. We audit the released mixture weights λ = Mmix (Dmix ), reporting the Clopper–Pearson empirical lower bound εLB at ε = 1, δ = 10−5 , ρ = 0.1, and k = 14 proxy sources, over 2000 runs on add/remove-neighboring datasets (attack fit on a SELECT half, bound computed on a disjoint EVAL half; 95% confidence). The natural canary is a real held-out example; the max-clip canary injects a worst-case gradient at the clipping norm. Private audits at ε = 1; No-noise control removes the DP noise. Every private εLB ≪ 1 while controls recover large leakage, confirming audit fidelity. Natural canary

Max-clipped-gradient canary

Class

% pos

Private (ε=1)

No-noise ctrl.

Private (ε=1)

No-noise ctrl.

Atelectasis Consolidation Infiltration Pneumothorax Edema Emphysema Fibrosis Effusion Pneumonia Pleural Thickening Cardiomegaly Nodule Mass Hernia

10.31 4.16 17.74 4.73 2.05 2.24 1.50 11.88 1.28 3.02 2.48 5.65 5.16 0.20

0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

0.261 0.501 0.088 0.068 0.404 0.619 1.188 0.000 0.344 1.226 0.148 0.539 0.000 2.360

0.080 0.052 0.000 0.080 0.000 0.054 0.091 0.028 0.098 0.131 0.000 0.000 0.000 0.086

3.71 5.19 2.27 5.18 4.93 4.93 5.18 3.73 5.60 5.60 4.93 5.60 5.60 5.60

max over tasks

—

0.000

2.360

0.131

5.60

28

Table 17: Empirical privacy audit of DP-MixMin’s mixture-learning step, MIMIC-CXR → NIH ChestX-ray14, at ε = 1. Identical protocol to Table 16: Clopper–Pearson εLB at ε = 1, δ = 10−5 , ρ = 0.1, and k = 14 MIMIC-CXR proxy sources, over 2000 neighboring-dataset runs (SELECT/EVAL split, 95% confidence). As for CheXpert, every private εLB ≪ 1 while controls recover substantial leakage. Natural canary

Max-clipped-gradient canary

Class

% pos

Private (ε=1)

No-noise ctrl.

Private (ε=1)

No-noise ctrl.

Atelectasis Consolidation Infiltration Pneumothorax Edema Emphysema Fibrosis Effusion Pneumonia Pleural Thickening Cardiomegaly Nodule Mass Hernia

10.31 4.16 17.74 4.73 2.05 2.24 1.50 11.88 1.28 3.02 2.48 5.65 5.16 0.20

0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

0.000 0.501 0.000 0.022 0.672 0.000 2.627 0.000 0.480 1.147 1.895 0.582 0.551 4.934

0.000 0.000 0.001 0.000 0.000 0.000 0.056 0.000 0.000 0.000 0.101 0.000 0.000 0.061

4.02 4.72 3.21 4.92 5.59 5.40 4.44 2.87 5.17 5.07 5.51 4.73 4.76 4.92

max over tasks

—

0.000

4.934

0.101

5.51

Table 18: DP-MixMin vs. its non-private version, CheXpert → NIH ChestX-ray14 at ε = 1. Per-class final AUC (mean ± std over 3–5 seeds, δ = 10−5 ). Classes are ordered by NIH index (Table 9). DP-MixMin matches the non-private upper bound to within 0.001 macro AUC. Class Atelectasis Consolidation Infiltration Pneumothorax Edema Emphysema Fibrosis Effusion Pneumonia Pleural Thickening Cardiomegaly Nodule Mass Hernia Macro AUC

Non-private MixMin

DP-MixMin (ε=1)

0.745 ± 0.007 0.753 ± 0.014 0.669 ± 0.005 0.796 ± 0.008 0.819 ± 0.012 0.715 ± 0.026 0.645 ± 0.024 0.850 ± 0.006 0.595 ± 0.015 0.688 ± 0.017 0.878 ± 0.007 0.585 ± 0.012 0.691 ± 0.004 0.545 ± 0.030

0.747 ± 0.006 0.756 ± 0.012 0.669 ± 0.006 0.789 ± 0.018 0.809 ± 0.019 0.746 ± 0.025 0.655 ± 0.019 0.851 ± 0.005 0.592 ± 0.019 0.671 ± 0.022 0.868 ± 0.013 0.570 ± 0.012 0.670 ± 0.011 0.567 ± 0.029

0.712

0.711

29

Table 19: DP-MixMin vs. its non-private version, MIMIC-CXR → NIH ChestX-ray14 at ε = 1. Per-class final AUC (mean ± std over seeds, δ = 10−5 ). Non-private MixMin finds the mixture with no privacy and fine-tunes on the full private set (both splits), so it is a strict upper bound; DP-MixMin reserves the mixing split (parallel composition) and privatizes both steps. DP-MixMin matches the non-private upper bound to within 0.006 macro AUC. Class Atelectasis Consolidation Infiltration Pneumothorax Edema Emphysema Fibrosis Effusion Pneumonia Pleural Thickening Cardiomegaly Nodule Mass Hernia Macro AUC

Non-private MixMin

DP-MixMin (ε=1)

0.758 ± 0.006 0.764 ± 0.015 0.678 ± 0.008 0.779 ± 0.004 0.838 ± 0.007 0.745 ± 0.018 0.654 ± 0.024 0.855 ± 0.005 0.656 ± 0.022 0.687 ± 0.023 0.850 ± 0.011 0.613 ± 0.007 0.700 ± 0.017 0.543 ± 0.064

0.749 ± 0.010 0.761 ± 0.013 0.676 ± 0.003 0.754 ± 0.014 0.835 ± 0.010 0.720 ± 0.028 0.672 ± 0.029 0.851 ± 0.004 0.658 ± 0.016 0.692 ± 0.014 0.832 ± 0.030 0.580 ± 0.007 0.675 ± 0.013 0.584 ± 0.069

0.723

0.717

30

Related documents

• Differentially Private Mixing of Public Datasets Improves Private Learning#1215731
• Optimizing Canaries for Privacy Auditing with Metagradient Descent#1061614
• Stable-Plastic Subspace Construction for Differentially Private Continual LoRA#1343103
• Privacy Auditing with Zero (0) Training Run#998229
• Deep Learning under Fractional-Order Differential Privacy#175148