arXiv CSopen access

From Multiplicity to Vulnerability: Privacy Amplification Risk from One-Dataset-Multiple-Model Exposure

From Multiplicity to Vulnerability: Privacy Amplification Risk from One-Dataset-Multiple-Model Exposure Qirui Huang1 , Na Li2 , Hongsheng Hu3 , Zhi Zhang1 , Anmin Fu2 , Yansong Gao1 1

arXiv:2607.05111v1 [cs.CR] 6 Jul 2026

2

The University of Western Australia, Australia Nanjing University of Science and Technology, China 3 The University of Newcastle, Australia

Abstract—To efficiently exploit a valuable data source (e.g., facial or medical images), it is frequently harnessed to fulfill multiple learning objectives (e.g., facial recognition, age estimation, and race classification). Each trained model is then deployed as an independent API service for corresponding inference. However, the privacy risk introduced by this one-dataset-multiple-model (ODMM) paradigm is completely overlooked by the community. For the first time, this work reveals that the ODMM setting substantially amplifies privacy leakage. We establish a theoretical framework that proves that privacy leakage accumulates as more ODMM models are exposed, a phenomenon we term ODMM privacy composition. Guided by this theoretical foundation, we propose PRIME (Privacy Amplification RIsk from One-Dataset-Multiple-Model Exposure) to systematically assess this risk and quantify the resulting leakage using membership inference attacks (MIAs). Under black-box access to ODMM models, we design an aggregation mechanism that collectively captures carefully identified privacy signals leaked by individual ODMM models, and construct an attack meta-classifier over the aggregated meta-information to infer the membership status of a given sample jointly. Our results provide strong evidence that dataset reuse across ODMM models strikingly jeopardizes privacy, which is consistently evident across five privacy-sensitive image and textual benchmark datasets and diverse model architectures (from ResNet and ViT to Qwen3-1.7B), spanning three domains: facial analysis, medical imaging, and textual attribution analysis. While mitigations such as differential privacy can reduce the effectiveness of PRIME with trade-offs, our attack still consistently outperforms single-task MIAs.

I. I NTRODUCTION The rapid advancement of deep learning (DL) has profoundly reshaped diverse domains such as autonomous driving [1], facial recognition [2], and medical diagnosis [3], while the recent emergence of Artificial Intelligence Generated Content (AIGC), ranging from text generation (e.g., ChatGPT [4]) to image synthesis (e.g., Midjourney [5]), has further propelled this transformative wave. For this success, high-quality data has been recognized as a critical factor for improving model performance in addition to scaled model size [6], [7]. However, this remarkable progress is fundamentally underpinned by the availability of massive task-specific datasets, which remain practically challenging to acquire due to prohibitive collection costs and inherent data scarcity. Fortunately, a given dataset typically harbors rich information suitable for diverse learning objectives [8], [9], [10]. Consequently, to overcome data scarcity and maximize valuable

resource utility, it has become standard practice across both academia and industry to train multiple individual models over the same dataset (e.g., CelebA), with each addressing a specific task (e.g., age estimation, race, gender, and facial expression recognition). For another instance, Facebook AI Research’s Detectron2 [11] releases multiple independent task-specific vision models in its model zoo, including object detection, instance segmentation, human keypoint estimation, and panoptic segmentation, all trained on the COCO dataset [12]. Overlooked Privacy Problem: Although repurposing the same data source for multiple tasks optimizes resource efficiency, the privacy risks posed by the collective exposure of one-dataset-multiple-model (ODMM) are essentially overlooked and poorly understood, despite the privacy vulnerabilities of individual DL model has been widely studied [13], [14], [15], [16], [17], [18], [19], [20]. MIAs serve as a critical tool for auditing such privacy leakage [21], [22], [23], [17], [24], [25]. By exploiting the DL model’s memorization of training data, MIAs distinguish subtle differences in model outputs between members and non-members, thus enabling attackers to infer whether a specific sample was included in the training set. For example, determining an individual’s membership in a facial recognition training dataset for specialized psychiatric cohorts could inadvertently disclose sensitive mental health information, thereby compromising personal privacy. Building on this, from another perspective, MIAs can be utilized to audit the risks of user privacy leakage and assess deployed model compliance with privacy regulations such as the General Data Protection Regulation (GDPR) [26]. Notably, each task-specific model learns distinct representations from the same underlying data distribution and is independently optimized toward its unique objective, which may lead to different memorization of the shared training samples, thereby raising potentially amplified privacy leakage. However, when multiple models upon one dataset, later referred to as ODMM models, are released and queried for inference, the privacy risks stemming from their divergent memorization behaviors remain overlooked. Hence, we pose the following research questions to highlight the urgent need for a comprehensive investigation into the completely overlooked privacy risks from the exposure of ODMM models.

Does the collective exposure of ODMM models derived exacerbate privacy leakage of their shared underlying dataset? If so, to what degree does it amplify privacy risks?

affirm that PRIME consistently and significantly outperforms all/maximum single-task/model in MIA performance, confirming that ODMM models trained on the same data source indeed substantially jeopardize privacy. For instance, PRIME attains 86.9% attack accuracy on UTKFace, surpassing the best single-task baseline by 18.4%, with consistent improvements on HAM10000 (+7.8%) and Blog Authorship Corpus (+9.8%). We also conduct ablation studies to analyze the effect of various factors, i.e., the number of exposed models on the attack performance. Finally, we provide further comparison and discussion on the privacy leakage of multi-task sharedbackbone models compared to independently trained ODMM models, as well as potential defense mechanisms. Our main contributions are: • We establish the theoretical foundation that privacy leakage accumulates across ODMM models, and the degree of amplification can be characterized by inter-task correlations. • We propose PRIME, the first framework for systematically assessing empirical evidence on the ODMM privacy composition through the lens of MIA, showing that such ODMM co-exposure indeed significantly jeopardizes privacy. • We conduct extensive experiments across classic image domains (including privacy-critical clinical AI) and the LLM field to demonstrate the effectiveness and validation of PRIME, which is also independent of MIA methods such as hard-labelonly [16] or recent RMIA attacks [31].

Our Work: This work, for the first time, unveils and proves that access to multi-task models trained on the same underlying dataset exacerbates privacy leakage. Firstly, we theoretically prove that privacy leakage accumulates as more ODMM models are exposed. Generally, our proof is inspired by the composition theorem in differential privacy (DP), which formally establishes that privacy loss accumulates across multiple queries, degrading the protection afforded to any individual [27], [28], [29], [30]. We apply this concept to this neglected new ODMM context and lay out a theoretical foundation as ODMM privacy composition. Guided by the theoretical foundation, through the lens of MIA as a privacy auditing tool, we propose and design PRIME as a framework for systematically and empirically assessing the degree of additional leakage across various factors and tasks. Aggregating accumulated information exposed by ODMM models amplifies the overall privacy risk, even in the focused black-box MIA. Below, we highlight the key findings of PRIME and briefly describe its core design. •Theoretical Foundation. We formalize ODMM privacy composition as a unified theoretical framework, demonstrating that collectively exposing ODMM models fundamentally amplifies privacy risk beyond the leakage from each individual model. First, we prove that the optimal membership inference advantage is monotonically non-decreasing as more models are exposed (Theorem 1), and that leakage accumulates additively across ODMM models with approximately independent permodel leakage signals (Theorem 2). However, tasks among ODMM models often have dependencies. Second, we theoretically characterize how such inter-task correlation affects the extent of privacy amplification, revealing that more heterogeneous or divergent (less correlated) tasks lead to more leakage than homogeneous ones (Theorem 3). These theorems are supported by corresponding experimental evidence. •PRIME Framework. The above theorem lays out the foundations of the PRIME framework for systematic and empirical evaluation of ODMM privacy composition. Overall, a black-box adversary leverages a given MIA feature extraction method by querying each ODMM model to obtain corresponding raw outputs (e.g., posteriors or predicted values), which are then constructively transformed into (unique) taskspecific feature vectors. Then, these vectors are strategically concatenated through a joint information fusion step to form MIA meta-data, which is utilized to train a binary metaclassifier for membership inference. We conduct comprehensive experiments on five image and textual benchmark datasets and five diverse model architectures (from ResNet to ViT and Qwen3-1.7B) across three domains, including facial analysis, medical imaging, and large language models (LLMs) instruction fine-tuning to comprehensively evaluate the effectiveness of PRIME. Results

II. BACKGROUND AND R ELATED W ORK MIAs have been widely used for evaluating the privacy risks of DL models by inferring whether a particular sample belongs to the target model’s training dataset [16], [18], [32], [15], [20], [33], [34]. Formally, given a target sample x and the victim model M, the inference algorithm A can be expressed as: A : x, M → {0, 1}.

(1)

If a target sample x was used to train M, A outputs 1 (i.e., member) and 0 otherwise (i.e., non-member). In practice, most MIAs operate under a black-box setting, where the adversary is restricted to observing only the posterior probabilities (i.e., confidence scores) of the victim model, and can be broadly categorized into training-based and metricbased methods, depending on whether a dedicated attack metaclassifier is employed. Training-based MIAs first train shadow models to mimic the victim model’s behavior, then use the outputs of these shadow models to construct labeled metadata, which is subsequently used to train a binary attack metaclassifier for membership inference. For example, Shokri et al. [17] use posteriors, i.e., prediction confidences, as the meta-data. Yuan et al. [35] integrate posteriors, sensitivity, and labels to train a transformer-based attack meta-classifier. In contrast, several studies [36], [22], [37], [31], [18], [38] introduced metric-based MIAs that leverage statistical measures calculated from the victim model, e.g., posterior [22], scaled confidence score [39], modified entropy [36], and likelihood ratio [18], to distinguish membership status without the need to train the attack meta-classifier. More recently, Du

2

et al. [38] utilize membership dependencies between instances to enhance attack performance. Moreover, some studies utilize MIA as a tool to investigate the privacy effect of specific systems or technologies, e.g., query-based systems [40], RAG datastore [41], [42], machine unlearning [43], model compression [44], speaker recognition systems [45], explainable machine learning methods [46], and visual encoders [47], [48]. Recently, MIAs have been extended to the foundation models, e.g., LLMs [49], [50], [51], [52], [53], diffusion models [54], [55], [56], vision-language models [57], and text-to-video models [58]. However, several state-of-the-art studies [59], [60] debate that MIAs on largescale generative foundation models typically perform at nearrandom levels. This is largely attributed to the substantially reduced overfitting resulting from massive training corpora combined with limited training iterations. To the best of our knowledge, no prior work has explored membership leakage in scenarios where ODMM models are independently trained on the same dataset without mutual interference. The most relevant work from Yan et al. [61] investigates MIA in the multi-task learning setting, where multiple tasks share an identical backbone and employ taskspecific heads, which substantially differs from our work.That is, our work involves multiple models trained on the same dataset, whereas Yan et al. [61] train only a single model on the dataset. In addition, one shortcoming of using the multitask model is that its utility for a specific task is notably inferior, as the underlying backbone is not optimal for such a task. From the privacy perspective, their investigation differs fundamentally from ours, where new exploration could be derived by correlating information from independently trained ODMMs on the same dataset, thus leaking substantially more privacy than that of a multi-task network (utility and privacy comparisons are in Section V-E).

analyze the amplification of privacy risks under the black-box MIA setting. Next, we formalize ODMM as a joint observation channel and prove the monotonicity of the optimal attack advantage in the number of exposed ODMM models. We then derive an additive accumulation of privacy leakage under conditional independence, and further analyze how inter-task correlations affect the extent of privacy risk amplification. A. Joint Observation Channel of ODMM We begin by modeling the outputs of ODMM models as a multi-channel information release (Definition 1), and formulate membership inference (MI) as a hypothesis testing over the joint information across channels (Definition 2). Definition 1 (Joint Multi-channel Observations): Let D be the shared training set and let z = (x, y) be a queried sample. Let f1 , . . . , fk be k models independently trained for different tasks on D. Each model induces a channel that maps the input x to an output. Under black-box access, the attacker observes the joint output vector across these channels:
O(z) = f1 (x), . . . , fk (x) . (2) Definition 2 (Membership Inference in ODMM): Given the joint output O(z) from Definition 1, the membership inference (MI) is a binary hypothesis test: H1 : z ∈ D

vs.

H0 : z ∈ / D. (k)

Let L(· | ·) denote the conditional distribution. Denote Pin =

(k) / D for the output L O(z) | z ∈ D and Pout = L O(z) | z ∈ distributions under H1 and H0 , respectively. The two definitions above establish that the MIA in the ODMM setting can be viewed as an empirical approximation to the Bayes-optimal test over the joint output O(z). Therefore, to formally characterize ODMM exacerbate privacy under composition, it suffices to show that access to more ODMM models strictly increases the attacker’s ability to distinguish (k) (k) Pin from Pout , surpassing what is achievable from any single model output.

III. T HEORETICAL F OUNDATION : ODMM P RIVACY C OMPOSITION We posit that the collective exposure of ODMM models fundamentally amplifies the privacy risk compared to the leakage arising from each model. We note that this claim conceptually aligns with the composition theorem in differential privacy (DP) [27], [28], [29], [30]. Specifically, under ε-DP, privacy loss ε measures the maximum change in output distributions induced by the inclusion or exclusion of a single record. The composition of k independent query-induced-release on the same underlying dataset, where each release Pk satisfies εi -DP, incurs a total privacy loss bounded by i=1 εi , implying a monotonic degradation of the privacy guarantee with each release. However, directly applying DP composition theorems to our setting is not straightforward, as the ODMM models are standard DL models that have nothing to do with the DP mechanism. Furthermore, we study empirical membership leakage under black-box access rather than the worst-case guarantees of formally private mechanisms. To fill this theoretical gap, we establish a novel theoretical framework for ODMM privacy composition to formally

B. Monotonicity of Optimal MI Advantage Here, we define the MI Advantage as a measure of dis(k) (k) tinguishability between Pin and Pout , and then establish its monotonicity with respect to the number of observed models in ODMM. Definition 3 (MI Advantage): For an attacker A observing the joint output of k models, the membership inference advantage is defined as: Adv(Ak ) = Pr[Ak (O1:k ) = 1 | z ∈ D] − Pr[Ak (O1:k ) = 1 | z ∈ / D]

(3) where O1:k = (f1 (x), . . . , fk (x)). The Bayes-optimal membership inference advantage is Adv(A∗k ) = supAk Adv(Ak ). The MI advantage measures the ability of an attacker to (k) (k) distinguish between Pin and Pout based on the observed joint outputs. It therefore quantifies the performance of the hypothesis test defined in Definition 2, with higher values indicating more membership leakage from the joint outputs.

3

Theorem 1 (Monotonicity of Optimal MI Advantage): For the Bayes-optimal attacker, introducing an additional model in the ODMM setting cannot decrease the optimal membership inference advantage: Adv(A∗k+1 ) ≥ Adv(A∗k ).

same dataset for different tasks inevitably exhibit correlated memorization patterns. This is fundamentally different from per-query DP analysis, where each query output is protected by independently and identically sampled noise, typically drawn from Gaussian or Laplace distributions. By recognizing this distinction, we further analyze how inter-task correlations modulate the degree of privacy amplification. Theorem 3 (Amplification Degree with Task Correlation): Let wi denote each model’s standalone leakage strength, and let ρij denote the pairwise correlation between models i and j’s leakage. The overall privacy amplification degree from k ODMM models is quantified by:
2 Pk i=1 wi . (6) α = Pk Pk i=1 j=1 wi wj ρij

(4)

Proof. See Appendix E. Theorem 1 holds without additional assumptions. It implies that evaluating ODMM models in isolation, as standard in prior MIA literature, inevitably underestimates the cumulative privacy exposure. C. Additive Leakage under Conditional Independence Theorem 1 only guarantees that the optimal MI advantage is non-decreasing as more ODMM models are exposed. Here, we further prove that, under the assumption of approximate independence among per-model leakage, the privacy leakage from each ODMM model accumulates additively. Assumption 1 (Conditional Independence of Leakage): Each model fi produces a per-sample leakage Si (z) derived from its black-box output. Conditional on the membership status M ∈ {0, 1}, these leakages are approximately independent:S1 (z), . . . , Sk (z) ⊥ ⊥ | M. Assumption 1 represents an idealized scenario in which each model’s leakage is driven solely by its own task-specific memorization, with no inter-task correlation. Theorem 2 (Additivity of Log-Likelihood Ratio): By the Neyman–Pearson lemma, the optimal attack for distinguishing (k) (k) Pin from Pout is given by a threshold test on the joint log-likelihood ratio Λk (z). Consequently, the optimal MI advantage Adv(A∗k ) is characterized by the distribution of Λk (z). Under Assumption 1, Λk (z) decomposes additively as: Λk (z) =

k X i=1

log

pi (Si (z) | z ∈ D) . pi (Si (z) | z ∈ / D)

It follows that 1 ≤ α ≤ k, with larger α indicating higher amplification effects. Specifically, α = k when all task-specific models are perfectly diverse (ρij = 0, i ̸= j), consistent with the scenario in Theorem 2; and 1 < α < k in the realistic setting, where each model contributes positively but with magnitude governed by the divergence of its memorization patterns relative to the existing task ensemble ( 0 < ρij < 1). Proof. See Appendix G. Theorem 3 bridges the gap between the idealized additivity in Theorem 2 and real-world ODMM deployments considered in our work, where task correlations are inherently non-zero. It identifies task divergence as the key factor governing privacy risk amplification, implying that combining heterogeneous (less correlated) tasks (lower ρij , higher α) leads to more leakage than homogeneous ones. Empirical evidence. This is supported by the experiments (Section V-B2), where task correlation ρij is operationalized as the cosine distance between two ODMM models’ mean 512-dim penultimate-layer avgpool representations, and the per-pair amplification is measured by ∆AUC(fanchor , fj ) = AUC(fanchor +fj )−AUC(fanchor ). On seven CelebA ResNet18 models with Male (gender classification) as the anchor, the most divergent partners (Nose, SHair) yield the largest amplifications (∆AUC +0.1584, +0.1432), while the least divergent (Hat) yields the smallest (+0.0165). These results confirm that more divergent tasks contribute more non-redundant leakage, substantially increasing the privacy amplification degree α.

(5)

where pi (· | ·) denotes the distribution of model’s leakage. Under Assumption 1, each additional model contributes an independent non-negative term to Λk , increasing the separability (k) (k) between Pin and Pout as k grows. As a result, the optimal MI advantage Adv(A∗k ) increases monotonically with k. Proof. See Appendix F. Theorem 2 reveals that ODMM privacy risk amplification arises from aggregating membership leakage information across tasks, similar to the accumulation of privacy loss in the DP composition theorem. Even when each ODMM model provides only a weak MIA signal, the joint log-likelihood ratio can act as a strong discriminator. Empirical evidence. This is supported by the experiments (Section V-F), where AUC increases monotonically as the number of exposed models grows from 3 to 6 (0.656 → 0.706 → 0.743 → 0.782). These results confirm that each additional model monotonically accumulates privacy leakage that composes into an additive membership risk.

E. Theoretical Foundations of PRIME Theorems 1–3 suggest that privacy leakage accumulates across ODMM models, and the extent of this amplification is governed by inter-task correlation. Motivated by these insights, we propose PRIME with two key designs: Task-Specific Feature Extraction (Section IV-B1): Since leakage strength depends on task-specific memorization and their correlations, we extract task-tailored features to preserve discriminative signals. This design captures heterogeneous leakage patterns across models, ensuring that non-redundant (i.e., weakly correlated) signals are retained. Joint Information Fusion (Section IV-B2): To exploit the compositional nature of leakage, we aggregate task-specific

D. Amplification Scaling with Task Correlation Theorem 2 establishes that privacy leakage accumulates additively under the idealized assumption of conditional independence. In practice, however, ODMM models trained on the

4

has three phases: 1 Task-Specific Feature Extraction, 2 Joint Information Fusion, and 3 Shadow-Assisted Inference. 1) Task-Specific Feature Extraction: Consider K models trained on the same private dataset Dprivate can be queried by the adversary. For a target sample (x, y), the attacker queries each model fk and obtains the raw outputs. To capture the subtle differences in how members and non-members behave across different OODM tasks (i.e., age regression, binary gender classification, multi-class race classification), we design specific feature extractors ψk (·). Based on model output observations, we extract the following multidimensional feature vectors vk for each task type: Regression Tasks (e.g., Age Estimation) For a target sample with ground-truth label y, a regression model freg returns a continuous predicted value ŷ. We construct a 3-dimensional feature vector focusing on prediction error:

Model A: Age Regression Wrinkles, Skin Texture, ... Model B: Gender Classification Jawline, Facial Hair, ... Input x

Vage

Vgender

Vjoint

Attack Model

MIA Performance

Model C: Race Detection Eye Shape, Nose Structure, ...

Vrace

Fig. 1: Overview of PRIME that builds on ODMM privacy composition.

features into a joint representation that approximates the cumulative evidence across models. This fusion enables the attacker to combine complementary leakage signals, effectively amplifying the overall membership inference power.

vreg = [ŷ,

IV. T HREAT M ODEL AND M ETHODOLOGY

LMSE (y, ŷ),

|y − ŷ|] ,

where LMSE is the mean squared error (MSE) loss. The absolute error term |y − ŷ| is explicitly included by our PRIME as it directly correlates with the model’s overfitting on outliers. Binary Classification Tasks (e.g., Gender Recognition) A binary classifier fbin outputs a logit z. We compute the probability p = σ(z) (Sigmoid) and extract a concatenated vector:

A. Threat Model Adversary Knowledge. We assume that the adversary can identify that ODMM models share the same data source through several realistic avenues. First, when a service provider releases multiple task-specific models in a public model zoo, the training dataset is typically disclosed with documentation. For example, Facebook AI Research’s Detectron2 [11] explicitly documents that its object detection, instance segmentation, and keypoint estimation models are all trained on the COCO [12]. Second, model cards on platforms such as Hugging Face routinely expose metadata, e.g., the training dataset. For instance, Hugging Face Inference API hosts both question answering [62] and question generation [63] models developed by different providers, which explicitly report SQuAD [64] as their training dataset. Following the widely adopted black-box MIAs setting [55], [45], [19], [65], [49], we assume that the adversary can query ODMM models and obtain only their posterior probability distributions. Following [19], [65], [31], the adversary is further assumed to possess a local shadow dataset Ds that has the same data distribution as the victim’s dataset but without overlapped samples. Adversary Capability. The adversary can neither access the internal parameters of ODMM models nor tamper with the underlying model before or after its deployment, such as data poisoning [66] or fault injection [67]. However, the adversary can leverage Ds to train a set of shadow models corresponding to each task-specific victim ODMM model. Adversary Goal. For a given target sample, the adversary aims to infer whether it is a member of a victim dataset used to train ODMM models.

vbin = [p,

LBCE (y, p),

|p − 0.5|] .

Here, |p − 0.5| represents the prediction confidence. Member samples often exhibit higher confidence. Multi-class Classification Tasks (e.g., Race Classification) For a classifier fmul with C classes, outputting a probability vector p = Softmax(z), the information is richer. We construct a (C + 3)-dimensional vector: vmul = [p,

LCE (y, p),

max(p),

H(p)] ,

where max(p) captures the confidence of the predicted class, P and H(p) = − pi log pi is the prediction entropy. Lower entropy typically signals that the model has “seen” the sample during training and is confident in its prediction. 2) Joint Information Fusion: After querying all K available ODMM models, the attacker possesses a set of feature vectors {v1 , v2 , ..., vK } explained aforesaid. The fusion module aggregates these signals to form a single representation. Our PRIME framework employs a Concatenation Fusion strategy, which preserves the raw inference signals from all tasks without loss of information. The joint feature vector Vjoint is defined as: Vjoint = v1 ||v2 ||...||vK . Take a later studied case to ease understanding. In our UTKFace setup involving Age (Regression), Gender (Binary), and Race (5 class), the resulting joint vector has a dimensionality of 3 + 3 + (5 + 3) = 14. This vector creates a highdimensional membership footprint of the sample x, where different dimensions absorb membership signals from different

B. The PRIME Framework Based on the ODMM privacy composition theory in Section III, we propose PRIME, a unified framework for systematically conducting joint MIA against multiple APIs, each corresponding to an ODMM model, as shown in 1. PRIME

5

“perspectives” (e.g., entropy from race classification vs. MSE from age regression). 3) Shadow Model Assisted Attack Inference: To determine the membership status of x based on Vjoint , PRIME employs a shadow model technique. Its ODMM scenario constructions are as follows. Task-Specific Shadow Training. For each target task Tk , where k ∈ {1, ..., K}, we independently train a set of N shadow models {Sk,1 , Sk,2 , ..., Sk,N } per task. In our experiments, we set N = 5. Crucially, to simulate the joint exposure scenario, we enforce data split consistency across tasks. The i-th shadow model of every task (e.g., SAge,i , SGender,i , SRace,i ) is trained on the exact same shadow training (i) subset Dshadow train and evaluated on the same non-member (i) subset Dshadow test . This alignment ensures that a sample x has a consistent ground-truth membership label m ∈ {0, 1} across all tasks within the same shadow index i. Joint Attack Dataset Construction. We construct the training dataset for the attack model by aggregating outputs from these aligned shadow models. For each shadow index i ∈ {1, ..., N }, we feed the corresponding samples into the shadow model group {S1,i , S2,i , ..., SK,i }. We extract the task-specific feature vectors {v1,i , v2,i , ..., vK,i } and fuse them to form the joint training samples:

V. E VALUATION We conduct comprehensive experiments to systematically evaluate the effectiveness of PRIME across diverse domains and model architectures. We first present our general experimental setup (Section V-A), then validate our approach on the facial scenario with an in-depth analysis that empirically confirms the ODMM privacy composition theory (Section V-B). We further extend our evaluation to medical imaging scenarios with heterogeneous architectures (Section V-C) and LLM instruction fine-tuning scenarios (Section V-D). We also present a comparison between shared-backbone multi-task learning and independent single-task models (i.e, ODMM) (Section V-E). Finally, we examine how the number of exposed task-specific models influences PRIME effectiveness (Section V-F), corresponding to Theorem 2. A. Experimental Setup Attack Pipeline. For each privacy-sensitive evaluation scenario (i.e., Facial, Medical, and LLM Fine-Tuning), we train N = 5 shadow models per task. Task-specific feature vectors are extracted based on output types: 3-dimensional vectors for regression and binary classification tasks (prediction, loss, confidence/error), and (C +3)-dimensional vectors for C-class classification tasks (class probabilities, loss, max probability, entropy). All other general settings follow Section IV-B. Evaluation Metrics. We use standard MIA evaluation metrics: Attack Accuracy (Acc), Precision (Prec), Recall, F1 Score, and Area Under the ROC Curve (AUC). We also report True Positive Rate at low False Positive Rates (TPR@FPR) where relevant, as it is critical for practical privacy auditing.

(i)

Vjoint = v1,i ||v2,i ||...||vK,i . The final attack dataset is the union of features collected from all N shadow groups:

B. Facial Scenario Analysis

N n o [ (i) (i) Dattack = (Vjoint (x), mx ) | x ∈ Dshadow ,

Facial analysis represents a canonical ODMM scenario, where the same face images are routinely repurposed for multiple prediction tasks such as age estimation, gender classification, and ethnicity recognition. We here evaluate PRIME indepth on three representative facial attribute datasets spanning different scales, task compositions, and model architectures. 1) Datasets, Tasks, and Model Configuration: Datasets and Tasks. UTKFace [68] comprises over 23,000 face images annotated with age, gender, and ethnicity attributes, representing a realistic ODMM scenario with heterogeneous task types. We configure three tasks: Age (regression), Gender (binary classification), and Race (5-class classification). CelebA [69] has over 200,000 celebrity face images with 40 binary attribute annotations. We select three binary classification tasks: Male (gender), Mouth Slightly Open (expression), and Big Nose (facial feature). FairFace [70] has 108,000 images with balanced demographic distributions. We employ three multi-class classification tasks: Race (7-class), Age (9-class), and Gender (binary). Model Architectures. For UTKFace and CelebA, we employ ResNet-18 [71] as the backbone architecture. For FairFace, we adopt Vision Transformer (ViT-Base) [72] with LoRA [73] fine-tuning (r = 8, α = 16), representing a typical parameterefficient fine-tuning (PEFT) paradigm. Note that within each

i=1

where mx = 1 if x is a member of the i-th split, and 0 otherwise. Attack Model Inference. A binary attack meta-classifier A (implemented as a Random Forest) is trained on Dattack to distinguish members from non-members based on the fused ODMM features. The Random Forest classifier is selected for its robustness in handling high-dimensional, heterogeneous feature spaces formed by the concatenation of regression losses, classification entropies, and confidence scores. During the attack phase, the attacker queries the victim’s API endpoints to obtain the joint feature vector Vvictim and computes the final membership score: Score(x) = A(Vvictim ) ∈ [0, 1], where the score represents the predicted probability of x being a member. A higher score indicates stronger evidence of membership. In practice, the continuous score is used for threshold-free evaluation via ROC analysis and AUC, which provide a more comprehensive assessment of attack effectiveness across all operating points.

6

TABLE I: Empirical evidence of Theorem 3 on CelebA, with Male as the anchor model. Each row reports the cosine distance between Male and a partner, and the resulting ∆AUC from joint MIA. Partners are sorted by distance.

dataset, all ODMM models share the same backbone architecture (i.e., ResNet-18 for UTKFace/CelebA and ViT for FairFace). The setting with heterogeneous backbones across ODMM models is explored in Section V-C. All models are trained for 50 epochs using Adam/AdamW optimizer with learning rate 10−3 (or 10−4 for ViT). Joint Features. The concatenated joint feature vectors yield 9d (CelebA: 3×3), 14d (UTKFace: 3+3+8), and 25d (FairFace: 2 + 12 + 10 + 1) representations, respectively. 2) Theorem 3 Evidence: Theorem 3 predicts that the privacy amplification degree α is governed by inter-task correlation: the more divergent (less correlated) two tasks are, the greater the joint leakage they yield. To empirically validate this prediction, we conduct a controlled pairwise study on CelebA with 7 representative binary classification tasks sharing an identical ResNet-18 backbone and training configuration: Male, Mouth Slightly Open (Mouth), Big Nose (Nose), Black Hair (BHair), Smiling (Smile), Straight Hair (SHair), and Wearing Hat (Hat). Fixing the architecture and dataset isolates inter-task divergence as the only varying factor. Quantifying Task Divergence. Since ρij in Theorem 3 reflects how similarly two models memorize the shared samples, we operationalize it through their internal representations. For each victim model fi , we extract the 512-dim feature vector from its penultimate avgpool layer for every training (member) sample, and compute the mean representation h̄i . The pairwise task divergence between fi and fj is quantified by the cosine distance: h̄i · h̄j Dist(fi , fj ) = 1 − . ∥h̄i ∥ · ∥h̄j ∥

∆AUC

Nose SHair BHair Mouth Smile Hat

0.0926 0.0799 0.0720 0.0670 0.0650 0.0606

+0.1584 +0.1432 +0.0993 +0.0526 +0.0613 +0.0165

TABLE II: MIA on facial scenario. Utility denotes each model’s utility, e.g., accuracy or MAE. Dataset

CelebA

Attack Type

Utility

MIA Acc.

MIA Prec.

MIA Recall

MIA F1

MIA AUC

Single-Male Single-Mouth Single-Big Nose PRIME Attack

98.1% 98.4% 98.2% –

0.512 0.530 0.591 0.661

0.512 0.525 0.587 0.642

0.519 0.627 0.608 0.728

0.516 0.572 0.598 0.682

0.521 0.544 0.649 0.732

Impr. over Best Single

UTKFace

Single-Age Single-Gender Single-Race PRIME Attack Impr. over Best Single

FairFace

Single-Race Single-Age Single-Gender PRIME Attack Impr. over Best Single

†

–

+11.8%

+9.4%

+19.7%

+14.0%

+12.8%

6.98† 98.5% 98.1% –

0.712 0.599 0.734 0.869

0.678 0.593 0.685 0.806

0.807 0.633 0.865 0.972

0.737 0.613 0.765 0.881

0.767 0.657 0.793 0.925

–

+18.4%

+17.7%

+12.4%

+15.2%

+16.6%

72.3% 68.5% 91.2% –

0.687 0.703 0.551 0.804

0.622 0.654 0.529 0.758

0.951 0.866 0.950 0.893

0.752 0.745 0.679 0.820

0.745 0.758 0.577 0.863

–

+14.4%

+15.9%

-6.1%

+9.0%

+13.9%

MAE for regression; lower is better. Improvement is computed relative to the best single-task baseline per metric.

and +0.1432), whereas the least divergent partner (Hat) yields
the smallest gain (+0.0165). Out of 62 = 15 ordering pairs constructed from these 6 partners, 14 are consistent with Eq. (9); only one exception between Mouth and Smile marginally deviates, where the two distances are closely matched (0.0670 vs. 0.0650). This is expected because there are variance factors, such as model accuracy and the partner’s single-task MIA vulnerability degree. Specifically, Smile (MIA Acc.=0.5326) itself is more vulnerable to MIA than Mouth (0.0.5264), as detailed in Table XIV in the Appendix. These observations provide direct empirical support for Theorem 3: more divergent task-specific models, by encoding the shared data through less correlated representations, contribute more non-redundant membership leakage when jointly exposed. Conversely, semantically similar tasks—e.g., Hat that captures coarse head-region cues highly correlated with gender-related appearance, or Mouth and Smile that share heavily overlapping mouth-region features—yield smaller marginal amplification, consistent with the larger ρij predicted by the theorem. 3) MIA Attack Performance: Table II details MIA results of PRIME and compares them with corresponding single-task attacks. Consistent Across Datasets and Architectures. The PRIME consistently outperforms all single-task baselines across three diverse datasets and two distinct model architectures. On CelebA, the PRIME achieves AUC of 0.732 (+12.8% over best single-task). On UTKFace with heterogeneous task types,

(7)

∆AUC(fanchor , fj ) = AUC(fanchor +fj ) − AUC(fanchor ), (8) where AUC(fanchor +fj ) is the PRIME attack AUC obtained by fusing the two models (Section IV-B), and AUC(fanchor ) is the single-task MIA AUC of the anchor. Validation Hypothesis. If Theorem 3 holds, then ideally for any anchor fanchor and any two candidate partners fj , fk : Dist(fanchor , fj ) > Dist(fanchor , fk ) →

Dist(Male, Partner)

Distance rank: Nose > SHair > BHair > Mouth > Smile > Hat ∆AUC rank: Nose > SHair > BHair > Smile > Mouth > Hat Pairwise ordering match: 14/15

A larger Dist(fi , fj ) indicates that fi and fj encode the same data through more divergent representations, corresponding to a smaller ρij in Theorem 3. Quantifying Privacy Amplification. For each pair (fanchor , fj ), we measure the marginal MIA gain that fj contributes when fused with the anchor fanchor :

∆AUC(fanchor , fj ) > ∆AUC(fanchor , fk ).

Partner

(9)

That is, fusing the anchor with a more divergent partner expects a larger amplification. Experimental Result. Table I reports the result with Male as the anchor model. The six candidate partners are listed in descending order of their cosine distance to Male, alongside the resulting ∆AUC from joint MIA. The two rankings are nearly perfectly aligned: the most divergent partners (Nose and SHair) jointly contribute the largest amplifications (+0.1584

7

UTKFace (ResNet-18)

(a) Age Only

FairFace (ViT+LoRA) Density

CelebA (ResNet-18)

0.75 0.50

Male (0.521) Mouth Slightl.. (0.544) Big Nose (0.649) Joint Attack (0.732)

0.25 0.00 0.0

0.2

0.4

0.6

0.8

False Positive Rate

1.0 0.0

Age (0.835) Gender (0.629) Race (0.780) Joint Attack (0.941)

0.2

0.4

0.6

0.8

False Positive Rate

1.0 0.0

Race (0.745) Age (0.758) Gender (0.577) Joint Attack (0.863)

0.2

0.4

0.6

0.8

False Positive Rate

=0.37

0.2

0.4

0.6

0.8

(c) Race Only

Fig. 2: ROC curves comparing single-task attacks versus PRIME attack across three facial attribute datasets.

=0.38

5.0

0.0

0.2

0.4

=0.63 0.6

Sep: 0.54

0.8

Attack Confidence Score

1.0

(d) Joint (PRIME)

Sep: 1.34

Sep: 2.69

=0.26

=0.73

=0.88

2.5 0.0 0.0

0.2

0.4

0.6

0.8

1.0

Attack Confidence Score Non-member

the amplification is most pronounced with AUC reaching 0.925 (+16.6% over the best). FairFace with ViT+LoRA fine-tuning has similar amplification patterns (AUC: 0.758 → 0.863, +13.9%), indicating that the PRIME generalizes beyond CNN architectures to modern transformer-based models. Task/Feature divergence Enhances MIA. UTKFace exhibits the strongest privacy amplification, which we attribute to its heterogeneous task types (regression + binary + multi-class classification). The regression task captures continuous prediction errors, while multi-class tasks provide rich probability distributions, and these complementary signals jointly reveal membership information invisible to any single task. In contrast, CelebA’s three homogeneous binary classification tasks yield comparatively lower yet still substantial amplification. This is consistent with Theorem 3, where heterogeneous task combinations (e.g., Age+Race) yield a more divergent distance than homogeneous ones. Weak Signals Contribute Meaningfully. Even tasks with near-random individual attack performance contribute substantially to joint attacks. On CelebA, the Male classifier shows weak MIA performance (AUC=0.521), yet the PRIME achieves 28.1% improvement. On FairFace, the Gender task (AUC=0.577) contributes only 9.4% feature importance but remains essential for the attack. This further validates ODMM privacy composition: different tasks capture non-redundant privacy signals that accumulate when fused. Improved Member-Nonmember Balance. A notable finding on FairFace is that single-task attacks exhibit severely imbalanced recognition rates (e.g., Gender: 95% member recognition vs. 15% non-member recognition), indicating overfitting to membership prediction. The PRIME dramatically improves non-member recognition (+92.2% relative improvement: 37.2% → 71.5%) while maintaining strong member recognition. Figure 2 further visualizes ROC across all datasets. The PRIME curves consistently lie far above all single-task curves at any FPR. In the operationally critical low-FPR region (FPR < 0.1), the PRIME achieves substantially higher TPR, enabling reliable inference with minimal false alarms. 4) In-Depth Analysis on UTKFace: To gain deeper insight into how multi-task fusion amplifies MIA, we conduct detailed analyses on UTKFace. Confidence Score Distribution. Figure 3 shows the distribution of attack meta-classifier confidence scores for members and non-members across single-task attacks and the PRIME. The three single-task attacks exhibit heterogeneous and lim-

=0.44

1.0

Attack Confidence Score

1.0

(b) Gender Only

Sep: 1.14

=0.72

5 0 0.0

Density

True Positive Rate

1.00

Member

0.0

0.2

0.4

0.6

0.8

Attack Confidence Score

Non-member

1.0

Member

ROC Curves: Single-Task vs. Joint Attack

1.00 0.75 0.25

0.00 0.0

FPR=0.1

0.50 FPR=0.01

True Positive Rate

Fig. 3: Distribution of attack meta-classifier confidence scores for members (red) and non-members (blue) on UTKFace. Single-task attacks (a–c) exhibit substantial overlap between the two distributions.

0.2

0.4

Age Only (AUC = 0.767) Gender Only (AUC = 0.657) Race Only (AUC = 0.793) Joint (PRIME) (AUC = 0.925) Random Guess 0.6 0.8 1.0

False Positive Rate

Fig. 4: ROC curves (UTKFace). PRIME (AUC=0.925) consistently outperforms all single-task baselines, achieving 16.6% relative improvement over the best single-task attack (Race, AUC=0.793).

ited discriminative power: Age achieves moderate separation with member mean µmem = 0.72 and non-member mean µnon = 0.37, but with heavy overlap in the 0.3–0.6 range. Gender shows the weakest separation (µmem = 0.63, µnon = 0.44), with nearly indistinguishable distributions consistent with its lowest individual AUC (0.657). Race achieves the best single-task discrimination (µmem = 0.73, µnon = 0.38), yet still exhibits substantial distribution overlap. The PRIME (Figure 3d) produces a qualitative shift in discriminability. The non-member distribution concentrates below 0.3 (µnon = 0.26), while the member distribution shifts above 0.8 (µmem = 0.88), yielding near-disjoint distributions with minimal overlap. This stands in stark contrast to the single-task cases. The resulting AUC improvement from 0.793 (best single-task) to 0.925 (PRIME), as quantified by the ROC analysis in the following, confirms that this visual separation translates directly into stronger attack performance. ROC Curve Analysis. Figure 4 compares detailed ROC curves on UTKFace. •Heterogeneous task-specific vulnerability: AUC values range from 0.657 (Gender) to 0.793 (Race), with Age at 0.767. This 20% spread reflects varying memorization patterns induced by different objectives. •Consistent PRIME superiority: The PRIME curve lies far above all single-task curves at every FPR level, confirming that multi-task fusion provides uniformly stronger attacks

8

TABLE III: Feature importance (%) by task in PRIME attacks. Dataset

Task

Importance (%)

CelebA/UTKFace/ FairFace

Big Nose/Race/Age Mouth Open/Age/Race Male/Gender/Gender

47.6/38.2/49.2 27.6/35.4/41.3 24.9/26.4/9.4

leakage between training and evaluation sets, we perform lesion-level splitting: all images from the same lesion are assigned to either the member or non-member set, ensuring no overlap at the lesion level. This rigorous splitting strategy makes our privacy evaluation more realistic but challenging. Configuration. All models are trained for 50 epochs with Adam optimizer (learning rate 10−3 , batch size 48). We extract task-specific features: 10d for diagnosis (7-class probabilities + loss + max probability + entropy), 3d for age (prediction + loss + error), and 16d for localization (13-class probabilities + loss + max probability + entropy), yielding a 29-dimensional PRIME joint feature vector.

rather than task-specific improvements. PRIME AUC=0.925 represents a 16.6% relative improvement (0.132 absolute) over the best single-task baseline. •High TPR@low-FPR gains: PRIME demonstrates substantial advantages in the low FPR regime critical for practical attacks. At FPR=0.01, PRIME achieves TPR=0.105 compared to TPR=0.051 for the best single-task attack (Race), representing a 2.07× improvement (+107% relative gain). At FPR=0.1, the advantage remains pronounced: PRIME reaches TPR=0.728 versus only TPR=0.315 for best single-task attack, a 2.32× improvement. 5) Task Contribution Analysis: Table III shows the contribution of each task to joint attack effectiveness across all three datasets. Notably, all tasks contribute non-trivially, with the most informative task varying across datasets. This distribution confirms that privacy signals are distributed across tasks rather than concentrated in a single model, consistent with the fact that each ODMM model contributes additivity-like leakage (Theorem 2).

TABLE IV: MIA performance (HAM10000). Utility denotes ODMM model utility. Attack

Architecture

Utility

MIA Acc.

MIA F1

MIA AUC

Diagnosis Age Localization

ResNet-18 VGG-16 DenseNet-121

71.8% 13.38† 29.1%

0.657 0.516 0.748

0.694 0.520 0.771

0.738 0.521 0.829

PRIME

All

–

0.806

0.821

0.886

Impr. over Best Single

–

–

+7.8%

+6.5%

+6.9%

†

MAE (Mean Absolute Error) for regression task; lower is better.

Result and Analysis. Table IV presents and compares the MIA performance. The PRIME attack achieves AUC of 0.886, representing a 6.9% relative improvement over the best singletask attack (0.829). •Architecture Heterogeneity. Unlike our facial analysis experiments, where tasks within the same dataset shared the same model architecture, HAM10000 employs three distinct CNN architectures (ResNet-18, VGG-16, DenseNet-121). Despite this architectural heterogeneity, the PRIME attack still achieves substantial privacy amplification. This finding suggests that ODMM privacy composition is primarily dependent on the data and task rather than the model architecture. •Weak Signals Remain Valuable. The age regression task exhibits near-random single-task attack performance (AUC = 0.521), suggesting minimal individual privacy leakage. However, feature importance analysis reveals that age-derived features still contribute 6.8% to the joint attack’s predictive power. More importantly, the diagnosis (41.3%) and localization (52.0%) tasks provide strong complementary signals that collectively amplify the attack. The localization task, which captures where on the body a lesion appears, proves particularly informative for membership inference, likely because spatial patterns are highly patient-specific. •Implications for Clinical AI Deployment. These results carry significant implications for healthcare AI systems. Many clinical workflows involve training multiple models on the same patient cohort for different diagnostic or prognostic purposes. Our findings demonstrate that even if individual models appear privacy-safe (i.e., Age with a close to 0.5 AUC) when audited in isolation, their collective deployment creates amplified privacy risks.

Takeaway 1: Compelling evidence confirms that ODMM model exposure substantially amplifies privacy risks. Regardless of dataset, its scale (23K to 200K images), model architecture (ResNet vs. ViT), training paradigm (full fine-tuning vs. LoRA), or task type composition (homogeneous vs. heterogeneous), PRIME consistently achieves 12.8% to 16.6% higher AUC compared to the strongest individual model baseline. C. Medical Imaging Scenario with Heterogeneous Model Architecture Medical imaging is particularly privacy-sensitive, as patient data often carries highly confidential health information. Dataset and Tasks. HAM10000 [74] is a large-scale dermatoscopic dataset containing 10,015 images of pigmented skin lesions. Each image is annotated with multiple clinical attributes, which can be repurposed for different tasks. We configure three clinically relevant tasks with heterogeneous model architectures. 1 Diagnosis (dx) is a 7-class classification (akiec, bcc, bkl, df, mel, nv, vasc) using ResNet-18. 2 Age regression predicts patient age using VGG-16. 3 Localization is a 13-class classification of lesion body location using DenseNet-121. This setup reflects realistic clinical scenarios where the same dermatoscopic images are analyzed for multiple diagnostic purposes, and different model architectures may be deployed based on task-specific requirements. Lesion-Based Data Splitting. A unique characteristic of HAM10000 is that multiple images may correspond to the same skin lesion (identified by lesion_id). To prevent data

D. LLM Instruction Fine-Tuning Scenario The LLMs as foundational models have led to widespread adoption of fine-tuning paradigms, where pre-trained models

9

TABLE V: MIA performance (Qwen3-1.7B Fine-tuning). Attack Configuration

Accuracy

F1

AUC

Single-Task Attacks Gender Horoscope Age

0.618 0.787 0.664

0.602 0.779 0.651

0.663 0.832 0.730

PRIME Attacks Gender + Horoscope (2-way) Gender + Age (2-way) Horoscope + Age (2-way) All three tasks (3-way)

0.835 0.764 0.849 0.864

0.831 0.758 0.844 0.860

0.859 0.792 0.873 0.888

Impr. over Best Single

+9.8%

+10.4%

+6.7%

•Weak Signal Remain Valuable. The Gender task exhibits the weakest individual attack performance (AUC = 0.663), yet its inclusion in fusion attacks provides meaningful improvements. Gender + Horoscope (AUC = 0.859) outperforms Horoscope alone (AUC = 0.832), demonstrating that even tasks with weak signals contribute when combined. •LLM PEFT Remain Vulnerable. Despite using PEFT (i.e., LoRA), which adds only a small fraction of model parameters, the privacy amplification effect remains substantial. This suggests that memorization occurs at the representation level rather than being tied to specific parameter updates, and privacy-preserving techniques must account for multi-task exposure scenarios.

are adapted to domain-specific tasks using proprietary datasets. We demonstrate that PRIME extends beyond discriminative vision models, including previously considered ViT, to generative LLMs, revealing that fine-tuning on the same corpus creates substantial privacy amplification risks. Dataset and Tasks. We use the Blog Authorship Corpus [75], containing blog posts annotated with author demographics. We configure three attribute prediction tasks representing diverse prediction types within the generative context. Gender is a binary classification (male/female). Horoscope is a 12 class classification (zodiac signs). Age regression forecasts author’s age. Each task fine-tunes the LLM to predict the corresponding attribute based on blog content, simulating realistic scenarios where generative text is analyzed for multiple profiling purposes. Prompt Template and Generation. To formulate each prediction task as a generative objective, we design task-specific prompt templates that instruct the LLM to generate the target attribute as free form text. Table XV in the Appendix illustrates the prompt template and example responses for each task. During fine-tuning, the model is trained to generate the response tokens autoregressively, with the prompt tokens masked from the loss computation. Model and Fine-Tuning Configuration. We employ Qwen31.7B [76] as the base model with LoRA [73] fine-tuning (r = 16, α = 32, dropout=0.05) targeting attention and MLP layers. Each task is fine-tuned independently for 10 epochs with a learning rate of 2 × 10−4 . We use 10,000 total samples, split equally between target model data and shadow model pool, with 5 shadow models per task. Feature Extraction. For LLM-based MIA, we extract 4dimensional features per task: loss, perplexity, confidence (average maximum token probability), and entropy (average token-level entropy). PRIME concatenates features from multiple tasks, yielding 8d (2-task) or 12d (3-task) feature vectors. Attack Meta-Classifier. We evaluate three meta-classifier classifiers: Logistic Regression, Random Forest (RF), and Multi-Layer Perceptron (MLP). Results are reported using the best-performing classifier for each configuration. Result and Analysis. Table V presents and compares MIA performance. The 3-way PRIME attack achieves AUC of 0.888, representing a 6.7% relative improvement over the best single-task attack (0.742). This demonstrates that the ODMM privacy composition holds for autoregressive language models beyond discriminative classifiers and regression.

Takeaway 2: In addition to Takeaway 1, first, the ODMM privacy composition is universal: PRIME achieves 6.7% to 16.6% AUC improvement regardless of data modality (image vs. text), model family (CNN vs. Transformer vs. LLM), or training paradigm (full fine-tuning vs. Qwen3-1.7B with LoRA). Second, weak MIA tasks contribute meaningfully: across all scenarios, tasks with near-random individual attack performance still provide non-redundant membership signals that amplify joint inference, validating the additive leakage accumulation in Theorem 2. Third, Heterogeneity/divergence enhances amplification: UTKFace (heterogeneous tasks) and HAM10000 (heterogeneous architectures) show the strongest gains, consistent with Theorem 3.

E. Multi-Tasks vs. Single-Task-Models A natural question arises: Can multi-task learning (MTL) with a shared backbone mitigate the privacy amplification risk identified in this work? Prior work by Yan et al. [61] investigates MIA in MTL settings where multiple tasks share an identical backbone with task-specific heads. This architectural choice differs fundamentally from our setting, where each task is served by an independently trained model. Experimental Setup. We compare two model architectures on UTKFace. 1 Multi-Tasks Model: A single ResNet-18 backbone shared across all three tasks (Age, Gender, Race), with task-specific linear heads attached to the shared feature extractor. This is what Yan et al. [61] investigate. 2 SingleTask Models: Three independently trained ResNet-18 models, each dedicated to a single task with full parameter independence. This is what PRIME investigates. Both configurations are trained for 50 epochs using the Adam optimizer with a learning rate 10−3 . For the multitasks model, we employ a joint training objective that sums the losses across all tasks. We evaluate both task performance and MIA vulnerability under identical data splits. Utility Comparison. Table VI presents the task-specific performance comparison between multi-tasks and single-task architectures. The results reveal a substantial utility gap favoring single-task models:

10

TABLE VI: Task utility comparison (UTKFace). Task

Metric

Multi-Tasks

Single-Task

∆

Age Gender Race

MAE ↓ Acc ↑ Acc ↑

7.30 80.89% 62.49%

7.22 87.51% 75.40%

-0.08 -6.62% -12.91%

Independent Optimization Trajectories: Without shared parameters, single-task models explore different regions of the loss landscape, resulting in complementary (rather than redundant) memorization patterns. Divergent Learning Features: Because a multi-tasks model shares the same backbone, the backbone is forced to learn representations that are simultaneously useful across all tasks. This constraint limits the learning of task-specific features and is therefore suboptimal from a utility perspective. In contrast, a single-task model can fully specialize in task-specific representations, achieving higher utility but learning features that are divergent across tasks. When such features are collectively exposed through ODMM models, they effectively expand the memorable feature space, thereby amplifying privacy leakage.

TABLE VII: MIA comparison. Attack Type

Metric

Multi-Tasks

Single-Task

Single

Age AUC Gender AUC Race AUC

0.778 0.494 0.529

0.789 0.688 0.844

PRIME

Accuracy F1 Score AUC Member Recall

0.730 0.741 0.799 0.773

0.921 0.925 0.960 0.979

Key Finding: There is a trade-off. A multi-task model with a shared backbone is less vulnerable to privacy protection through representation coupling, but at the cost of significant utility (e.g., accuracy) degradation. Singletask models achieve superior utility but create amplified privacy risks when jointly exposed.

Age Regression: Comparable performance (MAE difference of only 0.08 years), suggesting that age-related features are relatively task-agnostic. Gender Classification: Single-task models achieve 6.62% higher accuracy, indicating that gender-specific features benefit from dedicated optimization. Race Classification: The largest gap of 12.91% demonstrates significant negative transfer, the shared backbone learns compromised representations when forced to simultaneously optimize for unaligned task objectives. This performance degradation in multi-tasks models is consistent with the well-documented negative transfer phenomenon in multi-task learning [77], [78], where gradient conflicts between tasks lead to suboptimal solutions for individual objectives. To gain high utility, a single task/head model must always be chosen in practice. Privacy Vulnerability Comparison. Table VII presents the MIA performance, revealing: Single-task Models are More Vulnerable: The PRIME attack against single-task models achieves an AUC of 0.960, compared to 0.799 for multi-tasks models, a striking 20.1% absolute improvement in attack effectiveness. Task-Specific Vulnerability Amplification: Single-task attacks against single-task models consistently outperform those against multi-tasks models. The Race task shows the most dramatic difference (AUC: 0.844 vs. 0.529), corresponding to the task where single-task models achieve the largest performance advantage. Privacy Composition is Stronger in Independent Models: The PRIME over the best single-task attack is more pronounced for single-task models (0.960 vs. 0.844 = +13.7%) compared to multi-tasks models (0.799 vs. 0.778 = +2.7%). Why Single-Task Models are More Vulnerable? There are three potential factors: Task-Specific Overfitting: Each single-task model can fully overfit to its specific task objective without interference, leading to stronger memorization of training samples from the task’s perspective.

F. Number of Exposed Models We conduct this study on CelebA using ResNet-18 models trained for 50 epochs. We progressively expand the set of ODMM tasks from 3 to 7, selecting attributes that represent diverse facial characteristics: 3 Models of Male, Mouth Slightly Open, Big Nose, 4 Models by + Black Hair, 5 Models by + Smiling, 6 Models by + Straight Hair. Each configuration uses the same victim/shadow data splits to ensure fair comparison. The joint feature dimension scales linearly: 3 × k dimensions for k models.

PRIME AUC

0.80

0.782 0.743

0.75 0.70 0.65

0.706 0.656

3

4

5

Number of Exposed Models

6

Fig. 5: MIA AUC vs number of ODMM models. Figure 5 presents the PRIME performance as the number of exposed models increases. The PRIME AUC increases monotonically from 0.656 (3 models) to 0.782 (6 models), representing a 12.6% improvement. The incremental AUC gain per model shows diminishing returns: +0.050 (3→4), +0.037 (4→5), +0.039 (5→6). This confirms that each additional task-specific model contributes unique membership signals, confirming Theorem 2. VI. D ISCUSSION We discuss three important aspects that adhere to our study: (1) evaluating two potential defenses; (2) using the

11

TABLE IX: MIA performance under DP protection.

SOTA RMIA [31] method to show that PRIME consistently has superior performance over single-task, regardless of its underlying MIA method.

Configuration No DP Moderate DP (σ=0.2) Strong DP (σ=0.5)

Age AUC

Gender AUC

Race AUC

Avg. Single

PRIME (Joint)

0.740 0.533 0.514

0.567 0.524 0.512

0.717 0.526 0.509

0.674 0.528 0.511

0.879 0.550 0.519

A. Differential Privacy as a Defense TABLE X: Privacy-utility trade-off for DP defense.

We evaluate whether DP-SGD [79], the standard approach for training DL models with DP [27], can effectively mitigate PRIME attack. We conduct DP experiments on UTKFace using a compact CNN architecture compatible with Opacus [80] using the DP-SGD framework. The detailed experimental setup is in Appendix C.

Configuration Moderate DP (σ=0.2) Strong DP (σ=0.5)

σ

ε

Age MAE↓

Gender Acc↑

Race Acc↑

No DP Moderate DP Strong DP

– 0.2 0.5

∞ 635.2 22.7

7.47 8.72 10.20

89.16% 86.71% 86.03%

76.58% 71.98% 67.00%

AUCno-DP − AUCDP × 100%. AUCno-DP − 0.5

Protection Rate

Race Acc Loss

0.329 0.360

86.7% 95.1%

4.60% 9.58%

TABLE XI: MIA performance under hard-label on UTKFace.

Key Result. Comprehensive result interpretations are in Appendix C, while we provide key results herein. Table VIII presents the task performance under different DP configurations. As expected, adding noise during training degrades model utility across all tasks. Table IX shows the MIA performance under DP protection. The results demonstrate that DP-SGD provides substantial defense against both single-task and PRIME joint attacks. To quantify the defense effectiveness, we define the Protection Rate as the fraction of excess attack AUC (above the random baseline of 0.5) that is eliminated: Protection Rate =

AUC Drop

0.550 0.519

we discretize the scalar output into 10-year bins to calculate consistency metrics. More details are in Appendix D. Results. We evaluate on UTKFace. Table XI summarizes the results, including the original victim model utility.

TABLE VIII: Model utility under DP (UTKFace). Configuration

PRIME AUC

Attack Configuration

Utility

MIA Acc

MIA F1

MIA AUC

Single-Age (Hard) Single-Gender (Hard) Single-Race (Hard)

5.93 (MAE) 91.18% 79.70%

0.675 0.558 0.596

0.733 0.661 0.708

0.694 0.565 0.602

Impr. over Best Single

–

+10.1%

+5.5%

+11.7%

The results show that ODMM privacy risks persist even without access to confidence scores. While single-task hardlabel attacks are relatively weak (average AUC 0.620), the joint PRIME attack achieves an AUC of 0.775, representing a 24.9% improvement. Feature importance analysis reveals that Age (46.3%) and Race (35.7%) tasks contribute the most significant robustness signals, while Gender (18.0%) contributes less.

(10)

Key Finding: The privacy amplification effect of PRIME is not specific to the MIA method. It always outperforms single-task attacks across both traditional shadow-modelbased MIA and the SOTA RMIA regardless of accessible information (e.g., hard-label only).

Table X summarizes the privacy-utility trade-off, where we use Race classification accuracy loss as a representative utility metric. Key Finding: DP is an effective defense against PRIME but incurs a notable utility trade-off. Even under DP protection, PRIME consistently outperforms single-task MIAs, confirming the persistence of ODMM privacy composition.

C. Extend RMIA to PRIME Our experiments have employed the classical MIA paradigm, where a binary attack model is trained (using shadow models) to distinguish members from non-members. A natural question arises: Does the privacy amplification effect of PRIME generalize to SOTA MIA methods? To address this, we evaluate PRIME using RMIA [31], an SOTA attack that achieves superior performance, particularly in the low false-positive rate (FPR) regime critical for practical privacy auditing. 1) RMIA Methodology: RMIA [31] employs pairwise likelihood ratio testing instead of training attack classifiers. For a target sample x and target model θ, RMIA computes:   Pr(x|θ)/ Pr(x) ScoreRMIA (x) = Pr ≥γ , (11) z∼π Pr(z|θ)/ Pr(z)

B. Limiting Information: Hard-Label As another defense, model providers can restrict API outputs to hard labels (e.g., class predictions or scalar values) rather than full probability distributions [16]. We now evaluate PRIME that operates solely on hard labels. Methodology. We employ a perturbation-based approach [16], [81] to extract membership signals. The intuition is that member samples are generally more robust to input noise than non-members. We generate N = 30 perturbed copies of a target image by adding Gaussian noise and query the ODMM models. For each task, we extract a 4-dimensional feature vector based on the model’s discrete outputs: (1) prediction correctness, (2) consistency ratio (stability of prediction under noise), (3) correctness ratio under noise, and (4) prediction entropy across perturbations. For the regression task (Age),

where Pr(x|θ) is the prediction score (softmax) of the model’s output for the true-label class, Pr(x) is the baseline prediction score estimated from reference models, and γ is a threshold

12

TABLE XIII: Comparison of PRIME effectiveness across MIA paradigms on CelebA (RMIA method).

parameter. The ScoreRMIA represents the fraction of population samples z that x dominates in terms of likelihood ratio. Following [31], we employ the offline mode with linear correction:   1 Pr(x) ≈ (1 + a) · Pr (x) + (1 − a) , (12) OUT 2

MIA Method

where PrOUT (x) is the average confidence across OUT reference models and a is a correction parameter. 2) Joint RMIA for ODMM: To extend RMIA to PRIME and fit the ODMM setting, we joint likelihood ratios across task-specific models using geometric mean: !1/K K Y Pr(x|θk ) . (13) Ratiojoint (x) = Pr(x) k=1

Accuracy

Precision

Recall

F1

TPR@FPR 1%

0.1%

Traditional

Male Mouth Open Big Nose Average

0.521 0.544 0.649 0.571

0.517 0.542 0.617 0.544

0.511 0.525 0.574 0.541

0.750 0.905 0.904 0.585

0.608 0.664 0.702 0.562

0.010 0.012 0.020 0.014

0.001 0.001 0.002 0.001

RMIA

Male Mouth Open Big Nose Average

0.533 0.579 0.670 0.594

0.517 0.541 0.604 0.554

0.547 0.587 0.590 0.575

0.193 0.276 0.683 0.384

0.286 0.376 0.633 0.432

0.024 0.047 0.060 0.044

0.005 0.015 0.018 0.013

F1

TPR@FPR 1%

0.1%

Traditional

Avg. Single-Task PRIME (Joint) Improvement

0.571 0.732 +16.1%

0.544 0.670 +12.6%

0.562 0.721 +15.9%

0.014 0.047 +3.3%

0.001 0.008 +0.7%

RMIA

Avg. Single-Task PRIME (Joint) Improvement

0.594 0.722 +12.7%

0.554 0.626 +9.1%

0.432 0.727 +29.5%

0.044 0.087 +4.3%

0.013 0.017 +0.4%

For the first time, this work theoretically and empirically reveals a striking form of privacy leakage rooted in ODMM privacy composition, a practice increasingly adopted to maximize the value of high-quality data. Grounded in our established theoretical foundation, we propose PRIME, a principled framework for evaluating privacy risks in the ODMM paradigm. Comprehensive experiments across diverse privacy-sensitive datasets, multiple model architectures, and a wide range of application scenarios consistently validate our root-cause hypothesis of ODMM privacy composition. We hope this work raises awareness of this potent and previously unrecognized risk, and encourages the community to exercise greater caution when leveraging valuable data across multiple learning objectives.

TABLE XII: Single-task MIA performance on CelebA. AUC

Accuracy

VII. C ONCLUSION

where σ(·) is the sigmoid function. 4) Results: Table XII presents complete single-task results. RMIA achieves a higher AUC than traditional MIA across all tasks (+4.0% average), with a particularly pronounced advantage in the low-FPR regime: at TPR@1%FPR, RMIA reaches 4.4% on average versus only 1.4% for traditional MIA. The results yield several important observations:

Task

AUC

•PRIME Extends Effectively to RMIA. The PRIME under the RMIA paradigm achieves an AUC of 0.722, substantially outperforming the average single-task RMIA (AUC=0.594) by 12.7%. It also attains 4.3% higher TPR @ 1% FPR compared to single-task RMIA, demonstrating PRIME’s privacy amplification effect in the critical TPR@Low FPR regime. •RMIA Strengthens Low-FPR Detection. In the low-FPR regime, single-task RMIA detects 4.4% of members at 1% FPR (versus only 1.4% for traditional single-task MIA), and PRIME further amplifies this to 8.7%. Notably, although PRIME’s overall AUC under RMIA (0.722) is comparable to that under the traditional paradigm (0.732), its TPR@1%FPR (8.7%) nearly doubles that of the traditional joint attack (4.7%), consistent with RMIA’s design objective of maximizing detection at low false-positive rates rather than overall AUC. •Consistent Privacy Amplification Across Paradigms. The AUC performance gain from PRIME remains substantial under both traditional MIA (+16.1%) and RMIA (+12.7%) paradigms, reinforcing evidence ODMM privacy composition.

This design preserves the multiplicative nature of likelihood ratios while normalizing across tasks with different ratio magnitudes. We then compute the final pairwise joint score by comparing the joint ratio against population samples. Moreover, we employ binary search on pre-sorted population ratios for efficient computation, reducing per-sample complexity from O(|Z|) to O(log |Z|). 3) Experimental Configuration: Dataset and Models. We use CelebA with three binary classification tasks (Male, Mouth Slightly Open, Big Nose). All models are ResNet18 trained for 50 epochs with Adam optimizer (learning rate 10−3 , batch size 64), using the same data splits as Section V-B with N =5 shadow models per task. Parameters. Following [31], we set the correction parameter a = 0.5, the threshold γ = 2.0, and the population size |Z| ≈ 40, 000. Confidence Computation for Binary Classification. For a sample (x, y) with y ∈ {0, 1}: ( σ(fθ (x)), if y = 1 Pr(x|θ) = (14) 1 − σ(fθ (x)), if y = 0

Method

Attack Type

Table XIII presents a comparison of attack performance between single-task and ODMM (via PRIME), under both traditional MIA and RMIA paradigms on CelebA.

13

R EFERENCES

[20] Jing Shang, Jian Wang, Kailun Wang, Jiqiang Liu, Nan Jiang, Md. Armanuzzaman, and Ziming Zhao. Defending against membership inference attacks on iteratively pruned deep neural networks. In 32nd Annual Network and Distributed System Security Symposium, NDSS 2025, San Diego, California, USA, February 24-28, 2025. The Internet Society, 2025. [21] Zhiqi Wang, Chengyu Zhang, Yuetian Chen, Nathalie Baracaldo, Swanand Ravindra Kadhe, and Lei Yu. Membership inference attacks as privacy tools: Reliability, disparity and ensemble. In Proceedings of ACM Conference on Computer and Communications Security (CCS), Taipei, Taiwan, October 13-17, 2025, pages 1724–1738. ACM, 2025. [22] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pages 268–282. IEEE, 2018. [23] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 343–362, 2020. [24] Hao Li, Zheng Li, Siyuan Wu, Yutong Ye, Min Zhang, Dengguo Feng, and Yang Zhang. Enhanced label-only membership inference attacks with fewer queries. In 34th USENIX Security Symposium, USENIX Security 2025, Seattle, WA, USA, August 13-15, 2025, pages 5465–5483. USENIX Association, 2025. [25] Zitao Chen and Karthik Pattabiraman. A method to facilitate membership inference attacks in deep learning models. In 32nd Annual Network and Distributed System Security Symposium, NDSS 2025, San Diego, California, USA, February 24-28, 2025. The Internet Society, 2025. [26] https://gdpr-info.eu/. [27] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference, pages 265–284. Springer, 2006. [28] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014. [29] Srivatsava Ranjit Ganta, Shiva Prasad Kasiviswanathan, and Adam Smith. Composition attacks and auxiliary information in data privacy. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 265–273, 2008. [30] Irit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In Proceedings of the Twenty-second ACM SIGMOD-SIGACTSIGART Symposium on Principles of Database Systems, pages 202–210, 2003. [31] Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024. [32] Zihao Wang, Rui Zhu, Zhikun Zhang, Haixu Tang, and XiaoFeng Wang. Rigging the foundation: Manipulating pre-training for advanced membership inference attacks. In IEEE Symposium on Security and Privacy, S&P 2025, San Francisco, CA, USA, May 12-15, 2025, pages 2509–2526. IEEE, 2025. [33] Bo Hui, Yuchen Yang, Haolin Yuan, Philippe Burlina, Neil Zhenqiang Gong, and Yinzhi Cao. Practical blind membership inference attack via differential comparisons. In 28th Annual Network and Distributed System Security Symposium, NDSS 2021, virtually, February 21-25, 2021, 2021. [34] Kaiyuan Zhang, Siyuan Cheng, Hanxi Guo, Yuetian Chen, Zian Su, Shengwei An, Yuntao Du, Charles Fleming, Ashish Kundu, Xiangyu Zhang, and Ninghui Li. SOFT: selective data obfuscation for protecting LLM fine-tuning against membership inference attacks. In 34th USENIX Security Symposium, USENIX Security 2025, Seattle, WA, USA, August 13-15, 2025, pages 8135–8154. USENIX Association, 2025. [35] Xiaoyong Yuan and Lan Zhang. Membership inference attacks and defenses in neural network pruning. In 31st USENIX Security Symposium (USENIX Security 22), pages 4561–4578, 2022. [36] Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2615–2632, 2021. [37] Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. In 26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019, 2019.

[1] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planningoriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853–17862, 2023. [2] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2019. [3] Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep learning in medical image analysis. Medical Image Analysis, 42:60–88, 2017. [4] https://openai.com/chatgpt. [5] https://www.midjourney.com. [6] Jiachen T Wang, Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. Greats: Online selection of high-quality data for llm training in every iteration. Advances in Neural Information Processing Systems, 37:131197–131223, 2024. [7] Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7602– 7635, 2024. [8] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3614–3633, 2021. [9] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3712–3722, 2018. [10] Hanrong Ye and Dan Xu. Inverted pyramid multi-task transformer for dense scene understanding. In European Conference on Computer Vision, pages 514–530. Springer, 2022. [11] https://github.com/facebookresearch/detectron2/blob/main/MODEL ZOO.md. [12] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer, 2014. [13] Zheng Li, Yiyong Liu, Xinlei He, Ning Yu, Michael Backes, and Yang Zhang. Auditing membership leakages of multi-exit networks. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 1917–1931, 2022. [14] Teodora Baluta, Shiqi Shen, S Hitarth, Shruti Tople, and Prateek Saxena. Membership inference attacks and generalization: A causal perspective. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 249–262, 2022. [15] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093–3106, 2022. [16] Jiacheng Li, Ninghui Li, and Bruno Ribeiro. Membership inference attacks and defenses in classification models. In Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy, pages 5–16, 2021. [17] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy (S&P), pages 3–18. IEEE, 2017. [18] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In IEEE Symposium on Security and Privacy (S&P), pages 1897–1914. IEEE, 2022. [19] Hao Li, Zheng Li, Siyuan Wu, Chengrui Hu, Yutong Ye, Min Zhang, Dengguo Feng, and Yang Zhang. Seqmia: Sequential-metric based membership inference attack. In Proceedings of ACM Conference on Computer and Communications Security, pages 3496–3510, 2024.

14

[38] Yuntao Du, Jiacheng Li, Yuetian Chen, Kaiyuan Zhang, Zhizhen Yuan, Hanshen Xiao, Bruno Ribeiro, and Ninghui Li. Cascading and proxy membership inference attacks. In 33nd Annual Network and Distributed System Security Symposium, NDSS 2026, 2026. [39] Yuntao Du, Yuetian Chen, Hanshen Xiao, Bruno Ribeiro, and Ninghui Li. Imitative membership inference attack. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 2026. [40] Bozhidar Stevanoski, Ana-Maria Cretu, and Yves-Alexandre de Montjoye. Querycheetah: Fast automated discovery of attribute inference attacks against query-based systems. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 3451–3465, 2024. [41] Ali Naseh, Yuefeng Peng, Anshuman Suri, Harsh Chaudhari, Alina Oprea, and Amir Houmansadr. Riddle me this! stealthy membership inference for retrieval-augmented generation. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, CCS 2025, Taipei, Taiwan, October 13-17, 2025, pages 1245–1259. ACM, 2025. [42] Xinyu Gao, Xiangtao Meng, Yingkai Dong, Zheng Li, and Shanqing Guo. DCMI: A differential calibration membership inference attack against retrieval-augmented generation. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, CCS 2025, Taipei, Taiwan, October 13-17, 2025, pages 4184–4198. ACM, 2025. [43] Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 896–911, 2021. [44] Na Li, Yansong Gao, Hongsheng Hu, Boyu Kuang, and Anmin Fu. Compleak: Deep learning model compression exacerbates privacy leakage. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 2026. [45] Guangke Chen, Yedi Zhang, and Fu Song. SLMIA-SR: speaker-level membership inference attacks against speaker recognition systems. In 31st Annual Network and Distributed System Security Symposium, NDSS 2024, San Diego, California, USA, February 26 - March 1, 2024. The Internet Society, 2024. [46] Han Liu, Yuhao Wu, Zhiyuan Yu, and Ning Zhang. Please tell me more: Privacy impact of explainability through the lens of membership inference attack. In 2024 IEEE Symposium on Security and Privacy (S&P), pages 4791–4809. IEEE, 2024. [47] Jie Zhu, Jirong Zha, Ding Li, and Leye Wang. A unified membership inference method for visual self-supervised encoder via part-aware capability. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1241–1255, 2024. [48] Hongbin Liu, Jinyuan Jia, Wenjie Qu, and Neil Zhenqiang Gong. Encodermi: Membership inference against pre-trained encoders in contrastive learning. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 2081–2095, 2021. [49] Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye. Did the neurons read your book? document-level membership inference for large language models. In 33rd USENIX Security Symposium (USENIX Security 24), pages 2369–2385, 2024. [50] Rui Wen, Zheng Li, Michael Backes, and Yang Zhang. Membership inference attacks against in-context learning. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 3481–3495, 2024. [51] Yu He, Boheng Li, Liu Liu, Zhongjie Ba, Wei Dong, Yiming Li, Zhan Qin, Kui Ren, and Chun Chen. Towards label-only membership inference attack against pre-trained large language models. In 34th USENIX Security Symposium, USENIX Security 2025, Seattle, WA, USA, August 13-15, 2025, pages 1609–1628. USENIX Association, 2025. [52] Meng Tong, Yuntao Du, Kejiang Chen, Weiming Zhang, and Ninghui Li. Membership inference attacks on tokenizers of large language models. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 2026. [53] Yuetian Chen, Yuntao Du, Kaiyuan Zhang, Ashish Kundu, Charles Fleming, Bruno Ribeiro, and Ninghui Li. Window-based membership inference attacks against fine-tuned large language models. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 2026. [54] Yuefeng Peng, Ali Naseh, and Amir Houmansadr. Diffence: Fencing membership privacy with diffusion models. In 32nd Annual Network

and Distributed System Security Symposium, NDSS 2025, San Diego, California, USA, February 24-28, 2025. The Internet Society, 2025. [55] Yan Pang and Tianhao Wang. Black-box membership inference attacks against fine-tuned diffusion models. In 32nd Annual Network and Distributed System Security Symposium, NDSS 2025, San Diego, California, USA, February 24-28, 2025. The Internet Society, 2025. [56] Xiuling Wang, Xin Huang, Guibo Luo, and Jianliang Xu. Inference attacks against graph generative diffusion models. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 2026. [57] Yuke Hu, Zheng Li, Zhihao Liu, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen. Membership inference attacks against vision-language models. In 34th USENIX Security Symposium, USENIX Security 2025, Seattle, WA, USA, August 13-15, 2025, pages 1589–1608. USENIX Association, 2025. [58] Li Wang, Wenyu Chen, Ning Yu, Zheng Li, and Shanqing Guo. Vidleaks: Membership inference attacks against text-to-video models. In 35th USENIX Security Symposium (USENIX Security 26). USENIX Association, 2026. [59] Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? In First Conference on Language Modeling, 2024. [60] Jamie Hayes, Ilia Shumailov, Christopher A Choquette-Choo, Matthew Jagielski, Georgios Kaissis, Milad Nasr, Meenatchi Sundaram Muthu Selva Annamalai, Niloofar Mireshghallah, Igor Shilov, Matthieu Meeus, et al. Exploring the limits of strong membership inference attacks on large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [61] Hongyang Yan, Anli Yan, Li Hu, Jiaming Liang, and Haibo Hu. Mtlleak: Privacy risk assessment in multi-task learning. IEEE Transactions on Dependable and Secure Computing, 21(1):204–215, 2023. [62] https://huggingface.co/deepset/roberta-base-squad2. [63] https://huggingface.co/valhalla/t5-base-qa-qg-hl. [64] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016. [65] Yu He, Boheng Li, Yao Wang, Mengda Yang, Juan Wang, Hongxin Hu, and Xingyu Zhao. Is difficulty calibration all we need? towards more practical membership inference attacks. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1226–1240, 2024. [66] Hua Ma, Shang Wang, Yansong Gao, Zhi Zhang, Huming Qiu, Minhui Xue, Alsharif Abuadbba, Anmin Fu, Surya Nepal, and Derek Abbott. Watch out! simple horizontal class backdoor can trivially evade defense. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 4465–4479, 2024. [67] Shaofeng Li, Xinyu Wang, Minhui Xue, Haojin Zhu, Zhi Zhang, Yansong Gao, Wen Wu, and Xuemin Sherman Shen. Yes,{One-Bit-Flip} matters! universal {DNN} model inference depletion with runtime code fault injection. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1315–1330, 2024. [68] Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regression by conditional adversarial autoencoder. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5810– 5818, 2017. [69] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pages 3730–3738, 2015. [70] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1548–1558, 2021. [71] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770– 778, 2016. [72] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, 2021.

15

TABLE XIV: Single-task MIA performance and victim model task utility on CelebA (60 epochs). Train Acc and Val Acc denote the victim model’s accuracy on its training and validation sets, respectively, for its own classification task.

[73] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022. [74] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1):1–9, 2018. [75] Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. Effects of age and gender on blogging. In AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs, volume 6, pages 199–205, 2006. [76] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [77] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pages 9120–9132. PMLR, 2020. [78] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824–5836, 2020. [79] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308–318, 2016. [80] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, et al. Opacus: User-friendly differential privacy library in pytorch. In NeurIPS 2021 Workshop Privacy in Machine Learning, 2021. [81] Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In International Conference on Machine Learning, pages 1964–1974. PMLR, 2021. [82] Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (S&P), pages 866–882. IEEE, 2021.

Task

MIA AUC

MIA Acc.

Model Train Acc.

Model Val Acc.

Male Mouth Nose BHair Smile SHair Hat

0.5309 0.5489 0.6422 0.5981 0.5581 0.6395 0.5130

0.5166 0.5264 0.5842 0.5570 0.5326 0.5834 0.5088

100.00% 99.91% 99.68% 99.72% 99.71% 99.66% 100.00%

97.90% 92.64% 80.10% 88.13% 91.77% 72.71% 98.72%

TABLE XV: Prompt templates and example responses for LLM fine-tuning tasks. The model generates the response as free-form text following the prompt. Task

Prompt Template

Response

Gender

Based on the following blog post, predict the author’s gender (male or female).

male

Blog: {text} Gender: Horoscope

Based on the following blog post, predict the author’s zodiac sign.

Gemini

Blog: {text} Zodiac sign: Age

Based on the following blog post, predict the author’s age (as a number).

27

Blog: {text} Age:

A PPENDIX C. Differential Privacy as Defense

A. Single-Task MIA Performance and Victim Model Utility on CelebA

1) Experimental Setup: We conduct DP defense experiments on UTKFace using a compact CNN architecture compatible with Opacus [80]. Following the DP-SGD framework, we apply per-sample gradient clipping with maximum norm C = 1.0 and add calibrated Gaussian noise during training. We evaluate two noise multiplier configurations: σ = 0.2 (moderate noise) and σ = 0.5 (strong noise), compared against the non-private baseline. All models are trained for 50 epochs with privacy parameter δ = 10−5 . To simulate a realistic attack scenario, we train only the victim models with DP-SGD while keeping shadow models non-private. This represents a conservative evaluation where the adversary cannot replicate the victim’s privacy-preserving training procedure. 2) Result and Analysis: Impact on Model Utility. Table VIII presents the task performance under different DP configurations. As expected, adding noise during training degrades model utility across all tasks. With σ = 0.2, the Race classification accuracy drops by 4.6% (from 76.58% to 71.98%), while Age MAE increases by 1.25 years. The stronger σ = 0.5 configuration incurs larger utility loss: Race accuracy decreases by 9.58% and Age MAE increases by 2.73 years. Gender classification proves more robust, with accuracy

Table XIV reports the per-task MIA performance and the underlying victim model utility for the seven CelebA tasks used in the divergence study (Section V-B2). All models are ResNet-18 independently trained for 60 epochs with identical hyperparameters. The training accuracy approaches 100% for all tasks, indicating that each victim model fully fits its training set. However, the gap between training and validation accuracy varies notably across tasks: tasks with higher validation utility (e.g., Hat at 98.72%, Male at 97.90%) exhibit smaller trainval gaps and tend to expose weaker single-task MIA signals (AUC ≈ 0.51–0.53), while tasks with lower validation utility (e.g., SHair at 72.71%, Nose at 80.10%) exhibit larger generalization gaps and yield stronger single-task leakage (AUC ≈ 0.64). This is consistent with the well-known correlation between overfitting and membership leakage. Despite the heterogeneous per-task leakage strengths, PRIME consistently amplifies privacy risk through joint fusion, as analyzed in Section V-B2. B. Prompt Template The Prompt templates used are shown in Table XV.

16

Feature Extraction. Since the models do not return probabilities, we construct features based on label stability. For classification tasks (Gender, Race), we directly compare the predicted class indices. For the regression task (Age), we discretize the output into 12 bins (0-9, 10-19, etc.) to treat it as a classification problem for consistency calculation. The extracted features for each task are: • Original Correctness: Boolean indicating if the prediction on the clean image matches the ground truth. • Consistency Ratio: The fraction of perturbed samples xi that yield the same prediction as the clean image x. • Robustness Accuracy: The fraction of perturbed samples xi that are classified correctly. • Label Entropy: The entropy of the distribution of predicted labels across the N perturbations. Joint Attack Construction. The PRIME attack model aggregates the extracted 4-dimensional feature vectors from all three tasks, resulting in a 12-dimensional joint feature vector. A Random Forest classifier (100 estimators) is trained on this joint representation to infer membership.

remaining above 86% under strong DP. Defense Effectiveness Against MIA. Table IX shows the MIA performance under DP protection. The results demonstrate that DP-SGD provides substantial defense against both single-task and joint attacks. Without privacy protection, PRIME achieves an AUC of 0.879, substantially threatening membership privacy. With moderate DP (σ = 0.2), the joint attack AUC drops dramatically to 0.550, only marginally above random guessing. Strong DP (σ = 0.5) further reduces the AUC to 0.519, effectively neutralizing the attack. Single-task attacks are similarly mitigated, with average AUC decreasing from 0.674 to approximately 0.51 to 0.53. Privacy-Utility Trade-off Analysis. Results in Table X reveal a favorable privacy-utility trade-off. Moderate DP (σ = 0.2) reduces PRIME attack AUC by 0.329 (from 0.879 to 0.550), achieving 86.7% protection rate while incurring only 4.60% Race classification accuracy loss. Strong DP (σ = 0.5) provides even stronger defense with 95.1% protection rate, though at the cost of 9.58% accuracy degradation. For privacysensitive applications such as medical imaging, sacrificing moderate utility to achieve near-complete attack mitigation represents an acceptable trade-off. Theoretical vs. Empirical Privacy. A noteworthy observation is the gap between theoretical privacy guarantees and empirical attack effectiveness. The σ = 0.2 configuration yields ε ≈ 635, which theoretically provides weak privacy guarantees according to the standard DP framework. However, the empirical PRIME attack AUC of 0.550 indicates strong practical protection, nearly reducing the attack to random guessing. Conversely, σ = 0.5 achieves a much tighter ε ≈ 22.7, yet the empirical improvement is marginal (AUC: 0.550 → 0.519). This discrepancy aligns with recent findings that DP-SGD’s empirical privacy often exceeds its theoretical bounds [82], suggesting that ε alone may not fully capture real-world privacy risks against MIA. Residual Privacy Amplification Under DP. Despite the substantial protection offered by DP, we observe that PRIME consistently outperforms single-task MIAs even under strong privacy protection. As shown in Table IX, with σ = 0.5, the joint attack achieves AUC of 0.519 compared to the average single-task AUC of 0.511. While both values are close to random guessing, the persistent gap indicates that the privacy amplification effect from ODMM exposure is not entirely eliminated by DP. This residual risk, though small, underscores that multi-model deployments require more conservative privacy budgets than single-model scenarios.

E. Proof of Theorem 1 The proof follows from a data processing argument in reverse. Let σ(O1:k ) denote the σ-algebra generated by the first k model outputs and σ(O1:k+1 ) the σ-algebra generated by k + 1 outputs. Since O1:k is a deterministic function of O1:k+1 (projection onto the first k coordinates), we have σ(O1:k ) ⊆ σ(O1:k+1 ). Moving from k to k+1 models enlarges the observable σ-algebra. The Bayes-optimal membership inference attack corresponds to the likelihood-ratio test on the observed data. For any test A∗k that is measurable with respect to σ(O1:k ), we can construct the same test as a function of O1:k+1 by simply ignoring the (k + 1)-th output. Therefore, the supremum over all tests measurable with respect to σ(O1:k+1 ) is taken over a strictly larger class of functions, yielding: Adv(A∗k+1 ) = supA∈σ(O1:k+1 ) Adv(A) ≥ supA∈σ(O1:k ) Adv(A) = Adv(A∗k )

F. Proof of Theorem 2 Given the leakage S1 (z), . . . , Sk (z) from k ODMM models, the joint log-likelihood ratio is defined as: p(S1 , . . . , Sk | M = 1) Λk (z) = log , p(S1 , . . . , Sk | M = 0) where M ∈ {0, 1} denotes the membership status and p(· | M ) denotes the joint conditional distribution of all leakage given M . Under the conditional independence assumption (Assumption 1), the joint conditional distribution factorizes into a product of per-model terms:

D. Hard-Label Experiment Details p(S1 , . . . , Sk | M ) =

Perturbation Setup. For the hard-label experiments described in Section VI-B, we utilize a noise-based estimation method. For a given input image x, we generate N = 30 perturbed samples xi = x + δi , where δi ∼ N (0, σ 2 ) with σ = 0.05. We query each task-specific model with these perturbed samples to obtain a set of labels.

k Y

pi (Si | M ).

i=1

Substituting this factorization into the log-likelihood expression and applying the logarithm of a product yields: Λk (z) = log

k Y pi (Si (z) | M = 1) i=1

17

pi (Si (z) | M = 0)

=

k X i=1

log

pi (Si (z) | z ∈ D) , pi (Si (z) | z ∈ / D)

which is the additive decomposition stated in Eq. (5). G. Proof of Theorem 3 For each model fi , let wi denote the standalone leakage strength, defined as the expected separation of its leakage between members and non-members: wi = E[Si | M = 1] − E[Si | M = 0]. The meta-classifier of PRIME aggregates the leakage via concatenation, producing a joint statistic whose overall leakage is: k k k hX i hX i X E Si | M = 1 − E Si | M = 0 = wi . i=1

i=1

i=1

Since the per-model leakage may be correlated with pairwise correlation ρij = Corr(Si , Sj ), the variance of the aggregate statistic is: k k X k X  X Var Si = σi σj ρij , i=1

i=1 j=1

σi2

where = Var(Si ). Setting wi = σi without loss of generality (by rescaling), the signal-to-noise ratio (SNR) of the aggregate statistic becomes:
2 Pk i=1 wi = α. SNRk = Pk Pk i=1 j=1 wi wj ρij Since the discriminative power of the aggregate statistic is determined by SNRk , the overall amplification is directly proportional to α.

18