Fair Finetuning Mitigates Distribution Inference Attacks Rakshit Naidu Georgia Institute of Technology [email protected]
arXiv:2606.01719v1 [cs.LG] 1 Jun 2026
Abstract Machine learning models trained on sensitive data can inadvertently leak population-level information about their training distributions—a threat known as a distribution inference attack (DIA). An adversary with only black-box access to a model can infer sensitive demographic properties, such as the proportion of a subgroup represented in training, without observing a single data record directly. While defenses such as differential privacy and property unlearning have been proposed, the connection between fairness constraints and distributional leakage remains largely unexplored. We propose Fair Fine-tuning (FFt) as a principled defense: a trained model is fine-tuned on samples from the complementary distribution subject to an Equalized Odds (EO) constraint. We develop a complete theoretical characterisation. Our main result (Theorem 1) gives the tight bound Adv(A, Mf ) ≤ ∆EO · W , where W is a computable distributional shift weight that quantifies how distinguishable the two training distributions are by their sensitive-attribute composition. We further prove a necessary condition for FFt to reduce adversarial advantage (Theorem 2) and establish tightness of the bound (Proposition 1). We evaluate under a biased distribution protocol—pure demographic groups as G0 and G1 , so W = 1 and adversarial leakage is maximal—across six datasets spanning tabular (ACS Income, COMPAS, German Credit), image (UTKFaces), and NLP (Bias in Bios) modalities, with LSAC evaluated in the appendix. We illustrate that rehearsal-based FFt consistently reduces the adversarial accuracy gap below the detection threshold τ = 0.1 in the majority of runs across all settings and modalities. For instance, on ACS Income, the gap falls from ∼ 15% to under 4%, displaying the effectiveness of our method. Our work provides the first formal bound connecting a model’s measured EO disparity directly to its adversarial advantage in the DIA game, opening a new avenue for unified fairness-and-privacy defenses against inference attacks.
Introduction Machine learning models trained on sensitive data can inadvertently expose population-level properties of their training distribution to adversaries with only black-box access to the model’s outputs. Earlier work on membership inference attacks (Shokri et al. 2017; Yeom et al. 2018) established that querying a model can reveal whether a specific individual was in its training set. We focus on a more recent and
less-studied threat: distribution inference attacks (DIAs), in which an adversary recovers global properties of the training distribution — such as the proportion of a demographic subgroup, label priors, or correlations between sensitive attributes and outcomes — without observing any individual record directly. As a concrete example, consider a model trained on a loan recidivism dataset that includes gender as an attribute. An adversary who can query the model may be able to infer not just that gender is present, but the precise male-to-female ratio in the training data. Such population-level leakage can have serious consequences: it reveals the demographic composition of a private dataset and may enable downstream discrimination or targeted manipulation. Given these risks, a natural question arises: can training procedures that enforce fairness constraints reduce distributional leakage? Fairness interventions — such as equalized odds penalties, adversarial debiasing, or reweighting — are designed to suppress the model’s dependence on demographic structure. If distribution inference attacks exploit exactly these demographic cues, then enforcing fairness may implicitly limit the distributional information available to an adversary. Despite growing interest in both fairness and privacy, this connection has remained largely unexplored. We answer the question formally. Our central result (Theorem 1) proves that for any model Mf fine-tuned with an Equalized Odds (EO) constraint, the adversary’s advantage in the DIA game is bounded by ∆EO · W , where ∆EO is the EO disparity and W is a computable distributional shift weight. To our knowledge, this is the first formal bound directly connecting an operationalised fairness metric (EO disparity) to adversarial advantage in the distribution inference game — prior formal frameworks such as attribute privacy (Zhang, Ohrimenko, and Cummings 2022) and Pufferfish (Kifer and Machanavajjhala 2012) characterise leakage over data-generating mechanisms rather than yielding explicit, computable bounds on an adversary’s distinguishing accuracy as a function of a model’s measured fairness. We propose Fair Fine-tuning (FFt): after training a baseline model on the base distribution G0 , the defender then fine-tunes it on a sample from the complementary distribution G1 subject to an EO penalty, as shown in Figure 1. The EO constraint is the operative ingredient — it forces the residual per-label unfairness δy to zero, cancelling the base
prediction rate Py,0 from the leakage expression entirely. A rehearsal component (replaying a fraction ρ of G0 data during fine-tuning) prevents catastrophic forgetting. Throughout this work, the adversary is assumed to have only blackbox access to the released model. We evaluate FFt under a biased distribution protocol — G0 and G1 are pure single-demographic groups, so W = 1 and adversarial leakage is maximal — across six datasets spanning tabular (ACS Income, COMPAS, German Credit), image (UTKFaces), and NLP (Bias in Bios, LSAC) modalities. Rehearsal-based FFt reduces the adversarial accuracy gap below the detection threshold τ = 0.1 in the majority of runs across all settings, and the bound Adv ≤ ∆EO holds empirically in every row of Table 1. Our contributions are: • We introduce Fair Fine-tuning (FFt) as a principled, post-hoc defense against distribution inference attacks, requiring no cryptographic overhead and only black-box deployment1 . • We prove the tight bound Adv(A, Mf ) ≤ ∆EO · W (Theorem 1), where W is a computable distributional shift weight. This is the first formal result connecting a fairness metric to adversarial advantage in the DIA game. • We characterise when FFt is beneficial (Theorem 2), prove the bound is tight (Proposition 1), establish the biased protocol as the worst case for the defender (Corollary 1), and identify the group-size failure regime (Proposition 2). • We introduce a biased distribution evaluation protocol (W = 1) that creates maximal adversarial leakage, providing a stringent and reproducible testbed for DIA defenses. • We experiment across six datasets and three modalities and confirm that rehearsal-based FFt generalises broadly and that the theoretical bound holds empirically in every evaluated setting.
Related Work Many formal definitions of privacy have been established in prior work, including multiple notions of differential privacy (DP) at either the data level (Cormode et al. 2018) or the model level (Abadi et al. 2016) or the output (inference) level (Chaudhuri, Monteleoni, and Sarwate 2011) of the machine learning (ML) pipeline. The key idea of DP involves estimating how much one individual contributes to the entire dataset (Dwork et al. 2006). This does not capture the risks exhibited by inferring the data distribution. The Pufferfish framework (Kifer and Machanavajjhala 2012) generalises DP by allowing arbitrary semantic privacy secrets and adversary beliefs; Zhang et al. (Zhang, Ohrimenko, and Cummings 2022) operationalise this as “attribute privacy,” where the protected secret is a dataset-level property (e.g. the proportion of a sensitive attribute). While attribute privacy provides a rigorous compositional framework, it is defined over data-generating mechanisms rather than model 1 Code is available at https://anonymous.4open.science/r/fairdefense-dia-B2BF/README.md.
outputs and does not directly yield actionable bounds on an adversary’s accuracy in the DIA game. Our approach is operationally complementary: we measure protection empirically via the adversary’s accuracy gap and provide a formal bound through the EO disparity. Hence, below we summarize two main classes of inference attacks:
Inference attacks Membership Inference Membership inference attacks (MIA) are attacks performed by the adversary on the model in order to reveal whether a certain sample is part of the training set or not (Hu et al. 2022). (Shokri et al. 2017) perform membership inference on a target model by training a shadow model. The shadow model is expected to capture differences between the trained and the untrained parameters of the target model. The adversary here only has black-box access to the model and they also show a real-life usecase on how sensitive information from a hospital discharge dataset could be revealed. (Yeom et al. 2018) analyze how overfitting reveals information about the training set. Mainly, they examine the relationship between overfitting and influence and claim that overfitting is enough to perform a successful MIA. Next, we dive into the works on distribution inference. Distribution Inference (Suri and Evans 2021) formalize and establish distribution inference risks. They introduce the distribution inference game (explained in Figure 1). Later, (Suri et al. 2023) also work on understanding this class of attacks by restricting adversary’s access to the training distributions. A survey by (Salem et al. 2023) reveal connections between membership inference and distribution inference attacks and they also introduce game-based frameworks on understanding privacy risks in ML models. Recently, (Hartmann et al. 2023) identified three main sources of leakage of distributional information : (1) memorization of information of labels tied to specific features i.e. E[Y |X], (2) incorrect inductive bias of the model and finally (3) the finiteness of the training data. However, they have a strong underlying assumption on taking examples satisfying linear regression i.e. the relationship between the features X = (X1 , X2 , · · · , Xd ) and the label Y takes the form of Y = θ⊤ X + ϵ, where θ is the coefficients vector and ϵ ∼ N (0, σ 2 ) refers to the error term. They extend their empirical results to neural networks as well. They also conduct a simple experiment on synthetic data to demonstrate that a perfect associational model considers the joint distribution of the features to perform a prediction (hence, leaks distributional information) while a perfect causal model preserves the relationship between the features and the label by design (hence, does not leak distributional information). More recently, (Xu et al. 2024) reproduced these attacks in the federated learning (FL) setting and claimed that FL is vulnerable to client-side training data distribution inference attacks, where a malicious client could recreate a victim’s private data. Additionally, (Yu et al. 2025) proposed a novel
Trainer T 1: 2: 3: 4: 5:
Adversary A
b ←$ {0, 1} Sb ∼ Gb (D) train
M ←−−− Sb S¬b ∼ G¬b (D) finetune(M, S¬b )
Mf ←−−−−−−−−− M
6:
*New* *New* Mf b̂ = H(Mf )
7:
Figure 1: The Fair Fine-tuning (FFt) defense. The baseline protocol (Steps 1–3, 6–7) is from (Suri and Evans 2021). We introduce FFt, as a defense against distribution inference attacks, by adding Steps 4–5: sample from the complementary distribution G¬b and then fine-tune the model with an EO penalty before release. attack on FL based on shared model parameters which can deduce the data distribution of the global model, unlike most current attacks which were aimed at client-specific data reconstruction.
ing publicly known complementary data. Second, and most importantly, we provide the first formal result connecting a fairness metric (EO disparity) to adversarial advantage in the DIA game (Theorem 1), establishing why fairness serves as a principled defense—not merely an empirical one.
Defenses against Distribution Inference Attacks (Stock et al. 2023) propose “property unlearning”, a mitigation strategy against white-box distribution inference attacks. They also provide multiple defensive strategies against distribution inference attacks–for example, through data preprocessing, by adding Gaussian noise to images. (Hartmann et al. 2023) used causal learning techniques to create robust models that were resistant to distribution inference attacks. (Noorbakhsh et al. 2024) introduced “Inf2Guard”, a defensive method backed by information theory, to learn privacy-preserving representations against major types of inference attacks, including distribution inference attacks. (Zhang et al. 2021) present four broad categories of defense approaches against inference attacks–differential privacy, adversarial machine learning, watermarking techniques, and cryptographic techniques. Fairness and privacy: a nuanced relationship. Prior work has shown that fairness constraints do not automatically confer privacy: imposing demographic parity or EO can in some settings increase membership inference vulnerability by forcing the model to overfit group-level statistics (Chang and Shokri 2021). Our result is not in tension with this finding — we address distribution inference (population-level leakage) rather than membership inference (individual-level leakage), and we show that EO specifically bounds the adversary’s distributional distinguishing advantage. The two attack surfaces are governed by different mechanisms, and a defense against one does not imply defense against the other. Our work differs from the above defenses in two ways. First, FFt does not require differential privacy noise, cryptographic overhead, or white-box model access: it is a posttraining fine-tuning step that any model owner can apply us-
Problem Formulation Motivation We motivate our problem with two concrete examples, described in (Hartmann et al. 2023) and in (Suri and Evans 2021; Mahloujifar, Ghosh, and Chase 2022). (1) Consider an e-commerce platform that sells sports products (for example, surfboards). The platform trains a model for its own recommender system. A competitor may want to learn whether that platform sells to many or few young customers. If age was one of the sensitive features the model was trained on and assuming that surfboards are popular among younger than older users, the competitor may infer sensitive information (age, here) about the distribution the model was trained on. (2) A hospital trains a sepsis-risk classifier on patient records. An adversary—such as an insurance company with black-box query access—can infer whether the training cohort was drawn predominantly from the ICU (older, higheracuity patients, high positive-label rate) versus the emergency department (younger, more diverse, lower positivelabel rate). Because the two cohorts differ in their demographic composition, the model’s accuracy gap across age and race groups reveals which cohort was used—leaking protected demographic information about the hospital’s private patient population and potentially enabling discriminatory underwriting.
Threat model Let there exist an underlying, true, public data distribution D = {X , Y} where X refers to the input data and Y its labels. Assume that there exist two transformation functions G0 and G1 that map D onto different subdistributions. Selecting appropriate transformation functions enables a wide
range of attacks on different types of sensitive information that could be revealed by the trained model. Both the model trainer T and the adversary A are assumed to know D, G0 , and G1 —note that this is the standard assumption in the distribution inference literature (Suri and Evans 2021; Suri et al. 2023). Crucially, the secret is the bit b ∈ {0, 1} that determines which of the two transformations was applied when the trainer sampled the training set Sb . The adversary does not observe b directly or the training process; they only have black-box (query) access to the released model. Hence the adversary must infer b from the model’s outputs alone—this is precisely the distribution inference problem. The fact that both parties know the transformation functions makes the setting comparable to a known-plaintext model: the functions define the hypothesis space, but the training decision (which function was used) remains hidden. This standard formulation captures realistic scenarios such as auditing whether a deployed model was trained on a gender-balanced dataset versus a predominantly male one. These assumptions are illustrated in the original threat model and also our defense mechanism, as shown in Figure 1. The traditional distribution inference attack setting involves selecting one of the two available distributions at random (Step 1 in Figure 1) and training the model on a dataset sampled from the selected distribution (Steps 2 & 3 in Figure 1). This model is then deployed in the real world, over which the adversary performs the attack (Steps 6 & 7 in Figure 1). We introduce our finetuning setup in Figure 1. We apply two additional steps i.e. Steps 4 & 5 which correspond to sampling from the “other” distribution and finetuning the model on this freshly sampled dataset respectively. This approach is often used in out-of-distribution settings (such as (Tran et al. 2022) use this approach for mitigating unfairness incurred due to pruning ML models) where the model intends to generalize to new domains, which it hasn’t encountered before while training. Through finetuning, the model is expected to generalize to a never-before-seen domain and improve the accuracy on out-of-distribution test data.
∆EO = max Pr Ŷ = 1 | A = 0, Y = y − y∈{0,1}
Pr Ŷ = 1 | A = 1, Y = y
(2)
for all y ∈ {0, 1}, where Ŷ ∈ {0, 1} is the model’s prediction and A is the sensitive attribute the attacker wants to infer (e.g. gender). EO requires equal true positive rates (y = 1) and equal false positive rates (y = 0) across groups: an EO-satisfying model makes the same kinds of errors at the same rates regardless of group membership. We achieve perfect EO when ∆EO = 0, i.e. Pr Ŷ = 1 | A = 0, Y = y = Pr Ŷ = 1 | A = 1, Y = y for all y. Theorem 1 (Adv–EO Bound). Let Mf be the fine-tuned model with Equalized Odds disparity ∆EO w.r.t. a protected attribute A. Define the distributional shift weight X Pr[Y = y] ∆Py , W = y∈{0,1}
where ∆Py = Pr[A = 1 | Y = y, D1 ] − Pr[A = 1 | Y = y, D0 ] measures how much the sensitive attribute’s conditional distribution shifts between D0 and D1 . Then Adv (A, Mf ) ≤ ∆EO · W. Since |∆Py | ≤ 1 for every y, we immediately obtain W ≤ 1, yielding the simpler corollary Adv(A, Mf ) ≤ ∆EO (Corollary 1). Proof Sketch. Throughout this proof, we write D0 = G0 (D) and D1 = G1 (D) for the two transformed distributions corresponding to the training and fine-tuning datasets S0 and S1 , respectively. For simplicity, we analyze the difference in the overall positive prediction rate (PPR), Pr[Ŷ = 1 | D1 ] − Pr[Ŷ = 1 | D0 ]. For a fixed y, we define ∆y = Pr[Ŷ = 1 | Y = y, D1 ] − Pr[Ŷ = 1 | Y = y, D0 ] . By the law of total expectation, conditioning on the true label Y :
Theoretical Analysis Definition 1 (Adversarial Advantage (Suri and Evans 2021)). The adversarial advantage gained by an adversary A over a model M is defined as :
Pr[Ŷ = 1 | D1 ] − Pr[Ŷ = 1 | D0 ] X = Pr[Y = y] Pr[Ŷ = 1 | Y = y, D1 ] y
Adv (A, M ) = Pr b̂ M ← S0 − Pr b̂ M ← S1
− Pr[Ŷ = 1 | Y = y, D0 ]
(1)
where b̂ is the inferred bit by the adversary A using hypothesis H. It is the absolute difference between the probability of attaining the inferred bit b̂ given that the model was either trained on the sampled datasets S0 or S1 . Definition 2 (Equalized Odds (EO) (Hardt, Price, and Srebro 2016)). A model M satisfies Equalized Odds (EO) iff
≤
X
Pr[Y = y] · ∆y ,
y
where the last step applies the triangle inequality (the absolute value of a sum is at most the sum of absolute values weighted by probabilities). Since Adv(A, Mf ) is the absolute difference in prediction probabilities under D1 vs. D0 , we obtain:
X
Adv (A, Mf ) ≤
Pr[Y = y] · ∆y .
(3)
y∈{0,1}
We now bound ∆y . We expand this term by conditioning on A ∈ {0, 1} and writing the model’s conditional prediction rates as Py,a = Pr[Ŷ = 1 | Y = y, A = a] where ∆Py,a = Pr[A = a | Y = y, D1 ] − Pr[A = a | Y = y, D0 ]: X ∆y = Py,a · ∆Py,a . (4) a∈{0,1}
Since Pr[A = 0 | Y = y, D] + Pr[A = 1 | Y = y, D] = 1, the differences satisfy2 ∆Py,0 = −∆Py,1 = −∆Py . We introduce the residual unfairness δy through the Equalized Odds constraint: Py,1 = Py,0 + δy ,
|δy | ≤ ∆EO .
Substituting this relationship into ∆y from Equation 4: ∆y = Py,0 · (−∆Py ) + (Py,0 + δy ) · (∆Py ) = − Py,0 Py,0 ∆P ∆P y + y + δy ∆Py = δy ∆Py . The first line expands ∆y by substituting Py,1 = Py,0 + δy : the group-0 contribution is Py,0 · (−∆Py ) (its prediction rate weighted by how much group-0’s conditional prevalence decreases from D0 to D1 ), and the group-1 contribution is (Py,0 +δy )·(∆Py ) (weighted by how much group-1’s prevalence increases). The Py,0 ∆Py terms cancel exactly, leaving δy ∆Py . This cancellation is the central insight of the proof: group 0’s prediction rate Py,0 contributes equally to both distributions and drops out entirely. We see that leakage is governed solely by δy (the residual per-label EO gap) scaled by ∆Py , which is the compositional shift between D0 and D1 . In particular, a perfectly fair model (δy = 0) leaks nothing regardless of its accuracy, while an unfair model leaks in direct proportion to both its unfairness and the distributional shift. Thus, |∆y | ≤ |δy | · |∆Py | ≤ ∆EO · |∆Py |. Summing over y yields the final bound: X Adv (A, Mf ) ≤ Pr[Y = y] · ∆y
(Equation 3)
y∈{0,1}
≤
X
Pr[Y = y] · ∆EO · |∆Py |
y∈{0,1}
= ∆EO ·
Pr[Y = y] |∆Py |
y∈{0,1}
|
{z
=W
≤ ∆EO · W ≤ ∆EO . 2
See Appendix for intuition on this difference.
Theorem 1 demonstrates the effect of fairness through fine-tuning. By forcing the fine-tuned model Mf to adhere to the Equalized Odds constraint, our method bounds the residual unfairness δy , causing the base prediction rate Py,0 to drop out of the leakage expression entirely. The adversarial advantage therefore depends only on δy — the sole surviving term after the algebraic cancellation. Consequently, the adversarial advantage is directly bounded by the product of the EO imperfection ∆EO and the distributional shift weight W , formally establishing fair fine-tuning as a principled defense against distributional leakage.3 Remark 1 (Scope: PPR gap vs. accuracy gap). Theorem 1 bounds the positive-prediction-rate (PPR) gap | Pr[Ŷ = 1 | D1 ] − Pr[Ŷ = 1 | D0 ]|. The Loss Test adversary observes the accuracy gap, which differs from the PPR gap by a baserate floor term ∆π · (TPR + FPR − 1) that persists even at ∆EO = 0 when the two groups have different label prevalences (derived in Appendix). In all our experimental settings this floor is small relative to the reductions achieved by FFt; when base rates diverge sharply (LSAC race) the floor dominates and explains the residual gap. The bound is therefore informative for the accuracy gap whenever |∆π| is moderate. P Remark 2 (Diagnostic role of W ). W = y Pr[Y = y]|∆Py | is estimable from labeled samples of D0 and D1 , both of which are available to the defender under the standard threat model (both G0 and G1 are public). When W is small—e.g. the two training distributions share nearly the same conditional composition of A—the adversarial advantage is already small regardless of ∆EO , and FFt may be unnecessary. Conversely, when W ≈ 1 (biased distributions), the bound tightens to Adv ≤ ∆EO . Proposition 1 (Tightness). The bound Adv(A, Mf ) ≤ ∆EO · W is tight. Consider the deterministic model M ∗ with Pr[Ŷ = 1 | Y = y, A = 0] = 0 and Pr[Ŷ = 1 | Y = y, A = 1] = 1 for all y. Then ∆EO = 1. Under biased distributions with W = 1, the adversary observes the group that achieves higher accuracy also achieves Adv(A, M ∗ ) = ∆EO · W = 1. Corollary 1 (Biased Distribution). Under the biased distribution protocol where G0 and G1 are pure singledemographic groups, |∆Py | = 1 for all y, so W = 1 and the bound of Theorem 1 reduces to Adv(A, Mf ) ≤ ∆EO .
X
The intermediate quantity W ≤ 1 is attained with equality when each |∆Py | = 1, i.e. when D0 and D1 are pure singlegroup distributions (the biased setup established in our experiments). This concludes the proof.
} (5)
By Proposition 1 this bound is tight. The biased protocol is therefore the worst case for the defender: any FFt that controls ∆EO under pure groups is guaranteed to succeed under any mixed-group protocol with W < 1. 3 The analysis extends naturally to multiple simultaneous sensitive attributes; see Appendix for the formal statement.
Theorem 2 (FFt Improvement Condition). Let Mbase be the baseline model and Mf the FFt model. FFt reduces adversarial advantage (i.e. Adv(A, Mf ) < Adv(A, Mbase )) whenever fft base ∆fft < ∆base . EO · W EO · W base In the biased distribution setting W = W fft = 1, so the base . The practical value < ∆ condition simplifies to ∆fft EO EO of this characterisation is its identification of catastrophic forgetting as the principal failure mode: if fine-tuning on G1 causes the model to lose its calibration on G0 , then ∆fft EO rises rather than falls, and the bound of Theorem 1 no longer guarantees improvement. This directly motivates the rehearsal component of Algorithm 1. Proof sketch. Apply Theorem 1 to both models: base fft . and Adv(Mbase ) ≤ ∆base Adv(Mf ) ≤ ∆fft EO · W EO · W Whenever the right-hand side for Mf is strictly smaller, the bound guarantees improvement. Equality between W values holds in the biased setup because both distributions are pure groups, so |∆Py | = 1 for all y under both models. Proposition 2 (Group-Size Failure). Let α = |S1 |/|S0 | be the fine-tuning-to-training size ratio. Under rehearsal-based FFt with fraction ρ, each fine-tuning batch draws gradient signal from G1 at weight 1/(1+ρ). As α → 0, the G1 signal is insufficient to recalibrate the G0 -fitted decision boundary, base so ∆fft EO ̸< ∆EO ; by the contrapositive of Theorem 2, FFt then provides no improvement. Empirically, LSAC race (α ≈ 0.19, 5.3:1 size asymmetry) illustrates this regime: FFt halves the adversarial gap but cannot close it (results shown in Appendix). The W factor connects theory to practice. Corollary 1 shows that the biased distribution protocol (W = 1) is the worst case: FFt’s effectiveness depends entirely on how well it suppresses the EO gap. Theorem 2 and Proposition 2 together make this actionable—the defender should apply FFt when W > 0 and the baseline ∆EO is large, verify that α ≥ 0.2 to avoid the group-size failure regime, and expect diminishing returns when size asymmetry is severe. FFt vs. plain data augmentation. A natural question is whether the protection offered by FFt arises merely from training on data from both distributions i.e., from balanced data augmentation, rather than from the fairness constraint itself. The theoretical analysis above shows that the EO constraint is the driving force: Theorem 1 holds because |δy | ≤ ∆EO , where δy is the residual unfairness that survives after fine-tuning. Plain fine-tuning on the complementary distribution (without an EO penalty) does not guarantee small ∆EO and therefore does not yield the bound Adv(A, Mf ) ≤ ∆EO . The fairness regularization is the mechanism that suppresses the model’s sensitivity to the distributional split, not the additional data per se.
FFt Deployment Procedure Algorithm 1 operationalises the theoretical results above into a practical deployment procedure. A rehearsal buffer (mixing ρ · |S1 | examples from S0 into fine-tuning) prevents the catastrophic forgetting that would inflate ∆fft EO above ∆base . EO
Algorithm 1: Fair Fine-Tuning (FFt) — Deployment Procedure Require: Mbase baseline model trained on G0 S0test , S1test held-out test sets S1 ∼ G1 fine-tuning set S0 ∼ G0 rehearsal source τ, ρ, T threshold, rehearsal fraction, iterations Ensure: Mf — fine-tuned model with reduced adversarial advantage 1: g ← acc(Mbase , S0test ) − acc(Mbase , S1test ) 2: if g ≤ τ then 3: return Mbase // gap below threshold (no action needed) 4: end if 5: R ← sample ⌊ρ · |S1 |⌋ examples from S0 6: Sft ← S1 ∪ R // prevents catastrophic forgetting 7: Mf ← Mbase // warm-start from baseline 8: Fine-tune Mf on Sft for T steps minimising:
Lft = LCE + λ ∆EO (Mf ) ′
9: g ← acc(Mf , S0test ) − acc(Mf , S1test ) 10: if g ′ ≤ τ then 11: return Mf // gap below detection threshold 12: else 13: warn: augment S1 or reduce |S0 |/|S1 | imbalance 14: return Mf 15: end if
Experiments Datasets and Experimental Setup. We evaluate on six datasets across three modalities: tabular (ACS Income, COMPAS, German Credit, LSAC), image (UTKFaces), and NLP (Bias in Bios), all under the biased distribution protocol (W = 1). LSAC results are presented separately in the Appendix. ACS Income 2018 (California). The American Community Survey (ACS) Income dataset (Ding et al. 2021) for California (1-Year, 2018) contains 195,665 individuals with 10 features. The binary target is income >$50K. We experiment on sex (male vs. female) and race (white vs. non-white, i.e. RAC1P ̸= 1). COMPAS (ProPublica). The COMPAS recidivism dataset (Angwin et al. 2016) contains 6,172 criminal defendants after standard filtering (Suri and Evans 2021). Features include age, sex, prior counts, juvenile offense counts, and charge degree. The binary target is two-year recidivism. We experiment on race (African-American vs. Caucasian). German Credit (UCI Statlog). The German Credit dataset (Dua and Graff 2019) contains 1,000 credit applicants with 20 features (credit amount, duration, purpose, housing, employment, etc.). The binary target is creditworthiness (good/bad). We experiment on sex extracted from the personal status attribute (male vs. female). UTKFaces. The UTK Face dataset (Zhang, Song, and Qi 2017) contains approximately 20,000 face images annotated with age (0–116), gender, and race (White, Black, Asian, In-
dian, Other). The binary target is age group (young: ≤ 30, old: > 30). We use race as the sensitive attribute (G0 : White, G1 : non-White). The model is a pretrained ResNet18 fine-tuned end-to-end for 5 epochs; FFt uses 2 additional epochs with rehearsal (ρ = 0.2). Bias in Bios. The Bias in Bios dataset (De-Arteaga et al. 2019) contains ∼ 400K online biographies annotated with occupation (28 classes) and gender. We use the genderscrubbed hard text field to prevent trivial leakage via pronoun cues, and treat sex as the sensitive attribute (G0 : male, G1 : female). We sample 5K biographies per gender group for training and 2K for test to keep memory tractable. The model is a TF-IDF (500 features, sublinear TF) + MLP (128, 64) trained on G0 biographies.
Setup. We set G0 to one pure demographic group (e.g. white individuals) and G1 to the complementary group (e.g. nonwhite individuals). Both distributions are pure groups, so W = 1 (Remark 2), and the bound Adv ≤ ∆EO applies directly—making the EO constraint the operative control on leakage. We choose this protocol deliberately as the worst case for the defender: pure groups maximise the distributional shift |∆Py | = 1, giving the adversary the strongest possible signal, so any defense that succeeds here is guaranteed to succeed under milder mixed-distribution protocols where W < 1. We use an 80–20% train-test split, remove missing values, and normalize features (column-wise ℓ2 scaling, then row-wise ℓ2 normalization). Categorical columns (LSAC and German Credit) are one-hot encoded with column alignment between G0 and G1 . We train a Multi-layer Perceptron (MLP) with three hidden layers [32, 16, 8] for at most 500 iterations on G0 training data (200 for German Credit, whose one-hot-encoded 62-feature matrix converges in fewer iterations). This is the Baseline. For FFt we warm-start from the Baseline weights and fine-tune on G1 for 100 additional iterations (50 for German Credit). For ACS, COMPAS, and German Credit we additionally mix 20% of G0 training data into the fine-tuning batch (rehearsal); refer to the Discussion Section for when rehearsal is appropriate. All results are averaged over 10 independent runs.
Metrics. We evaluate against two black-box metrics from (Suri and Evans 2021): • Loss Test: The adversary computes test accuracy on both S0test ∼ G0 (D) and S1test ∼ G1 (D) and predicts b̂ = I[acc(M, S0test ) < acc(M, S1test )]. In this paper, we compute the adversarial accuracy gap |acc(M, S0test ) − acc(M, S1test )| which quantifies the strength of this signal; it is reported as the Adv gap columns in Table 1. • Threshold Test: The adversary predicts b̂ = I[|acc(M, S0test ) − acc(M, S1test )| ≤ τ ]. We set τ = 0.1 following (Suri and Evans 2021), i.e. gaps below 10% are considered indistinguishable. The ≤ τ ? column in Table 1 reports how many of the 10 runs fall below this threshold.
Results Table 1 and Figure 2 summarise the adversarial accuracy gap |acc(M, S0test ) − acc(M, S1test )| before and after FFt, together with the corresponding EO disparity ∆EO . We display per-run bar plots for UTKFaces and Bias in Bios in the Appendix. LSAC results are also shown in Appendix. Reading Table 1. Each row is one (dataset, sensitive attribute) pair. The Adv gap columns report the adversarial accuracy gap under the loss test (Suri and Evans 2021): how much more accurately the adversary can classify samples from one group than the other, averaged over 10 random seeds. A large gap means the model’s behaviour leaks which demographic group it was trained on; a gap below τ = 0.1 is considered undetectable by the threshold test. The ∆EO columns report the equalized-odds disparity of the same model on the same test split. The Bound holds? column is the central empirical check for Theorem 1: it verifies that Adv(A, Mf ) ≤ ∆fft EO holds after fine-tuning (with W = 1 throughout, since all settings use the biased distribution protocol where G0 and G1 are pure demographic groups). Three major findings stand out in Table 1: (1) The theoretical bound holds in every row. Across all settings, modalities (tabular, image, NLP), and sensitive attributes (sex and race), the post-FFt adversarial gap never exceeds the post-FFt EO disparity. The uniform empirical verification strongly supports Theorem 1. (2) FFt consistently controls adversarial leakage, with one exception. In five of six settings, the adversarial gap decreases after FFt: reductions range from 1.6% (UTKFaces) to 11.8% (ACS sex). COMPAS is the exception: the gap increases slightly from 2.0% to 3.4%. We are transparent about this: in COMPAS, the baseline has near-zero distributional leakage to begin with, and the EO penalty redistributes prediction errors across groups in a way that marginally widens the accuracy gap without crossing τ (all 10 seeds remain below τ ). The Appendix base-rate floor analysis explains the mechanism: when |∆π| is small and EO enforcement shifts the accuracy-EO decoupling regime, the accuracy gap can rise even as the PPR gap falls. Crucially, the EO penalty still reduces ∆EO from 37.5% to 15.4%, tightening the formal guarantee of Theorem 1 by 2.4× even in this setting. (3) Rehearsal-based FFt reliably drives the gap below τ . Across all settings in the main table (ACS, COMPAS, German Credit, Bios, UTKFaces), rehearsal-based FFt brings the gap below or near τ in the majority of runs, consistent with the practical guideline in Section . ACS Income (sex and race) FFt with the EO penalty reduces the gap to 2.5% (sex) and 3.7% (race), below τ in all 10 runs. The EO penalty drives ∆EO down dramatically — from ∼ 78% at baseline to 4.3% (sex) and 5.5% (race) after FFt — making the bound Adv ≤ ∆EO nearly tight. COMPAS (race) The Baseline gap is small (∼ 2.0%), which is already below the threshold τ , because AfricanAmerican and Caucasian defendants in this dataset have similar recidivism rates after standard filtering. This is a case where the adversary cannot reliably distinguish the two
ACS Income CA-2018 (sex (M F))
Baseline vs FFt+EO ( =1.0)
ACS Income CA-2018 (race (W NW)) = 0.1 Baseline FFt+EO
0.14
Adversarial Accuracy Gap
Adversarial Accuracy Gap
0.08 0.06 0.04
0.12 0.10 0.08 0.06 0.04
0.02 0.00
= 0.1 Baseline FFt+EO
0.14
0.12 0.10
Baseline vs FFt+EO ( =1.0)
0.16
0.02 0
2
4
6
Run (seed)
0.00
8
0
(a) ACS (sex: male → female)
2
4
Run (seed)
6
8
(b) ACS (race: white → non-white)
Figure 2: Adversarial accuracy gap for ACS Income CA-2018 across 10 runs. Rehearsal-based FFt brings the gap below τ = 0.1 in all 10 runs for both attributes. COMPAS (race: AA Cau)
Baseline vs FFt+EO ( =1.0)
German Credit (sex: M F)
0.10
0.30 Adversarial Accuracy Gap
Adversarial Accuracy Gap
= 0.1 Baseline FFt+EO
0.04 0.02 0.00
= 0.1 Baseline FFt+EO
0.35
0.08 0.06
Baseline vs FFt+EO ( =1.0)
0.25 0.20 0.15 0.10 0.05
0
2
4
6
Run (seed)
0.00
8
(a) COMPAS (race: African-American → Caucasian)
0
2
4
Run (seed)
6
8
(b) German Credit (sex: male → female)
Figure 3: Adversarial accuracy gap for COMPAS and German Credit across 10 runs. COMPAS gaps are below τ in all runs; German Credit drops below τ in 8/10 runs. Table 1: Adversarial accuracy gap and EO disparity (mean over 10 runs). W = 1 for all settings (biased protocol). The bound Adv ≤ ∆EO is empirically verified in every row. Setting (G0 → G1 )
Adv gap
∆EO
≤ τ?
Bound holds?
4.3% 5.5%
✓ all ✓ all
✓ ✓
37.5% 28.8%
15.4% 17.3%
✓ all 8/10
✓ ✓
17.7% 9.1%
15.5% 8.3%
✓ all ✓ all
✓ ✓
Base
FFt
Base
FFt
ACS sex (M → F) ACS race (W → NW)
14.3% 15.3%
2.5% 3.7%
78.0% 78.6%
COMPAS race (AA → Cau) German sex (M → F)
2.0% 14.0%
3.4% 6.0%
UTKFaces race (W → NW) Bios sex (M → F)
7.1% 5.2%
5.5% 0.9%
groups even before any defense is applied. Rehearsal-based FFt preserves this protection at ∼ 3.4% (below τ in all 10 runs); the small absolute increase is within run-to-run variance and does not reflect a degradation in security. Crucially, the EO penalty substantially reduces ∆EO from ∼ 37.5% to ∼ 15.4%, tightening the theoretical bound of Theorem 1 by a factor of 2.4× and providing a stronger formal guarantee even when the empirical gap was already negligible. German Credit (sex) The Baseline gap is ∼ 14.0%: a model trained on male applicants classifies female applicants less accurately, reflecting genuine distributional differences in the German Credit data. Rehearsal-based FFt reduces the gap to ∼ 6.0%, below τ in 8 of 10 runs. The 2.2:1
size asymmetry (690 male vs. 310 female) and the small dataset (n = 1000) limit full convergence, consistent with the practical guideline in Section . UTKFaces (race) The Baseline gap is ∼ 7.1%: a ResNet18 trained exclusively on White faces generalises poorly to non-White faces whose age-distribution statistics differ. FFt reduces the gap to ∼ 5.5%, with both values below τ in all 10 runs. The primary contribution of this setting is breadth: it demonstrates that the Adv–EO bound (Theorem 1) holds for a deep convolutional model fine-tuned end-to-end, not just tabular MLPs, and that rehearsal (ρ = 0.2) prevents the ResNet’s larger parameter space from catastrophically forgetting the White-face distribution. The ∆EO reduction from 17.7% to 15.5% is more modest than in tabular settings, consistent with the tighter baseline gap leaving less room for EO improvement. Bias in Bios (sex) The Baseline gap is ∼ 5.2%: an MLP trained on male biographies (TF-IDF features, genderscrubbed text) predicts occupation less accurately on female biographies, reflecting the well-documented gender skew in Bias in Bios (De-Arteaga et al. 2019). Rehearsal-based FFt reduces the gap to ∼ 0.9%, below τ in all 10 runs. This extends FFt to NLP classification and shows that the defence is not limited to tabular tasks.
Effect of W = |2 1| (ACS sex)
Effect of ( = 1.0 fixed)
17.5
= 0.1 default
60
EO
15.0 12.5 10.0 7.5 5.0
EO
= 0.1 default
50 40 30 20
0.0
0.1
0.5 1.0 (EO penalty weight)
2.0
5.0
(a) Effect of λ (ρ = 0.2 fixed). λ = 5.0 overcorrects.
0
12 10 8 6 4 2
10
2.5
= 0.1 Baseline FFt
14
Adv gap
Adversarial gap (%)
Adv gap
Gap / EO (%)
Gap / EO (%)
Effect of ( = 0.2 fixed) 20.0
0 0.0
0.05
0.1 0.2 (rehearsal fraction)
0.3
0.5
(b) Effect of ρ (λ = 1.0 fixed). Any ρ ≥ 0.05 prevents forgetting.
0.5 (W=0.0)
0.6 (W=0.2)
0.7 (W=0.4)
0.8 (W=0.6)
Male fraction in 0
0.9 (W=0.8)
1.0 (W=1.0)
(c) Effect of W (mixed distribution, λ = 1.0, ρ = 0.2). Sharp jump at α = 1.0.
Figure 4: Ablation on ACS sex (10 seeds each). Blue = adversarial gap; orange = ∆EO (4a, 4b); dotted red = τ = 0.1; dashed gray = default.
Discussion. Our results are consistent with Theorem 2 and show a clear pattern across all evaluated settings. Why rehearsal helps. Plain warm-start FFt can cause catastrophic forgetting — the fine-tuning step adapts the model so strongly to G1 that G0 accuracy collapses, increasing ∆fft EO rather than reducing it. Adding a rehearsal component (20% of G0 data mixed into fine-tuning) prevents this collapse while still closing the gap: ACS achieves 2.5% (sex) and 3.7% (race), COMPAS 3.4% (race), and German Credit 6.0% (sex) — all below τ in the majority of runs. Practical guideline. Use rehearsal-based FFt (mixing ∼ 20% of G0 into fine-tuning) to prevent catastrophic forgetting while closing the gap. Verify that |S1 |/|S0 | is not too small; if the minority group is very small (e.g. < 20% of G0 ), consider supplementing with synthetic data for G1 . We also perform ablations on the sensitivity to λ, ρ, and the distributional shift W , as shown in Figure 4. Adversary strength. Our experiments evaluate the Loss Test adversary (Suri and Evans 2021), who observes only the model’s test accuracy on each group. This is the natural adversary for the black-box setting. More powerful adversaries such as meta-classifiers trained on model weight snapshots (Suri and Evans 2021) or output-distribution statistics (Suri et al. 2023) can extract signals beyond the PPR gap. We do not evaluate against meta-classifiers and acknowledge this as a limitation. Nevertheless, there is a structural reason to expect that EO enforcement degrades meta-classifier performance as well: meta-classifiers succeed by exploiting systematic differences in a model’s group-conditional prediction patterns—precisely the signal that EO minimises. A model with ∆EO ≈ 0 produces nearly identical conditional prediction rates for G0 and G1 , removing the demographic signature that weight-space or logitspace meta-classifiers rely on. Formally extending Theorem 1 to meta-classifier adversaries is an important open question we leave to future work. The Loss Test is the standard black-box evaluation in the DIA literature and is directly comparable to prior work.
Ablation: Hyperparameters λ, ρ, and Distributional Shift W We ablate the two key hyperparameters of rehearsal-based FFt and the distributional shift weight W on the ACS sex task (G0 =male, G1 =female), running 10 independent seeds per configuration. Effect of λ (Figure 4a, ρ = 0.2 fixed). λ = 0 recovers plain rehearsal without an EO penalty; the gap is reduced (4.2%) but ∆EO stays large (20.6%). Increasing λ monotonically reduces ∆EO : at our default λ = 1.0 (marked ⋆) the gap reaches 2.5% with 10/10 runs below τ . However, λ = 5.0 overcorrects, that is, the EO term dominates crossentropy, widening the accuracy gap to 10.8% and causing the bound to fail in 8 of 10 runs. Hence, we recommend choosing λ ∈ [0.5, 2.0]. Effect of ρ (Figure 4b, λ = 1.0 fixed). Without rehearsal (ρ = 0) catastrophic forgetting causes the gap to jump to 23% and ∆EO to 64.5%. Any non-zero ρ immediately prevents this: ρ = 0.05 restores the gap to 3.4% with 10/10 runs below threshold τ . Beyond ρ = 0.20 gains are marginal (2.5%→1.6% at ρ = 0.50), therefore we use ρ = 0.20 as the default throughout. Effect of distributional shift W (Figure 4c). We vary the male fraction α ∈ {0.5, . . . , 1.0} in G0 (with G1 = (1−α) male + α female), so W = |2α−1| ranges from 0 to 1. For all α ∈ [0.5, 0.9] (W ≤ 0.8), both baseline and FFt gaps are ≤ 1.6% and already below τ in all 10 runs — the adversary has little signal to exploit. At α = 1.0 (W = 1, the biased protocol), the gap jumps sharply to 13.9% for the baseline and 4.9% for FFt (8/10 below τ ). This nonlinear jump reflects the joint product ∆EO ·W : at α = 1.0 the model trained on pure males has never seen females, driving ∆base EO to ∼ 78%, while at α = 0.9 the training set already includes 10% females, keeping ∆base EO small. This figure empirically confirms Corollary 1: the biased protocol is qualitatively the hardest case, and any FFt defense that succeeds at W = 1 is guaranteed to succeed at all W < 1. Full numerical results for the λ and ρ sweeps are displayed in Appendix.
Conclusion & Future Work We introduce Fair Finetuning (FFt) with Equalized Odds (EO) as a principled defense against distribution inference attacks and develop a complete theoretical characterisation. Theorem 1 establishes the tighter bound Adv(A, Mf ) ≤ ∆EO · W , where W is the distributional shift weight (Remark 2). Under the biased distribution protocol, where W = 1, the bound reduces to Adv ≤ ∆EO , making FFt’s effectiveness directly controllable through the EO constraint. Theorem 2 characterises exactly when FFt is beneficial and predicts failure when catastrophic forgetting raises ∆fft EO above ∆base while Proposition 1 confirms the bound is tight. CorolEO lary 1 establishes the biased protocol as the worst case for the defender and Proposition 2 identifies the group-size failure regime. Empirically, we evaluate a biased distribution setup (one pure demographic group as G0 , the complementary group as G1 ) across six datasets spanning three modalities—tabular (ACS Income, COMPAS, German Credit), image (UTKFaces), and NLP (Bias in Bios, LSAC), with LSAC evaluated separately in the Appendix. Rehearsal-based FFt reduces the adversarial gap to 2.5% (ACS sex) and 3.7% (ACS race)—below τ in all 10 runs. COMPAS’s baseline gap of 2.0% remains below τ at 3.4% after FFt, with the EO penalty tightening ∆EO from 37.5% to 15.4%. German Credit drops from 14.0% to 6.0% (8/10 runs below τ ); UTKFaces from 7.1% to 5.5% (all runs); and Bias in Bios from 5.2% to 0.9% (all runs). On LSAC, FFt reduces the gap from ∼ 74% to ∼ 8.8% (sex, below τ in the majority of runs); for race (5:1 size asymmetry), FFt halves the gap (49.8% → 25.2%) but cannot close it entirely—an instance of the failure condition in Theorem 2. These results confirm that rehearsal-based FFt generalises across datasets, modalities, and sensitive attributes. A mixed-distribution ablation (Figure 4c) additionally validates Corollary 1: for α ∈ [0.5, 0.9] (i.e. W ≤ 0.8) the adversarial gap is ≤ 1.6% and already below τ , with a sharp jump to 13.9% only at α = 1.0 (W = 1), confirming that the biased protocol is the qualitatively hardest case.
Limitations 1. Data Acquisition. FFt requires sampling from the complementary distribution G¬b . While this distribution is publicly known under the standard threat model, obtaining a labeled sample from it still incurs practical cost. Future work should explore whether synthetic data generation or differentially private mechanisms can substitute for real complementary samples. 2. Attribute Awareness. The current formulation assumes the defender knows which sensitive attribute the adversary targets. In practice, an adversary could probe any undisclosed attribute (e.g. income, political affiliation). Extending FFt to simultaneously enforce EO across multiple sensitive attributes—or to use attribute-agnostic representation learning—is an important direction (a theoretical extension to multiple attributes is given in the Appendix). 3. Group Size Asymmetry. When |G1 | ≪ |G0 |, fine-tuning on G1 cannot fully recalibrate the model (LSAC race).
Complementary approaches such as synthetic oversampling of G1 or importance-weighted EO constraints should be explored. 4. Dataset Scope. While our evaluation covers tabular, image, and NLP modalities across six datasets, it is limited to binary classification tasks with binary group membership. Extending FFt to multi-class sensitive attributes, continuous protected attributes, and larger-scale visionlanguage models remains important future work. 5. Adversary Strength. Experiments evaluate the blackbox Loss Test adversary. Meta-classifier adversaries that operate on model weight distributions (Suri and Evans 2021) may extract signals beyond the PPR gap; we provide a structural argument for why EO enforcement should degrade their performance (Section ), but a formal extension of Theorem 1 to meta-classifiers and empirical validation remain important open questions. 6. Comparison to Existing Defenses. We include a direct comparison to DP-SGD (results in Appendix). Direct comparison to property unlearning (Stock et al. 2023) and Inf2Guard (Noorbakhsh et al. 2024) is left as future work; we anticipate FFt will be complementary, since it is orthogonal (fairness-based post-processing, no cryptographic overhead) and the two defenses target different leakage surfaces.
Future Directions Future work may explore how fairness constraints can be embedded directly in the training objective rather than applied as post-hoc fine-tuning, whether the Adv–EO bound of Theorem 1 can be tightened for specific model families, and how rehearsal-based FFt interacts with differential privacy guarantees.
Impact Statement Ethical Considerations. All datasets used in this work (ACS Income, COMPAS, German Credit, UTKFaces, Bias in Bios, and LSAC) are publicly available benchmarks; no new data was collected and no human subjects were recruited, so IRB review was not required. That said, several datasets carry ethical weight that warrants acknowledgement. COMPAS is a commercial recidivism risk-score instrument with well-documented racial disparities (Angwin et al. 2016) and is used in real criminal-sentencing contexts; treating it purely as a benchmark normalizes its deployment and may implicitly lend credibility to its continued use. UTKFaces encodes race and gender through thirdparty annotation, embedding socially constructed categories as ground-truth labels. Across all datasets we operationalize race and sex as binary attributes, which reflects the limitations of existing benchmarks rather than a normative claim about identity, but risks invisibilizing non-binary and multiracial individuals in the evaluation. We flag this as a known gap that future work should address by sourcing datasets with richer demographic representation. Researcher Positionality. This work sits at the intersection of ML security and algorithmic fairness, and the author’s training in both areas shapes several design choices.
Framing the problem as a cryptographic game (adversarial advantage, bit-guessing) reflects a security-first lens that defines harm as information leakage and success as bounding an adversary’s distinguishing advantage. This framing is powerful for formal guarantees but may underweight harms that are harder to quantify, such as the degrading effects of surveillance or the symbolic harm of demographic profiling. The choice of Equalized Odds as the operative fairness criterion reflects a group-statistical view of fairness; individual or causal notions of fairness would yield different bounds and potentially different empirical conclusions. The author approaches the problem with the assumption that fairness and privacy are complementary objectives — a perspective that motivates the central thesis but may occlude settings where reducing distributional leakage conflicts with other fairness criteria (e.g. individual fairness or counterfactual fairness). Adverse Impact. The primary intended use of FFt is defensive: reducing an adversary’s ability to infer demographic composition from a deployed model. However, several misuse and misapplication risks deserve explicit attention. First, the defense could engender false confidence if applied without checking the preconditions established in Theorems 1 and 2: a practitioner who applies FFt with severe group-size asymmetry (α ≪ 0.2) or an over-large EO penalty (λ ≫ 2) may produce a model whose adversarial gap remains high or whose accuracy-EO coupling breaks, yet who reports the deployment as “protected.” Second, FFt could be exploited as a form of fairness-washing: an organization could apply a minimal EO constraint (λ just large enough to reduce the measured adversarial gap below τ ) while the underlying model continues to encode discriminatory representations along unmeasured attributes. The bound Adv ≤ ∆EO applies specifically to the positive-prediction-rate gap under black-box access; it does not protect against white-box attacks, side-channel leakage via confidence scores or latency, or inference about sensitive attributes not included in the EO constraint. Deployers should treat the bound as a guarantee about one specific attack surface, not a blanket privacy certificate.
References Abadi, M.; Chu, A.; Goodfellow, I.; McMahan, H. B.; Mironov, I.; Talwar, K.; and Zhang, L. 2016. Deep Learning with Differential Privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS ’16, 308–318. New York, NY, USA: Association for Computing Machinery. ISBN 9781450341394. Angwin, J.; Larson, J.; Mattu, S.; and Kirchner, L. 2016. Machine Bias. ProPublica. Chang, H.; and Shokri, R. 2021. On the Privacy Risks of Algorithmic Fairness. In IEEE European Symposium on Security and Privacy (EuroS&P), 292–303. Chaudhuri, K.; Monteleoni, C.; and Sarwate, A. D. 2011. Differentially Private Empirical Risk Minimization. J. Mach. Learn. Res., 12(null): 1069–1109. Cormode, G.; Jha, S.; Kulkarni, T.; Li, N.; Srivastava, D.; and Wang, T. 2018. Privacy at Scale: Local Differential
Privacy in Practice. In Proceedings of the 2018 International Conference on Management of Data, SIGMOD ’18, 1655–1658. New York, NY, USA: Association for Computing Machinery. ISBN 9781450347037. De-Arteaga, M.; Romanov, A.; Wallach, H.; Chayes, J.; Borgs, C.; Chouldechova, A.; Geyik, S.; Kenthapadi, K.; and Kalai, A. T. 2019. Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAccT), 120–128. Ding, F.; Hardt, M.; Miller, J.; and Schmidt, L. 2021. Retiring Adult: New Datasets for Fair Machine Learning. In Advances in Neural Information Processing Systems, volume 34, 6478–6490. Dua, D.; and Graff, C. 2019. UCI Machine Learning Repository. Dwork, C.; McSherry, F.; Nissim, K.; and Smith, A. 2006. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference on Theory of Cryptography, TCC’06, 265–284. Berlin, Heidelberg: SpringerVerlag. ISBN 3540327312. Hardt, M.; Price, E.; and Srebro, N. 2016. Equality of opportunity in supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, 3323–3331. Red Hook, NY, USA: Curran Associates Inc. ISBN 9781510838819. Hartmann, V.; Meynent, L.; Peyrard, M.; Dimitriadis, D.; Tople, S.; and West, R. 2023. Distribution Inference Risks: Identifying and Mitigating Sources of Leakage. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 136–149. Hu, H.; Salcic, Z.; Sun, L.; Dobbie, G.; Yu, P. S.; and Zhang, X. 2022. Membership Inference Attacks on Machine Learning: A Survey. ACM Comput. Surv., 54(11s). Kifer, D.; and Machanavajjhala, A. 2012. A Rigorous and Customizable Framework for Privacy. In Proceedings of the 31st ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems (PODS), 77–88. Mahloujifar, S.; Ghosh, E.; and Chase, M. 2022. Property Inference from Poisoning. In 2022 IEEE Symposium on Security and Privacy (SP), 1120–1137. Noorbakhsh, S. L.; Zhang, B.; Hong, Y.; and Wang, B. 2024. Inf2Guard: An Information-Theoretic Framework for Learning Privacy-Preserving Representations against Inference Attacks. In 33rd USENIX Security Symposium (USENIX Security 24), 2405–2422. Philadelphia, PA: USENIX Association. ISBN 978-1-939133-44-1. Salem, A.; Cherubin, G.; Evans, D.; Kopf, B.; Paverd, A.; Suri, A.; Tople, S.; and Zanella-Beguelin, S. 2023. SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning . In 2023 IEEE Symposium on Security and Privacy (SP), 327–345. Los Alamitos, CA, USA: IEEE Computer Society. Shokri, R.; Stronati, M.; Song, C.; and Shmatikov, V. 2017. Membership Inference Attacks Against Machine Learning
Models . In 2017 IEEE Symposium on Security and Privacy (SP), 3–18. Los Alamitos, CA, USA: IEEE Computer Society. Stock, J.; Wettlaufer, J.; Demmler, D.; and Federrath, H. 2023. Lessons Learned: Defending Against Property Inference Attacks. In Proceedings of the 20th International Conference on Security and Cryptography - SECRYPT, 312– 323. INSTICC, SciTePress. ISBN 978-989-758-666-8. Suri, A.; and Evans, D. 2021. Formalizing and Estimating Distribution Inference Risks. ArXiv, abs/2109.06024. Suri, A.; Lu, Y.; Chen, Y.; and Evans, D. 2023. Dissecting Distribution Inference . In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 150–164. Los Alamitos, CA, USA: IEEE Computer Society. Tran, C.; Fioretto, F.; Kim, J.-E.; and Naidu, R. 2022. Pruning has a disparate impact on model accuracy. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22. Red Hook, NY, USA: Curran Associates Inc. ISBN 9781713871088. Wightman, L. F. 1998. LSAC National Longitudinal Bar Passage Study. Research report, Law School Admission Council. Xu, Y.; Yin, M.; Fang, M.; and Gong, N. Z. 2024. Robust Federated Learning Mitigates Client-side Training Data Distribution Inference Attacks. In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, 798–801. New York, NY, USA: Association for Computing Machinery. ISBN 9798400701726. Yeom, S.; Giacomelli, I.; Fredrikson, M.; and Jha, S. 2018. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting . In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), 268–282. Los Alamitos, CA, USA: IEEE Computer Society. Yousefpour, A.; Shilov, I.; Sablayrolles, A.; Testuggine, D.; Prasad, K.; Malek, M.; Nguyen, J.; Ghosh, S.; Bharadwaj, A.; Zhao, J.; Fernandes, G.; Foley, S.; and Annamalai, A. 2021. Opacus: User-Friendly Differential Privacy Library in PyTorch. arXiv preprint arXiv:2109.12298. Yu, D.; Zhang, H.; Huang, Y.; and Xie, Z. 2025. Data distribution inference attack in federated learning via reinforcement learning support. High-Confidence Computing, 5(1): 100235. Zhang, W.; Ohrimenko, O.; and Cummings, R. 2022. Attribute Privacy: Framework and Mechanisms. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, 757–766. New York, NY, USA: Association for Computing Machinery. ISBN 9781450393522. Zhang, X.; Chen, C.; Xie, Y.; Chen, X.; Zhang, J.; and Xiang, Y. 2021. Privacy Inference Attacks and Defenses in Cloud-based Deep Neural Network: A Survey. ArXiv, abs/2105.06300. Zhang, Z.; Song, Y.; and Qi, H. 2017. Age Progression/Regression by Conditional Adversarial Autoencoder. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Appendix Extension to Multiple Sensitive Attributes The single-attribute bound of Theorem 1 extends directly to K simultaneous sensitive attributes by applying the proof independently per attribute. Theorem 3 (Multi-attribute Extension). Suppose Mf satis(k) fies EO with disparity ∆EO for each of K sensitive attributes A1 , . . . , AK , enforced simultaneously. Then for any adversary targeting attribute Ak : (k)
Adv(AAk , Mf ) ≤ ∆EO · W (k) , where W (k) is defined as in Theorem 1 with respect to Ak . The bounds across attributes are independent: suppressing leakage about Ak does not require sacrificing protection for Ak ′ . Proof sketch. Theorem 1 is proved for a single attribute by conditioning on A. The proof is attribute-specific: the can(k) cellation ∆y = δy ∆Py depends only on Py,a (the model’s prediction rates conditioned on Ak ). Applying the same argument independently for each k yields the stated bounds. (k) Simultaneous EO enforcement on K attributes sets |δy | ≤ (k) ∆EO for every k, completing the proof.
Proof of the Identity ∆Py,0 = −∆Py,1 For any fixed value of the true label Y = y and distribution D, the sensitive attribute A ∈ {0, 1} satisfies Pr[A = 0 | Y = y, D] + Pr[A = 1 | Y = y, D] = 1. Define the distributional differences ∆Py,a := Pr[A = a | Y = y, D1 ]−Pr[A = a | Y = y, D0 ]. Applying the above identity to both D1 and D0 gives Pr[A = 0 | Y = y, D1 ] + Pr[A = 1 | Y = y, D1 ] = 1, Pr[A = 0 | Y = y, D0 ] + Pr[A = 1 | Y = y, D0 ] = 1. Subtracting the first equation from the second yields Pr[A = 0 | Y = y, D1 ] − Pr[A = 0 | Y = y, D0 ] + Pr[A = 1 | Y = y, D1 ]−Pr[A = 1 | Y = y, D0 ] = 0, which is equivalent to ∆Py,0 + ∆Py,1 = 0. Thus, ∆Py,0 = −∆Py,1 . In other words, because A is binary and its conditional probabilities must sum to one under each distribution, i.e., any increase in the probability of A = 1 among individuals with label Y = y under D1 must produce an equal decrease in the probability of A = 0. Hence the two changes are exact opposites. It is therefore convenient to define ∆Py := ∆Py,1 ,
so that
∆Py,0 = −∆Py .
Scope of the PPR Bound and the Base-Rate Floor Theorem 1 bounds the positive-prediction-rate (PPR) gap | Pr[Ŷ = 1 | D1 ] − Pr[Ŷ = 1 | D0 ]|. The Loss Test adversary, however, observes the accuracy gap |acc(M, S0test ) − acc(M, S1test )|. This section derives the exact relationship between the two and explains the base-rate floor that appears when ∆EO = 0. Accuracy in terms of TPR and FPR. For a binary classifier on distribution D with positive base rate π = Pr[Y = 1 | D]: acc(M, D) = π · TPR(D) + (1 − π) · (1 − FPR(D)), where TPR(D) = Pr[Ŷ = 1 | Y = 1, D] and FPR(D) = Pr[Ŷ = 1 | Y = 0, D]. Accuracy gap when EO is perfectly enforced. Suppose ∆EO = 0, so that TPR and FPR are identical across groups: TPR(D0 ) = TPR(D1 ) = TPR and FPR(D0 ) = FPR(D1 ) = FPR. Let ∆acc = acc(M, D1 ) − acc(M, D0 ). Then: ∆acc = π1 · TPR + (1 − π1 )(1 − FPR) − π0 · TPR − (1 − π0 )(1 − FPR) = (π1 − π0 ) · TPR + FPR − 1 = ∆π · TPR + FPR − 1 .
(6)
where ∆π = π1 − π0 = Pr[Y = 1 | D1 ] − Pr[Y = 1 | D0 ] is the label base-rate difference between groups. This quantity is the base-rate floor: the accuracy gap that persists even after perfect EO enforcement, arising purely from the two groups having different label prevalences. It is zero only when ∆π = 0 (equal base rates) or when TPR + FPR = 1 (the model is no better than random). Interpretation. Theorem 1 bounds the PPR-driven component of adversarial distinguishability; the base-rate floor is a separate, EO-independent contribution. When groups have similar label distributions (|∆π| small), the floor is negligible and the bound is informative for the accuracy gap as well. When base rates diverge sharply, the floor dominates even at ∆EO = 0, which is precisely the mechanism behind the λ = 5.0 failure (Figure 4a) and the LSAC race partial failure (shown below). Meta-classifier adversaries. Adversaries more powerful than the Loss Test (e.g. meta-classifiers trained on model weight snapshots (Suri and Evans 2021)) may extract signals beyond both the PPR gap and the accuracy gap. Bounding their advantage requires tighter coupling with the model’s full output distribution, which we leave as future work.
LSAC Results (Near-Saturated Regime) The LSAC bar-passage dataset (Wightman 1998) contains records for 20,800 law-school students. Features include LSAT score, undergraduate GPA, family income, age, gender, race, law-school cluster (1–6), and full/part-time status. The binary target is bar passage. We experiment on two sensitive attributes: race (white vs. non-white) and sex (male vs. female). LSAC represents the near-saturated regime: the MLP trained on G0 achieves ∼ 99.9% accuracy, causing ∆EO ≈ 1. Plain warm-start FFt (without rehearsal) is used here; rehearsal would preserve the near-perfect G0 accuracy and keep the gap large. Table 2: LSAC adversarial gap and EO disparity (mean over 10 runs, W = 1). The bound Adv ≤ ∆EO holds in both rows. Adv gap
Setting (G0→G1) LSAC sex (M → F) LSAC race (W → NW)
∆EO
Base
FFt
Base
FFt
74.3% 49.8%
8.8% 25.2%
99.9% 99.9%
11.8% 68.7%
≤ τ?
Bound holds?
6/10 ✗
✓ ✓
LSAC (sex). The Baseline gap is very large (∼ 74%): a model trained exclusively on male students generalises poorly to female students whose bar-passage rates differ significantly. Plain warm-start FFt reduces the gap to ∼ 8.8%, below τ in 6 of 10 runs. LSAC (race). The Baseline gap is ∼ 50%: a model trained on white students performs near-randomly on non-white students. FFt reduces the gap to ∼ 25% — a substantial improvement, but still above τ . The 5.3:1 size asymmetry (17,493 white vs. 3,307 non-white) combined with a 33percentage-point label-rate difference (whites ∼ 85% vs. non-whites ∼ 52% bar passage) prevents full recalibration.
Why ∆EO ≈ 1 in the Near-Saturated Regime The ∆EO ≈ 99.9% entries in the LSAC rows of Table 2 may appear surprising, but they follow directly from the definition. A model that achieves near-perfect accuracy (∼ 99.9%) on S0 has calibrated its conditional prediction rates almost exactly to the G0 distribution: P (Ŷ = 1 | Y = 1, G0 ) ≈ 1 (near-perfect true-positive rate) and P (Ŷ = 1 | Y = 0, G0 ) ≈ 0 (near-perfect true-negative rate). When this model is evaluated on S1 (a group it never saw during training), it imposes its G0 -fitted decision boundary on G1 ’s feature distribution. Because G1 has systematically different feature patterns (different LSAT/GPA profiles for nonwhite students; different score ranges for female students), the G0 -trained boundary classifies most G1 samples incorrectly, yielding a near-zero TPR on G1 . The EO disparity is therefore ∆EO = max P (Ŷ = 1 | Y = y, G0 ) y
Distribution Inference Attack Game The original distribution inference attack game as introduced by (Suri and Evans 2021) is displayed in Figure 6.
− P (Ŷ = 1 | Y = y, G1 ) ≈ |1 − 0| ≈ 1.
(7)
LSAC (sex: G0=male, G1=female)
0.8
Baseline vs FFt
LSAC (race: G0=white, G1=non-white) = 0.1 Baseline FFt
0.6
Adversarial Accuracy Gap
Adversarial Accuracy Gap
0.7
0.5 0.4 0.3 0.2
= 0.1 Baseline FFt
0.4 0.3 0.2 0.1
0.1 0.0
Baseline vs FFt
0.5
0
2
4
Run (seed)
6
8
0.0
(a) LSAC (sex: male → female)
0
2
4
Run (seed)
6
8
(b) LSAC (race: white → non-white)
Figure 5: Adversarial accuracy gap for LSAC (mean over 10 runs, blue=Baseline, orange=FFt). FFt reduces the sex gap below τ = 0.1 in the majority of runs. For race, the 5:1 group-size asymmetry limits FFt’s effectiveness. This is not a pathological result. In fact, it is the expected consequence of exclusive single-group training: a model that is maximally accurate for G0 is simultaneously maximally unfair toward G1 by the EO metric. The adversarial gaps (74.3% for sex; 49.8% for race) are correctly bounded below ∆EO = 99.9%, consistent with Theorem 1; the bound is simply loose here because the adversary’s observable signal (accuracy difference) is a coarser measure than population-level EO disparity. After FFt, ∆EO falls dramatically to 11.8% (sex) and 68.7% (race). Theorem 1 then guarantees that the new adversarial advantage is bounded by these lower values and this is empirically confirmed by the post-FFt gaps (8.8% and 25.2%). For LSAC sex the post-FFt bound is nearly tight (8.8% ≤ 11.8%), illustrating that the residual leakage is almost entirely explained by the residual unfairness δy , which is the sole surviving term in the proof’s algebraic cancellation.
Utility Impact Table 3 reports G0 test accuracy before and after FFt. The utility cost tracks how aggressively the EO constraint must be enforced: ACS sex incurs a ∼4% drop (79.6%→75.7%), while ACS race incurs a larger ∼9% drop (79.7%→71.1%), reflecting the stronger weight updates needed to close a bigger EO gap (78.6%→5.5% vs. 78.0%→4.3% for sex). COMPAS loses ∼4% (68.2%→63.8%). German Credit is the outlier: accuracy barely changes (70.3%→69.7%, ≤ 1% delta). On this 1,000-sample dataset the model converges in fewer iterations and rehearsal is especially effective at preserving G0 accuracy, since even a small rehearsal buffer (ρ=0.2) covers a proportionally large fraction of the training distribution. Across all settings the utility cost is within the range routinely accepted for fairness interventions (Hardt, Price, and Srebro 2016), and it is directly offset by the EO reductions shown in Table 1 that close the adversarial gap.
Ablation Hyperparameter Tables The hyperparameters for the ablation study is shown in Tables 4 and 5.
Table 3: G0 test accuracy before and after FFt (mean, 10 runs), for the four tabular/structured settings where architectures are comparable. UTKFaces (ResNet-18) and Bias in Bios (TF-IDF+MLP) use different model families; their per-epoch accuracy curves are not directly comparable to the MLP training curve and are omitted here.
Setting (G0 → G1)
Baseline
FFt
ACS sex (M → F) ACS race (W → NW) COMPAS race (AA → Cau) German sex (M → F)
79.6% 79.7% 68.2% 70.3%
75.7% 71.1% 63.8% 69.7%
Table 4: λ ablation (ρ = 0.2 fixed, ACS sex, 10 seeds). ⋆ = default used in all main experiments.
λ
Adv gap
∆EO
≤ τ?
Bound?
0.0 0.1 0.5 1.0⋆ 2.0 5.0
4.2% 3.2% 2.8% 2.5% 4.0% 10.8%
20.6% 12.7% 4.1% 4.3% 4.1% 1.3%
10/10 10/10 10/10 10/10 10/10 2/10
✓ ✓ ✓ ✓ ✓ ✗
DP-SGD vs. FFt: Full Results Table 6 reports the full numerical comparison between DP-SGD and FFt on ACS Income 2018 (California), sex split (G0 =male, G1 =female), averaged over 10 independent seeds. DP-SGD is trained from scratch on G0 using Opacus (Yousefpour et al. 2021) with δ=10−5 , C=1.0, and batch size 256. FFt warm-starts from the Baseline and finetunes on S1 with EO penalty λ=1 and rehearsal fraction ρ=0.2, identical to the main experiments. DP-SGD fails to reduce the adversarial gap at any privacy budget: the gap remains 13–14% regardless of ε, and all 0/10 runs fall below τ . This is structurally expected: DP-
Trainer T
Adversary A
2:
b ←$ {0, 1} S ∼ Gb (D)
3:
M ←−−− S
1:
train
M
4:
b̂ = H(M )
5:
Figure 6: The traditional distribution inference attack setting, as described in (Suri and Evans 2021). Step 2 involves sampling a dataset S from the function Gb (where b is the selected bit from Step 1). Then, Step 3 involves training the model on this sampled dataset S. Finally, this trained model is released to the public and the adversary applies a hypothesis H(·) to recover the training distribution via computing b̂. Table 5: ρ ablation (λ = 1.0 fixed, ACS sex, 10 seeds). ⋆ = default used in all main experiments.
ρ
Adv gap
∆EO
≤ τ?
Bound?
0.00 0.05 0.10 0.20⋆ 0.30 0.50
23.0% 3.4% 3.1% 2.5% 2.4% 1.6%
64.5% 6.5% 4.8% 4.3% 3.6% 3.2%
0/10 10/10 10/10 10/10 10/10 10/10
✓ ✓ ✓ ✓ ✓ ✓
Table 6: DP-SGD vs. FFt on ACS sex (mean ± 10 seeds). τ = 0.1. Adv gap ↓
∆EO ↓
S0 acc
≤τ ?
Baseline DP-SGD ε=1 DP-SGD ε=2 DP-SGD ε=5
14.3% 13.2% 13.6% 13.8%
78.0% 76.5% 77.9% 78.4%
79.7% 78.6% 78.9% 79.1%
0/10 0/10 0/10 0/10
FFt (ours)
2.5%
4.3%
75.7%
10/10
Method
SGD clips and noises per-sample gradients to protect individual records, but does not alter the model’s group-level behaviour. A G0 -trained model will still generalise poorly to G1 , leaving the demographic accuracy gap. Hence the Loss Test adversary’s signal here is intact. Tighter ε (stronger privacy) trades 1.1 percentage points of G0 accuracy but yields no DIA protection. FFt closes the gap to 2.5% (10/10 below τ ) because it directly minimises ∆EO , which by Theorem 1 (W = 1) bounds the adversary’s advantage. The utility cost is 4.0 pp on G0 accuracy. The bound Adv ≤ ∆EO holds (2.5% ≤ 4.3%), confirming the theoretical guarantee. Since DP and FFt suppress different aspects of leakage (individual-level vs. group-level), they are orthogonal and can in principle be composed. We leave a systematic composition study to future work.
Per-Run Adversarial Gap Plots Figures 7–8 show the per-seed adversarial accuracy gap for UTKFaces and Bias in Bios. Both follow the same pattern: rehearsal-based FFt consistently reduces the gap, and the bound Adv ≤ ∆EO holds in every run.
UTKFaces (race: White Non-White)
Baseline vs FFt = 0.1 Baseline FFt
0.10
Adversarial Accuracy Gap
0.08 0.06 0.04 0.02 0.00
0
2
4
Run (seed)
6
8
Figure 7: Adversarial accuracy gap for UTKFaces (race: White → Non-White, mean over 10 runs).
Bias in Bios (sex: M F)
Baseline vs FFt+EO ( =1.0) = 0.1 Baseline FFt+EO
0.10
Adversarial Accuracy Gap
0.08 0.06 0.04 0.02 0.00
0
2
4
Run (seed)
6
8
Figure 8: Adversarial accuracy gap for Bias in Bios (sex: Male → Female, mean over 10 runs).