arXiv CSopen access

FairBED: A Bayesian Experimental Design Approach to Gathering Fairer Data

FairBED: A Bayesian Experimental Design Approach to Gathering Fairer Data

arXiv:2606.23515v1 [stat.ML] 22 Jun 2026

Marcel Hedman∗ Department of Statistics University of Oxford Brieuc Lehmann Department of Statistical Science University College London

Emily Alger CeBAM, Nuffield Department of Medicine University of Oxford Chris Holmes Department of Statistics University of Oxford

Tom Rainforth Department of Statistics University of Oxford

Abstract Frameworks for ensuring fairness in machine learning typically focus on learning fair models from existing data. But this endeavor is often undermined by biases already present in that data. We therefore look to modify the data acquisition process itself to help gather fairer data that is inherently more suitable for training fair predictors. To this end, we introduce FairBED, which provides novel formulations for quantifying the fairness of datasets themselves based on the idea that fair datasets should be uninformative about sensitive attributes. We then use this to construct practical fairness-aware Bayesian experimental design (BED) objectives that maximize expected information gain about the target quantity of interest while minimizing expected information gain about sensitive attributes. We further derive a theoretical link between FairBED and demographic parity, and show empirically that models trained on data gathered using FairBED provide improved fairnessaccuracy trade-offs compared to randomly acquired data and conventional BED.

1

Introduction

In impactful domains such as healthcare [75], online advertising [72], and criminal justice [20], training machine learning algorithms on datasets perpetuating societal biases can lead to unfair predictions and decision-making with significant negative consequences. Various downstream mitigation strategies have been introduced to try and address this, such as data reweighting [18, 48], constrained optimization during training [79], and pre-processing or “data-repair” methods [16, 26]. However, to date, the vast majority of such work has been devoted to intervening in how pre-existing datasets are modeled, with comparatively little attention paid to the data-acquisition process itself [35, 69, 71]. This represents a major shortfall in the fairness arsenal, as data that is inherently unfair and already encodes unwanted societal biases will at best hinder, and at worst completely undermine, our ability to use such data to produce fair models and decisions [14]. Thus, we should look to guide the data acquisition process toward producing fairer data from which we can learn fairer models. Importantly, this should be done in a way that ensures the data is still as informative as possible for the task at hand. To this end, we introduce FairBED, a Bayesian experimental design (BED, [19, 51, 52, 63]) approach that formalizes the utility of data in a way that balances the needs of effective learning and fairness. Namely, we first introduce two novel measures on the fairness of datasets (as opposed to the fairness of a given model, which existing work generally focuses on) that are based on how much information the data leaks about sensitive factors that we wish to remain invariant to (e.g., demographic identifiers ∗ [email protected]

Preprint.

or site-specific factors). We then construct BED objectives for optimizing design decisions which trade off these fairness metrics with the informativeness of the data about target variables of interest. The core idea is thus to make design decisions in a way that intentionally shapes the underlying data distribution to limit how much can be learned about sensitive attributes from the gathered data: if our data does not contain information about sensitive attributes, then downstream models and ML algorithms cannot exploit these attributes, naturally improving their fairness. Critically, by also maximizing the informativeness of the data for predicting our target variables of interest alongside this, FairBED ensures this shaping of the gathered data does not undermine the ability to make effective downstream predictions or generalize at test-time. While FairBED provides its own intuitive notion of fairness, we provide a formal theoretical link between it and the notion of demographic parity (DP) [22]: we show that if the data conveys no information about a sensitive variable, then models trained on this data achieve DP under certain assumptions. Empirically, we find that FairBED consistently achieves a favorable informativeness–fairness trade-off in the data it collects, with high information gain about task-relevant parameters or predictions, while substantially reducing information leakage for sensitive attributes. Moreover, we find that these benefits generalize across different models trained on the collected data, showing that FairBED is able to improve the properties of the gathered datasets themselves, not just of specific models. We also show that FairBED can be effectively combined with techniques for improving fairness at the modelling stage, improving their fairness–accuracy compared to when used on randomly sampled data. FairBED thus provides a new perspective on fairness that is directly focused on the fairness of datasets themselves, along with a principled and practical approach to fairness-aware data acquisition, opening up new research directions and deployment opportunities in the quest for algorithmic fairness.

2

Background

2.1

Bayesian Experimental Design (BED)

Bayesian experimental design (BED) [39, 51, 52, 63] is a principled approach to data gathering wherein designs ξ ∈ X are selected to maximize some expected utility of the generated data. To do this, it postulates some Bayesian model, p(θ)p(y | θ, ξ), over hypothetical possible observations, y ∈ Y, and some target variables of interest θ, which might themselves represent a real-world quantity, model parameters, or downstream predictions and decisions we wish to make. This model is then used to both simulate possible data and evaluate the utility of possible data. The latter is done by defining some utility function, U (ξ, y), that should depend on the posterior p(θ | y, ξ) ∝ p(θ)p(y | θ, ξ) [19, 67]. The optimal design is then the one that optimizes the R expected utility over possible observations: ξ ∗ = arg maxξ∈X Ep(y|ξ) [U (ξ, y)] where p(y | ξ) = p(y | θ, ξ) p(θ) dθ. A canonical choice of utility in BED is information gain (IG) about θ [51, 52], defined as the reduction in Shannon entropy from prior to posterior: IGθ (ξ, y) = H(p(θ)) − H(p(θ | ξ, y)), yielding the expected information gain (EIG) objective: EIGθ (ξ) = Ep(y|ξ) [KL (p(θ | ξ, y) ∥ p(θ))] ,

(1)

which is equivalent to the mutual information between θ and y under our chosen model. Optimizing the EIG is typically non-trivial due to its doubly intractable nature, with different estimation approaches proposed to overcome this [6, 27–29, 31, 43, 62]. Adaptive design. BED can also be extended to sequential settings, wherein it is sometimes referred to as Bayesian Adaptive Design [17, 56]. Here designs are selected conditioned on past observations. Traditionally, at round t, the design ξt is chosen to maximize the incremental EIG given the history ht−1 : h
i EIGθ (ξt | ht−1 ) = Ep(yt |ξt ,ht−1 ) KL p(θ | ht−1 , ξt , yt ) ∥ p(θ | ht−1 ) . (2) Here, ht−1 = {(ξ1 , y1 ), . . . , (ξt−1 , yt−1 )} denotes the previous designs and outcomes. This allows the design process to adapt to previous data by targeting novel information not already uncovered. More recently, policy-based Bayesian adaptive design approaches have also been proposed, that instead directly learn end-to-end design networks, πd : ht−1 7→ ξt , that map from the history to the next design [12, 29, 36, 44], typically using the total EIG across all experiment iterations as their 2

objective. Using this idea of a design policy, we can more generally talk about the EIG of a data gathering process parameterized by some πd (as opposed to a single design): EIGθ (πd ) = Ep(D|πd ) [KL (p(θ | D) ∥ p(θ))] ,

(3)

where D = hend now represents all the data gathered from the process and πd can be (i) directly optimized, (ii) restricted to a fixed finite set of static designs, or (iii) defined implicitly via iterative optimization of the incremental EIGs (or a related objective). 2.2

Fairness in Machine Learning

Machine learning systems are increasingly used in real-world decision-making processes, and consequently biases in the way they predict can produce meaningful societal harms, such as disparate access to opportunities, amplification of historical injustices, and loss of trust. A growing literature of methods has thus been developed to improve the fairness of machine learning predictors [18, 26, 48, 79]. What it means for a model to be fair has been formalized in a variety of, often incompatible, ways (see e.g., [16] for a recent survey). One broad distinction is between “individual” notions of fairness— treating similar individuals similarly—and “group” notions of fairness [35], promoting equity by encouraging fair model behavior with respect to sensitive attributes (e.g. gender, age). Our main focus will be on gathering data that improves group fairness, for which common metrics include demographic parity [23, 79], predictive parity [20], and equal opportunity [32] (see Appendix G for more details). Importantly, a common feature of these metrics is their focus on remaining fair with respect to some sensitive attribute we denote as ϕ, e.g. prohibiting the use of race to impact loan decisions. We later see that FairBED has inherent links to demographic parity. Demographic parity is defined by our prediction distributions matching when we condition only on group members, ensuring each group defined by some sensitive attribute(s) exhibits the same selection (or acceptance) rate as all other groups [22]. Mathematically, demographic parity is upheld if our model’s predictions z satisfy p(z|ϕ = ϕ1 ) = p(z|ϕ = ϕ2 )

3

∀z, ϕ1 , ϕ2 .

(4)

Fairer Data Acquisition

Imposing fairness on downstream predictors and decision makers after data collection is naturally hampered by unwanted correlations and biases in the data. Our aim is therefore to introduce a framework that can quantify the fairness of the data itself and then explicitly target fairer data by appropriately controlling the acquisition process. 3.1

Fair Worlds

As a motivating example, consider estimating insurance risk where we want to be fair to people’s skin pigmentation, which thus forms our sensitive attribute. There is no direct causal link from pigmentation to risk, so in an ideal world a fair predictor would not use pigmentation when estimating risk [58]. Unfortunately, as skin pigmentation may correlate with observed outcomes in real-world settings due to societal inequities, models trained on a naively gathered dataset are unlikely to be naturally fair. Instead, they will reflect the unfairness of the data gathered from an unfair world. Imagine, however, a hypothetical alternative world where these societal biases do not exist and risk is independent of pigmentation. Models trained and applied in this world would be inherently fair and could safely ignore skin pigmentation when predicting risk. Our key insight is that we can now measure the unfairness of a dataset in terms of how much it violates this assumed independence, as this is our defining characteristic of a fair world. That is, fair data should not allow information to be transferred between the target variables we want to predict (here insurance risk) and the sensitive attributes (here skin pigmentation), as we believe these should be independent in any hypothetical fair world. We can then further look to operationalize this new notion of fairness by guiding our experimental design process to gather data that ensures such information transfer is avoided. Models trained on such fairer data should then themselves naturally preserve the desired independence and thus be fairer. Of course, we also need to make sure that the gathered data is representative of the true world within which we need to act, as without this there may be a significant generalization gap that not only 3

undermines our predictive performance but potentially also the fairness of our predictions. For example, if we were to only gather data from people with a particular skin pigment, our model would generalise very poorly and likely be unfair in practice for people with different skin pigments. Here though we can note that there is not a single hypothetical fair world: any world that maintains our desired independence should be fair. We can thus try to gather data that is representative of the fair world that is closest to the true world in its underlying relationship between covariates and the target variables we want to predict from them. As discussed in Sec. 3.3, we achieve this by also explicitly targeting data to be as informative as possible about the predictions we want to make, noting that, crucially, as we only control the designs ξ (without intervening on true outcome mechanism y | ξ), the data we gather still inherently supports accurate real–world prediction and estimation. 3.2

Fairer Data

To now formulate this notion of the fairness of a dataset D, we consider what information it reveals about sensitive attributes, ϕ. Namely, if D itself provides no information about ϕ, then models trained on such data will naturally produce predictions that are independent of ϕ (or conditionally independent to ϕ given any provided inputs), assuming we do not instil any prior information about ϕ into the model ourselves. This is an easy requirement to satisfy by simply using downstream models that do not take in ϕ as an input. Specifically, we can quantify the fairness of the data as the negative of the information gain (IG) that it provides about ϕ, −IGϕ (D) (c.f. Sec. 2.1). For experimental design problems, we do not yet know the data and must instead use the expectation of this negative IG instead. Further, both the IG and EIG are functions of the model being used and data should not generally be considered fair simply due to deficiencies in our model. We therefore define our fairness objective for a design policy, πd , to be the negative of the maximum EIG over a set of possible models M of the form pm (ϕ)pm (D|ϕ, πd ) as follows  
m Fu (πd ) = −max Epm (D|πd ) IGm (5) ϕ (D) = −max EIGϕ (πd ) = −max Im ϕ; D | πd m∈M

m∈M

m∈M

where Im (·; ·) denotes mutual information under model m ∈ M. Because the EIG is strictly nonnegative, the maximum possible fairness score under this metric is Fu (πd ) = 0, which represents the data conveying no information about ϕ under any of the models. We can also consider an alternative fairness metric: the negative of the conditional information gain that the data provides about ϕ given θ, −IGm ϕ|θ (D). In turn this yields the fairness objective h i Fc (πd ) = −max Ep(θ)pm (D|θ,πd ) IGm (6) ϕ|θ (D) m∈M h i 
 = −max Ep(θ) EIGm . (7) ϕ|θ (πd ) = −max Ep(θ) Im ϕ; D | θ, πd m∈M

m∈M

This is a stricter metric in the sense that Fc (πd ) = 0 directly implies Fu (πd ) = 0 whenever θ and ϕ are a priori independent; it conveys the fact that we do not want the data to convey information about ϕ for any value of the target θ, as opposed to only marginally. This helps to guard against explaining away effects, where the sensitive attribute might be recoverable once the target variable is established. However, it can also be more computationally challenging to work with and more complicated to define models for (as we need a joint on θ, ϕ rather than just marginal models on each). 3.3

FairBED objective

In practice, Fu and Fc are not appropriate experimental design objectives in their own right, as they include no provision for the data actually being useful for the task at hand: to learn about θ and gather data that is useful for downstream tasks. Indeed, they can be trivially optimized by simply gathering no data at all, or deliberately selecting degenerate data with no meaningful information in it. Viewed from our earlier alternative worlds perspective, we need the data generating process we induce, p(D|πd ), to produce data which is actually helpful for predictions. That is, the hypothetical world we created needs to be as representative as possible of the real world. Thankfully, because we are only controlling the designs without influencing the outcomes themselves, our experimental design procedure is not influencing either the posteriors we derive from given data p(θ|D) or the true distribution on y|ξ. Thus, this need for representativeness is captured simply by ensuring the data is informative about θ. 4

In other words, we ensure data is both useful and fair by making it informative about θ but uninformative about ϕ (or for the conditional fairness metric ϕ|θ). To this end, we now introduce two specific FairBED objectives, building on the fairness criteria above. A conditional FairBED objective, Rc , utilizing Fc , admits a theoretical DP link, and an unconditional FairBED objective, Ru , utilizing Fu , which is often simpler in practice:   Rc (πd ) = EIGθ (πd ) − β Ep(θ) EIGϕ|θ (πd ) , (8)

Ru (πd ) = EIGθ (πd ) − β EIGϕ (πd ), where β > 0 controls the strength of fairness enforcement.

(9)

Unlike Fu and Fc , these are defined under a single model which is typically more practical when using them as experimental design objectives. This is a joint model over θ, ϕ, and data: p(θ)p(ϕ|θ)p(D|θ, ϕ, πd ), but for Ru the objective only depends on the marginals p(θ)p(D|θ, πd ) and p(ϕ)p(D|ϕ, πd ), so these can be defined completely independently if desired (leaving the full joint implicit). We can account for multiple models on ϕ by defining a model class of the form p(θ)p(D|θ, πd )pm (ϕ|θ, D), so that EIGθ remains fixed but we can still choose the worst case model on ϕ; this leads to the second terms in the equations being replaced by −βFc and −βFu respectively. Despite their simplicity, these FairBED objectives provide a powerful basis for informative data gathering while limiting information about sensitive attributes to remain fair. It is important to appreciate, perhaps counterintuitively, that the informativeness and fairness terms in the FairBED objectives do not always trade-off: gathering data that is informative about θ can be necessary to achieve fairness in the real world. Without this, there could be a significant generalization gap between the collected data and the downstream predictions and decisions. Gathering data that is truly informative thus protects against this generalization gap itself varying with sensitive attributes, and thus unfairness manifesting indirectly. For instance, if we collect data only from men, predictions for women are unlikely to be fair because the model cannot generalize. By contrast, if we collect data that makes gender hard to predict while still covering the cases relevant to downstream use, fair predictions are more likely.

4

FairBED and Demographic Parity

We now show, under mild assumptions, that achieving Fc (πd ) = 0 formally links to achieving demographic parity on downstream models derived from any of the resulting conditional posteriors pm (θ|D, ϕ) for models in M. As discussed in Sec. 2.2, demographic parity requires target predictions to be the same when we condition only on the sensitive attributes ϕ (noting that this does not necessarily mean that predictions are still invariant to ϕ when conditioned on additional covariates or inputs that are themselves dependent on ϕ). Let z denote the downstream variable that we wish to predict. Given some class of allowed joint models p(θ)p(D|θ, πd )pm (ϕ|θ, D), then under a Bayesian framework, downstream predictive distributions will take the form Z pm (z|ϕ, D) = p(z|ϕ, θ)pm (θ|D, ϕ)dθ (10) where p(z|ϕ, θ) is a fixed predictive model parameterised by θ and taking ϕ as input. All D conditioning flows through θ. Following (4), demographic parity for dataset D now requires that pm (z|ϕ, D) is invariant with ϕ, that is ∀z, ϕ1 , ϕ2 : pm (z|ϕ = ϕ1 , D) = pm (z|ϕ = ϕ2 , D). (11) A sufficient condition for (11) to hold is that both (i) the θ–posterior and (ii) the downstream predictive model are ϕ-invariant. Here (ii) is addressed in-part by a normative choice to not take ϕ as an input to our downstream model, that is we use a downstream predictor of the form p(z|θ). As we now show, (i) follows directly whenever Fc (πd ) = 0, provided our allowed priors pm (θ, ϕ) = p(θ)pm (ϕ) are separable, such that θ and ϕ are independent. This is a natural assumption as the basis for our fairness definition itself. Violating it would mean we are unfair even before seeing any data. Proposition 4.1. If all priors under our assumed model class are factorizableR such that pm (θ, ϕ) = p(θ)pm (ϕ)∀m ∈ M, then any downstream predictor of the form p(z|ϕ, D) = p(z|θ)pm (θ|D, ϕ)dθ achieves demographic parity for all D whenever data is acquired under a policy that achieves Fc (πd ) = 0. The proof is provided in Appendix A. It should be noted that this result does not necessarily directly extend to the case where our downstream predictor also takes in some additional covariates that are themselves dependent on ϕ, as these can then provide a separate source of unfairness to either our data or model. Discussion of this setting is provided in Appendix A.1. 5

5

Active Learning

Active learning (AL) is an important special case of experimental design in which we sequentially select inputs, x, to gather labels for, y, in order to learn a supervised predictive model for y|x as efficiently as possible [56, 68]. FairBED can be applied to active learning problems whenever the model is probabilistic (whether it is updated in a Bayesian manner or not), such that it can be written in the form p(y | x) = Ep(υ) [p(y | x, υ)] ,

(12)

with υ denoting latent stochastic model variables (e.g., weights, ensemble index, or other randomness). Various acquisition strategies for choosing x have been proposed in AL literature [7, 11, 76]. Of most relevance to our work is the expected predictive information gain (EPIG) acquisition function [10], which targets the EIG in future predictions y∗ under an assumed test-time input distribution p∗ (x∗ ): EPIG(x) = Ep∗ (x∗ ) [I(y; y∗ |x, x∗ )] = I(y; (x∗ , y∗ )|x).

(13)

It thus directly targets labels that are most informative about future predictions, meaning y∗ | x∗ rather than x∗ itself. However, as x∗ is itself stochastic, EPIG is formally the EIG of the pair (x∗ , y∗ ), not of y∗ alone. Unlike BALD [37], EPIG thus directly prioritizes queries that are expected to yield the largest predictive gain on the test distribution itself. 5.1

FairBED for Active Learning

Adapting FairBED to Active Learning (AL), we seek to acquire a labeled dataset that (i) improves predictions for target labels y∗θ for test time inputs x∗ ∼ p∗ (x∗ ), while (ii) limiting prediction of a sensitive test-time label y∗ϕ . We operationalize “leakage” via predictability: if an additional query improves predictions of y∗ϕ on p∗ , then it increases the sensitive information extractable from the released dataset. We want the acquired dataset to carry little information about the sensitive label. As EPIG is just a particular EIG, namely the EIG in (x∗ , y∗ ), it is directly amenable to our FairBED objectives, Rc and Ru . Specifically, we can set θ = (x∗ , y∗θ ) and ϕ = (x′∗ , y∗ϕ ), where x∗ and x′∗ are independent samples from our test time distribution, with y∗θ the yθ label associated with x∗ and y∗ϕ the yϕ label associated with x′∗ . We opt to focus on the unconditional form of FairBED, Ru , for the AL setting, yielding RFairEPIG (x) = I(yθ ; (x∗ , y∗θ )|x) − βI(yϕ ; (x′∗ , y∗ϕ )|x)

(14)

where β controls the trade-off between selecting inputs whose labels will be informative for future prediction of y∗θ but uninformative for future prediction of y∗ϕ . Focusing on the unconditional FairBED form for AL is pragmatic: it is simpler to estimate since it only depends on the marginal predictive distributions for the target and sensitive tasks. The objective is thus invariant to the coupling of the two predictors: any joint distribution over (yθ , yϕ |x) that has the same marginals yθ |x and yϕ |x yields the same acquisition score. Thus, we can train completely separate models on the corresponding histories hθt = {xi , yθi }i=1:t and hϕt = {xi , yϕi }i=1:t , retraining these as new data becomes available in the standard AL manner. This decoupling allows each model to be as predictive as possible for its own label; in particular, a strong predictive model on ϕ acts as an effective adversary for detecting sensitive information.

6

Related Work

Notions of fairness. Fairness criteria split broadly into group fairness (demographic parity, equalised odds; [33, 78]) and individual or metric fairness, which requires similar individuals to receive similar outcomes under a task-specific similarity metric [23, 58, 71]. While our focus has been on group fairness, FairBED does contain some interesting links to individual fairness, particularly its conditional objective: by ensuring data does not contain information about ϕ, it also leads to individual predictions that are independent of ϕ (Eq. 23 in the Appendix). Future work might thus look to explore whether FairBED can also be helpful for individual fairness applications. Fairness mitigations in ML. Most mitigations act after data collection: pre-processing reweights, relabels, or transforms inputs [2, 24, 26, 49, 70]; representational fairness learns latent features predictive of the target but uninformative about the sensitive attribute [55, 79]; in-processing adds 6

Percentage change in EIG (%)

0 −20

Table 1: Location Finding (T =10, β=0.8). Ratio of lower-bound EIG estimates. Ratio of 0 is optimal. FairBED uses conditional objective (8).

θ φ

Unconditional Conditional

−40

−80 −100

EIGϕ /EIGθ

EIGϕ|θ /EIGθ

Random 1.009 ± 0.034 BED (Static) 0.981 ± 0.007 BED (DAD) 0.898 ± 0.007 FairBED (Static) (8) 0.077 ± 0.004 FairBED (DAD) (8) 0.378 ± 0.007

1.402 ± 0.044 1.050 ± 0.007 0.960 ± 0.007 0.290 ± 0.009 0.507 ± 0.009

Method

−60

0

2

4 6 β Coefficient

8

10

Figure 1: Conditional vs. unconditional FairBED across β (static; T =10). Lowerbound EIG. Error bars ± 1 std err (8 seeds). fairness constraints at training time [3, 4, 33, 53, 57, 78]; and post-processing adjusts outputs after training [21, 45]. FairBED is complementary to all above approaches as they can be run alongside using FairBED in the original data acquisition if desired. Yang et al. [77] further show that fairness gains from post-hoc interventions can often fail to generalise in real-world deployment, strengthening the case for upstream interventions like FairBED. Active learning for fairness. Prior AL approaches have targeted individual/metric fairness [71], bandit formulations [73], group-fairness during querying [5, 25, 60, 69], group-level sampling [15], and parity-constrained training with active selection [69]. Unlike FairBED, these require sensitive labels [5, 25] or a labelled validation set [60, 69]. Moreover, they do not introduce their own notions for fairness of the data itself, as opposed to actively training the model to be fair under existing fairness metrics. For example, Anahideh et al. [5] use a nested model retraining per candidate–label pair to directly evaluate the fairness achieved by the downstream model, making it orders of magnitude more expensive than FairBED in the general setting (though they do provide a more efficient covariancebased approximation for the specific case of generalized linear models). By contrast, FairBED formalises and then targets dataset-level fairness by explicitly limiting sensitive-attribute information leakage during acquisition. It does not require sensitive labels at query time, only marginal models for the target and sensitive attributes which can be pretrained or handcrafted models when labels or sensitive attributes are unavailable, and can apply beyond AL to the general BED setting.

Experiments

7.1

Source Location Finding Sensitive Parameter ϕ

7

4

Designs Prior mean

We consider the source location finding BED benchmark [29] in a two-dimensional setting, wherein we infer a hid3 den source location ψ ∈ Rp from signal strength observations y. Unlike the standard BED setup, we only want 2 to learn about one coordinate of the source location, such 0.0 0.5 1.0 1.5 2.0 2.5 3.0 that ψ = (θ, ϕ), with θ the target coordinate and ϕ sensiTarget Parameter θ tive. We optimize designs using FairBED (8), encouraging learning in θ and discouraging learning in ϕ. Appendix F.1 Figure 2: Location Finding Designs provides experimental details. FairBED-optimized de- (Static, T =10, β=0.7). Source prior sign policies are parameterized as static designs [28] and mean is at (3, 3). policy-based DAD [29]. In Fig. 1 we show the change in information gathered about θ and ϕ, respectively as β is increased, when using the unconditional and conditional forms of the FairBED objectives to make design decisions (here we optimize all designs upfront, but consider instead using DAD [29] and other ablations in Appendix F.1). The desired outcome is a smaller reduction in EIGθ than the reduction in EIGϕ (which is the negative expected value of our marginal data fairness metric −IGϕ (D)), indicating that the data we have gathered is more informative about θ than ϕ. We see that both the unconditional and conditional FairBED variants behave similarly for small β, but the conditional variant more strongly suppresses learning in both θ and ϕ for larger β. In Table 1 we further compare FairBED against Random designs and two variants of the standard BED approach of maximising (a sPCE lower bound on) EIGθ . We see that it provides a far better 7

Target 1 Test Time Accuracy (%)

Target 1 Test Time Accuracy (%)

β coefficient 10.0 3.0 1.0 0.1 EPIG Random

82 80 78 Number of Labels 100 200 400

76 74

62

64 66 68 70 Target 2 Test Time Accuracy (%)

β coefficient 10.0 3.0 1.0 0.1 EPIG Random

82 80 78 76 74 72

Number of Labels 100 200 400

60 62 64 66 68 70 72 74 76 Target 2 Test Time Accuracy (%)

Figure 3: Trade-off in target (y axis) vs sensitive attribute (x axis) prediction accuracy for different β values and acquisition sizes (marker shapes) on Student Graduation dataset. Points towards top left preferred.

Figure 4: Trade-off in target (y axis) vs sensitive attribute (x axis) prediction accuracy for different β values and acquisition sizes (marker shapes) on Census dataset. Points towards top left preferred.

ratio of information learned in the two quantities than the baselines. Interestingly, static design also achieves a better ratio than DAD (though this is because it learns less about both θ and ϕ). The designs chosen by FairBED have an intuitive structure for this problem (Fig. 2): they spread along the target coordinate to inform θ and collapse to the prior mean of ϕ. This is desirable behaviour as the lack of variation in ϕ hampers learning it. 7.2

Active Learning Evaluations

Target 1 Test Time Accuracy (%)

We next evaluate FairBED in an active learning setting. Here we first consider the binary classification task dataset: Predict Students’ Dropout and Academic Success [64], where we adapt the task such that yθ indicates whether an individual has graduated and the sensitive label yϕ corresponds to gender (male/female). Second, we consider the Census Income dataset [1], with target yθ indicating whether income is beβ coefficient Number of Labels low or above $50k and sensitive label yϕ cor82 10 100 responding to gender (male/female). Finally, 3 400 we consider CelebA [54], a large-scale, non1 80 0.1 tabular image dataset, to test whether the acquisiEPIG tion strategy extends beyond structured records. 78 Random Here we consider the task of predicting if an individual is smiling with gender as the sensitive 76 attribute. Models. We first model the predictive distribu74 tion with a random forest ensemble, interpreting each tree as a sample from an implicit posterior 0.625 0.650 0.675 0.700 DP Ratio over parameters υ. This provides uncertainty estimates and enables computation of information gain without explicit posterior inference. We Figure 5: Accuracy vs DP ratio for student graduause a label budget of 400 and average over 8 tion dataset for varying β and random acquisition. Points towards top right preferred. random seeds. Baselines. We compare against Random acquisition, the standard acquisition functions EPIG [10] and predictive entropy that only target uncertainty in θ, and an ablation of our FairBED setup based on replacing the EIGs with predictive entropies, that is RHFE (x) = H[p(yθ | x, ht−1 )] − β H[p(yϕ | x, ht−1 )], that we call Heuristic Fair Entropy (HFE).

Metrics. We report test-time predictive accuracy on targets, demographic parity (DP) ratios p(y∗θ = 1|y∗ϕ = 1)/p(y∗θ = 1|y∗ϕ = 0), and equalised odds ratios (App. G).

Table 2 reports DP and EO ratios for FairBED and the baselines. Results show that data gathered with FairBED yields improved downstream DP in all cases and improved EO everywhere except for false positive rate on CelebA. 8

Table 2: DP, EO–True Positive Rate, and EO–False Positive Rate ratios across Census, Student Graduation, and CelebA. FairBED & HFE at β = 10. A ratio of 1 is optimal. Census Method

EO-FPR

DP

EO-TPR

EO-FPR

DP

EO-TPR

EO-FPR

0.54 ± 0.01 0.49 ± 0.01 0.49 ± 0.01 0.56 ± 0.05 0.68 ± 0.06

0.88 ± 0.01 0.86 ± 0.01 0.86 ± 0.01 0.92 ± 0.02 0.93 ± 0.02

0.72 ± 0.04 0.60 ± 0.02 0.64 ± 0.02 0.86 ± 0.04 0.86 ± 0.03

0.61 ± 0.01 0.63 ± 0.01 0.62 ± 0.01 0.65 ± 0.02 0.70 ± 0.02

0.91 ± 0.01 0.90 ± 0.02 0.91 ± 0.01 0.92 ± 0.01 0.92 ± 0.01

0.58 ± 0.02 0.60 ± 0.02 0.60 ± 0.02 0.67 ± 0.02 0.69 ± 0.05

0.65 ± 0.09 0.77 ± 0.08 0.80 ± 0.02 0.82 ± 0.07 0.83 ± 0.03

0.70 ± 0.08 0.80 ± 0.07 0.84 ± 0.02 0.90 ± 0.04 0.90 ± 0.06

0.71 ± 0.10 0.82 ± 0.08 0.89 ± 0.05 0.61 ± 0.03 0.77 ± 0.09

84

β coefficient

84

10.0 3.0 1.0 0.1 EPIG

82 80 78 76 74 72 70

CelebA

EO-TPR

Target 1 Test Time Accuracy (%)

Target 1 Test Time Accuracy (%)

Random Predictive Entropy EPIG HFE FairBED-EPIG

Student Graduation

DP

Model mlp rf svm rbf xgb

56 58 60 62 64 66 68 70 72 74 76 Target 2 Test Time Accuracy (%)

β coefficient 10.0 3.0 1.0 0.1 EPIG

82 80 78 Model

76

mlp rf svm rbf xgb

74

62 64 66 68 70 Target 2 Test Time Accuracy (%)

Figure 6: Generalization across downstream models on Census Dataset with RF used during acquisition. Results are averaged over 8 seeds (400 acquired labels).

Figure 7: Generalization across downstream models on Student Graduation with RF used during acquisition. Points towards top left preferred.

Target 1 Test Time Accuracy (%)

To check that we can achieve improved fairness-accuracy trade-offs rather than merely yielding naturally fairer predictors, Figs. 3 and 4 show the pareto fronts achieved by different choices of β in the predictive accuracy of the learned models on θ (Target 1) and ϕ (Target 2) for the student graduation and census datasets, respectively. We see that FairBED provides an improved β coefficient pareto front compared to random acquisition Method 100 FairBED-LFR 82 while further allowing for customization in the 10 FairBED 3 trade-off. Figure 5 further provides an alterna1 tive view of the trade-off showing the pareto 0.1 80 front of target accuracy to DP ratio, again showEPIG Random-LFR ing an improvement over random acquisition. 78 Interestingly, we also see here a clear improvement in the pareto front when the label budget is larger. Additional ablations on generalization 76 capability and fairness analysis can be found in the Appendix (Figs. 15, 16, 17, 18). 0.60 0.65 0.70 0.75 DP Ratio

7.3 Fairness as a property of the acquired data

Figure 8: Composition of FairBED with LFR on Graduation dataset (with LFR weight Az =1.0) compared with using LFR on model trained with To show that fairness is a property of the acrandom data. Points towards top right preferred. quired data itself, we next acquire labeled points with the random forest ensemble and FairBED objective, freeze the acquired dataset, and train model classes (random forest, XGBoost, MLP, SVM-RBF) on the data. As shown in Figs. 6 and 7, varying β yields the same trade-off across all downstream classifiers, confirming that the reduced sensitiveattribute predictability is a property of the acquired dataset, not an artifact of the acquisition model. 7.4

Contextualising Performance: Additional Comparisons

FairBED targets fair data, so it neither precludes nor competes with downstream fairness interventions. We therefore consider combining it with downstream fairness approaches and compare to just 9

using these in isolation. Namely, we consider using Learned Fair Representations (LFR; [79], pre-processing, App. J). Using Reject Option Classification post-processing (ROC; [45]) instead is considered in App. I. Figure 8 compares running LFR on FairBED acquired data to running LFR on randomly acquired data, as well as just FairBED with no pre-processing to anchor performance. We see that FairBEDLFR Pareto-dominates LFR applied to randomly acquired data, showing that upstream acquisition and downstream representation learning compose constructively—the gains are additive rather than substitutive.

8

Conclusion

In this work, we proposed FairBED, which provides a novel formalism for the fairness of data and operationalizes this to construct practical fairness-aware Bayesian experimental design objectives that actively target the gathering of fair data. Specifically, it measures the unfairness of data in how much information it conveys about sensitive attributes and then constructs objectives that trade this off with the information the data provides for the variables we wish to predict. We showed that FairBED is theoretically linked to demographic parity, and empirically showed that the data gathered using FairBED naturally leads to fairer downstream models and allows improved trade-offs with predictive accuracy.

References [1] Census-Income (KDD). UCI Machine Learning Repository, https://doi.org/10.24432/C5N30T. 8

2000.

DOI:

[2] Abernethy, J., Awasthi, P., Kleindessner, M., Morgenstern, J., Russell, C., and Zhang, J. Active sampling for min-max fairness. arXiv preprint arXiv:2006.06879, 2020. 6 [3] Abraham, S. S., Sundaram, S. S., et al. Fairness in clustering with multiple sensitive attributes. arXiv preprint arXiv:1910.05113, 2019. 7 [4] Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., and Wallach, H. A reductions approach to fair classification. In International conference on machine learning, pp. 60–69. PMLR, 2018. 7 [5] Anahideh, H., Asudeh, A., and Thirumuruganathan, S. Fair active learning. Expert Systems with Applications, 199:116981, 2022. ISSN 0957-4174. doi: https://doi.org/10.1016/ j.eswa.2022.116981. URL https://www.sciencedirect.com/science/article/pii/ S0957417422004055. 7 [6] Ao, Z. and Li, J. On estimating the gradient of the expected information gain in bayesian experimental design. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 20311–20319, 2024. 2 [7] Ash, J. T., Zhang, C., Krishnamurthy, A., Langford, J., and Agarwal, A. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ryghZJBKPS. 6 [8] Barlas, Y. Z. and Salako, K. Performance comparisons of reinforcement learning algorithms for sequential experimental design, 2025. URL https://arxiv.org/abs/2503.05905. 20 [9] Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., Nagar, S., Ramamurthy, K. N., Richards, J., Saha, D., Sattigeri, P., Singh, M., Varshney, K. R., and Zhang, Y. AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias, October 2018. URL https://arxiv.org/abs/1810.01943. 31 [10] Bickford Smith, F., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., and Rainforth, T. Predictionoriented bayesian active learning. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 7331–7348. PMLR, 25–27 Apr 2023. URL https://proceedings.mlr.press/v206/bickfordsmith23a.html. 6, 8, 19, 20 10

[11] Bickford-Smith, F., Foster, A., and Rainforth, T. Making better use of unlabelled data in bayesian active learning. ArXiv, abs/2404.17249, 2024. URL https://api.semanticscholar.org/ CorpusID:269430415. 6 [12] Blau, T., Bonilla, E. V., Chades, I., and Dezfouli, A. Optimizing sequential experimental design with deep reinforcement learning. In International conference on machine learning, pp. 2107–2128. PMLR, 2022. 2, 20 [13] Bracher, N., Kühmichel, L., Ivanova, D. R., Intes, X., Bürkner, P.-C., and Radev, S. T. Jadai: Jointly amortizing adaptive design and bayesian inference, 2025. URL https://arxiv.org/ abs/2512.22999. 20 [14] Cai, W., Encarnacion, R., Chern, B., Corbett-Davies, S., Bogen, M., Bergman, S., and Goel, S. Adaptive sampling strategies to construct equitable training datasets. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, pp. 1467–1478, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. doi: 10.1145/3531146.3533203. URL https://doi.org/10.1145/3531146.3533203. 1 [15] Cai, W., Encarnacion, R., Chern, B., Corbett-Davies, S., Bogen, M., Bergman, S., and Goel, S. Adaptive sampling strategies to construct equitable training datasets. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 1467–1478, 2022. 7 [16] Caton, S. and Haas, C. Fairness in machine learning: A survey. ACM Comput. Surv., 56(7), April 2024. ISSN 0360-0300. doi: 10.1145/3616865. URL https://doi.org/10.1145/3616865. 1, 3 [17] Cavagnaro, D. R., Myung, J. I., Pitt, M. A., and Kujala, J. V. Adaptive design optimization: A mutual information-based approach to model discrimination in cognitive science. Neural computation, 22(4):887–905, 2010. 2 [18] Chai, J. and Wang, X. Fairness with adaptive weights. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2853– 2866. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/chai22a. html. 1, 3 [19] Chaloner, K. and Verdinelli, I. Bayesian experimental design: A review. Statistical Science, pp. 273–304, 1995. 1, 2 [20] Chouldechova, A. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments, 2017. URL https://arxiv.org/abs/1703.00056. 1, 3 [21] Cruz, A. F. and Hardt, M. Unprocessing seven years of algorithmic fairness. arXiv preprint arXiv:2306.07261, 2023. 7 [22] del Barrio, E., Gordaliza, P., and Loubes, J.-M. Review of mathematical frameworks for fairness in machine learning, 2020. URL https://arxiv.org/abs/2005.13755. 2, 3, 30 [23] Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness, 2011. URL https://arxiv.org/abs/1104.3913. 3, 6 [24] Dwork, C., Immorlica, N., Kalai, A. T., and Leiserson, M. Decoupled classifiers for group-fair and efficient machine learning. In Conference on fairness, accountability and transparency, pp. 119–133. PMLR, 2018. 6 [25] Fajri, R. M., Saxena, A., Pei, Y., and Pechenizkiy, M. Fal-cur: Fair active learning using uncertainty and representativeness on fair clustering. Expert Syst. Appl., 242(C), May 2024. ISSN 0957-4174. doi: 10.1016/j.eswa.2023.122842. URL https://doi.org/10.1016/j. eswa.2023.122842. 7 [26] Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’15, pp. 259–268, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336642. doi: 10.1145/2783258.2783311. URL https://doi.org/10.1145/2783258.2783311. 1, 3, 6 11

[27] Foster, A., Jankowiak, M., Bingham, E., Horsfall, P., Teh, Y. W., Rainforth, T., and Goodman, N. Variational Bayesian Optimal Experimental Design. In Advances in Neural Information Processing Systems 32, pp. 14036–14047. Curran Associates, Inc., 2019. 2 [28] Foster, A., Jankowiak, M., O’Meara, M., Teh, Y. W., and Rainforth, T. A unified stochastic gradient approach to designing bayesian-optimal experiments. In International Conference on Artificial Intelligence and Statistics, pp. 2959–2969. PMLR, 2020. 7, 18 [29] Foster, A., Ivanova, D. R., Malik, I., and Rainforth, T. Deep adaptive design: Amortizing sequential bayesian experimental design. Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR 139, 2021. 2, 7, 18, 20, 21, 22 [30] Gao, J., Chou, B., McCaw, Z. R., Thurston, H., Varghese, P., Hong, C., and Gronsbell, J. What is fair? defining fairness in machine learning for health. Statistics in Medicine, 44(20-22): e70234, 2025. 30 [31] Goda, T., Hironaka, T., and Kitade, W. Unbiased mlmc stochastic gradient-based optimization of bayesian experimental designs. arXiv preprint arXiv:2005.08414, 2020. 2 [32] Hardt, M., Price, E., Price, E., and Srebro, N. Equality of opportunity in supervised learning. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_files/paper/2016/file/ 9d2682367c3935defcb1f9e247a97c0d-Paper.pdf. 3, 30 [33] Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016. 6, 7 [34] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 29 [35] He, X. and Li, Y. Fairness in machine learning: A review for statisticians. Journal of the American Statistical Association, 120(552):2834–2851, 2025. doi: 10.1080/01621459.2025. 2579579. URL https://doi.org/10.1080/01621459.2025.2579579. 1, 3 [36] Hedman, M., Ivanova, D. R., Guan, C., and Rainforth, T. Step-DAD: Semi-amortized policybased bayesian experimental design. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=JRg8P2bX8P. 2, 20 [37] Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011. 6, 19 [38] Hsu, B., Mazumder, R., Nandy, P., and Basu, K. Pushing the limits of fairness impossibility: Who’s the fairest of them all? Advances in Neural Information Processing Systems, 35: 32749–32761, 2022. 30 [39] Huan, X., Jagalur, J., and Marzouk, Y. Optimal experimental design: Formulations and computations. Acta Numerica, 33:715–840, 2024. 2 [40] Huang, D., Guo, Y., Acerbi, L., and Kaski, S. Amortized bayesian experimental design for decision-making. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9798331314385. 20 [41] Huang, D., Wen, X., Bharti, A., Kaski, S., and Acerbi, L. Aline: Joint amortization for bayesian inference and active data acquisition. ArXiv, abs/2506.07259, 2025. URL https: //api.semanticscholar.org/CorpusID:279251169. 20 [42] Iollo, J., Heinkelé, C., Alliez, P., and Forbes, F. Bayesian experimental design via contrastive diffusions, 2025. URL https://arxiv.org/abs/2410.11826. 20 [43] Iqbal, S., Corenflos, A., Särkkä, S., and Abdulsamad, H. Nesting particle filters for experimental design in dynamical systems, 2024. URL https://arxiv.org/abs/2402.07868. 2 12

[44] Ivanova, D. R., Foster, A., Kleinegesse, S., Gutmann, M., and Rainforth, T. Implicit Deep Adaptive Design: Policy–Based Experimental Design without Likelihoods. In Advances in Neural Information Processing Systems, volume 34, pp. 25785–25798. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/ d811406316b669ad3d370d78b51b1d2e-Paper.pdf. 2, 20 [45] Kamiran, F., Karim, A., and Zhang, X. Decision theory for discrimination-aware classification. In 2012 IEEE 12th international conference on data mining, pp. 924–929. IEEE, 2012. 7, 10, 31 [46] Kerrigan, G., Naesseth, C. A., and Rainforth, T. A geometric approach to optimal experimental design. In The 29th International Conference on Artificial Intelligence and Statistics, 2026. URL https://openreview.net/forum?id=u0aepMHQ5p. 20 [47] Kleinegesse, S., Drovandi, C., and Gutmann, M. U. Sequential Bayesian Experimental Design for Implicit Models via Mutual Information. Bayesian Analysis, pp. 1 – 30, 2021. doi: 10.1214/20-BA1225. 20 [48] Krasanakis, E., Spyromitros-Xioufis, E., Papadopoulos, S., and Kompatsiaris, Y. Adaptive sensitive reweighting to mitigate bias in fairness-aware classification. In Proceedings of the 2018 World Wide Web Conference, WWW ’18, pp. 853–862, Republic and Canton of Geneva, CHE, 2018. International World Wide Web Conferences Steering Committee. ISBN 9781450356398. doi: 10.1145/3178876.3186133. URL https://doi.org/10.1145/3178876.3186133. 1, 3 [49] Krasanakis, E., Spyromitros-Xioufis, E., Papadopoulos, S., and Kompatsiaris, Y. Adaptive sensitive reweighting to mitigate bias in fairness-aware classification. In Proceedings of the 2018 world wide web conference, pp. 853–862, 2018. 6 [50] Lim, V., Novoseller, E., Ichnowski, J., Huang, H., and Goldberg, K. Policy-based bayesian experimental design for non-differentiable implicit models. arXiv preprint arXiv:2203.04272, 2022. 20 [51] Lindley, D. V. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, pp. 986–1005, 1956. 1, 2, 20 [52] Lindley, D. V. Bayesian statistics, a review, volume 2. SIAM, 1972. 1, 2 [53] Liu, E. Z., Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., and Finn, C. Just train twice: Improving group robustness without training group information. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6781–6792. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/liu21f.html. 7 [54] Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3730–3738, 2014. URL https://api.semanticscholar.org/CorpusID:459456. 8, 29 [55] Locatello, F., Abbati, G., Rainforth, T., Bauer, S., Scholkopf, B., and Bachem, O. On the fairness of disentangled representations. In Neural Information Processing Systems, 2019. URL https://api.semanticscholar.org/CorpusID:173187920. 6 [56] MacKay, D. J. Information-based objective functions for active data selection. Neural computation, 4(4):590–604, 1992. 2, 6 [57] Martinez, N., Bertran, M., and Sapiro, G. Minimax pareto fairness: A multi objective perspective. In International conference on machine learning, pp. 6755–6764. PMLR, 2020. 7 [58] Mitchell, S., Potash, E., Barocas, S., D’Amour, A., and Lum, K. Algorithmic fairness: Choices, assumptions, and definitions. Annual Review of Statistics and Its Application, 2021. URL https://api.semanticscholar.org/CorpusID:228893833. 3, 6 [59] Myung, J. I., Cavagnaro, D. R., and Pitt, M. A. A tutorial on adaptive design optimization. Journal of mathematical psychology, 57(3-4):53–67, 2013. 20 13

[60] Pang, J., Wang, J., Zhu, Z., Yao, Y., Qian, C., and Liu, Y. Fairness without harm: An influenceguided active sampling approach. Advances in Neural Information Processing Systems, 37: 61513–61548, 2024. 7 [61] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc., 2019. 29 [62] Rainforth, T., Cornish, R., Yang, H., Warrington, A., and Wood, F. On nesting monte carlo estimators. In International Conference on Machine Learning, pp. 4267–4276. PMLR, 2018. 2, 20 [63] Rainforth, T., Foster, A., Ivanova, D. R., and Bickford Smith, F. Modern bayesian experimental design. Statistical Science, 39(1):100–114, 2024. 1, 2, 20 Predict Students’ [64] Realinho, V., Vieira, M. M., Machado, J., and Baptista, L. Dropout and Academic Success. UCI Machine Learning Repository, 2021. DOI: https://doi.org/10.24432/C5MC89. 8 [65] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015. 29 [66] Ryan, E. G., Drovandi, C. C., Thompson, M. H., and Pettitt, A. N. Towards bayesian experimental design for nonlinear models that require a large number of sampling times. Computational Statistics & Data Analysis, 70:45–60, 2014. ISSN 0167-9473. doi: https: //doi.org/10.1016/j.csda.2013.08.017. 20 [67] Ryan, E. G., Drovandi, C. C., McGree, J. M., and Pettitt, A. N. A review of modern computational algorithms for bayesian optimal design. International Statistical Review, 84(1):128–154, 2016. 2 [68] Settles, B. Active learning literature survey. Technical report, University of Wisconsin— Madison, 2009. 6 [69] Sharaf, A., Daume III, H., and Ni, R. Promoting fairness in learned models by learning to active learn under parity constraints. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pp. 2149–2156, 2022. 1, 7 [70] Shekhar, S., Fields, G., Ghavamzadeh, M., and Javidi, T. Adaptive sampling for minimax fair classification. Advances in Neural Information Processing Systems, 34:24535–24544, 2021. 6 [71] Shen, J., Cui, N., and Wang, J. Metric-fair active learning. In International conference on machine learning, pp. 19809–19826. PMLR, 2022. 1, 6, 7 [72] Sweeney, L. Discrimination in online ad delivery. Commun. ACM, 56(5):44–54, May 2013. ISSN 0001-0782. doi: 10.1145/2447976.2447990. URL https://doi.org/10.1145/ 2447976.2447990. 1 [73] Tae, K. H., Zhang, H., Park, J., Rong, K., and Whang, S. E. Falcon: Fair active learning using multi-armed bandits, 2024. URL https://arxiv.org/abs/2401.12722. 7 [74] Vincent, B. T. and Rainforth, T. The DARC toolbox: automated, flexible, and efficient delayed and risky choice experiments using bayesian adaptive design. PsyArXiv preprint, 2017. 20 [75] Williams, D. R. . M. S. A. Discrimination and racial disparities in health: evidence and needed research. Journal of behavioral medicine, 32(1), 20–47., 2009. URL https://doi.org/10. 1007/s10865-008-9185-0. 1 14

[76] Xia, Y., Mukherjee, S., Xie, Z., Wu, J., Li, X., Aponte, R., Lyu, H., Barrow, J., Chen, H., Dernoncourt, F., Kveton, B., Yu, T., Zhang, R., Gu, J., Ahmed, N. K., Wang, Y., Chen, X., Deilamsalehy, H., Kim, S., Hu, Z., Zhao, Y., Lipka, N., Yoon, S., Huang, T. K., Wang, Z., Mathur, P., Pal, S., Mukherjee, K., Zhang, Z., Park, N., Nguyen, T. H., Luo, J., Rossi, R. A., and McAuley, J. J. From selection to generation: A survey of llm-based active learning. CoRR, abs/2502.11767, 2025. doi: 10.48550/ARXIV.2502.11767. URL https://doi.org/10. 48550/arXiv.2502.11767. 6 [77] Yang, Y., Zhang, H., Gichoya, J. W., Katabi, D., and Ghassemi, M. The limits of fair medical imaging ai in real-world generalization. Nature Medicine, 30:2838 – 2848, 2024. URL https://api.semanticscholar.org/CorpusID:270822280. 7 [78] Zafar, M. B., Valera, I., Rogriguez, M. G., and Gummadi, K. P. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics, pp. 962–970. PMLR, 2017. 6, 7 [79] Zemel, R., Wu, Y., Swersky, K., Pitassi, T., and Dwork, C. Learning fair representations. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 325–333, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR. URL https://proceedings.mlr.press/ v28/zemel13.html. 1, 3, 6, 10, 32

15

A

Proofs

Proof. Because the EIG is strictly non-negative, Fc (πd ) = 0 implies that EIGm ϕ|θ = 0 almost surely for all m ∈ M, which is equivalent to Im (D; ϕ | θ, πd ) = 0. By the defining property of conditional mutual information, this implies conditional independence, such that ∀m ∈ M pm (D | θ, ϕ, πd ) = p(D | θ, πd )

∀θ

(15)

and by extension also pm (D | ϕ, πd ) = p(D | πd ) given our factorisable prior assumption. Now consider the posterior over θ given (D, ϕ): pm (D | θ, ϕ, πd ) pm (θ | ϕ) pm (D | ϕ, πd ) pm (D | θ, ϕ, πd ) p(θ) = by the factorisable prior assumption pm (D | ϕ, πd ) p(D | θ, πd ) p(θ) = by (15) p(D|πd ) = p(θ | D),

pm (θ | D, ϕ) =

(16) (17) (18) (19)

where our dependence on πd has gone away because it only influences learning through D and the dependence on our choice of model from M has gone away because all models are equivalent once ϕ is marginalised out. Finally, evaluate the posterior predictive distribution: Z Z p(z | ϕ, D) = p(z | θ) pm (θ | D, ϕ) dθ = p(z | θ) p(θ | D) dθ = p(z|D) (20) yields a predictor that is independent of ϕ for all D, regardless of our choice of p(z | θ) or our chosen model m ∈ M. We thus obtain the required demographic parity defined in (11). A.1

Exploring the Fixed Predictive Model p(z|θ)

Within (20) we assumed that the fixed predictive model had the form p(z | θ): the predictor depends only on the target parameters. This section explores several sufficient conditions under which this form is naturally recovered. The more general form of the fixed predictive model takes x, θ, and ϕ as inputs, where x is some input provided at test time that might itself be dependent on ϕ. The corresponding posterior predictive distribution is Z Z p(z | ϕ, D) = p(z | x, θ, ϕ) p(θ | D) p(x | ϕ) dx dθ, (21) where p(θ | D) is ϕ-invariant by (19). While FairBED ensures no ϕ-dependence due to the data D, two channels of ϕ-dependence remain: the fixed predictor p(z | x, θ, ϕ) itself, and the test-time covariate distribution p(x | ϕ). A natural normative choice at deployment is to exclude ϕ from the predictor entirely, so that p(z | x, θ, ϕ) = p(z | x, θ).

(22)

This eliminates the first channel of ϕ-dependence by construction. The question then reduces to the role of the test-time covariates x and their relationship to ϕ. To begin, observe that regardless of the relationship between x and ϕ, the conditional predictor at any fixed x is already ϕ-independent: Z p(z | x, ϕ, D) = p(z | x, θ) p(θ | D) dθ = p(z | x, D), (23) since neither p(z | x, θ) nor p(θ | D) depends on ϕ. This is a fair trained predictor: for any specific input x, the prediction does not depend on the sensitive attribute. The trained model itself is entirely ϕ-invariant. Any unfairness in (21) therefore comes entirely from the test-time covariate distribution p(x | ϕ). We argue that this should be dealt with as a post-processing step at test-time rather than at 16

data acquisition, as p(x | ϕ) is neither a function of the data nor even the model, and different uses of the model might require different corrections. We now consider though the conditions under which marginal demographic parity is achieved without requiring any such post-processing. Specifically, we present two alternative sufficient conditions, with Demographic parity achieved if either holds. The first operates on the marginal covariate distribution p(x | ϕ), while the second operates at the level of a fixed θ, introducing the conditional distribution p(x | θ).

Sufficient Condition 1 (Marginal covariate independence: x ⊥ ϕ). The test-time covariate distribution satisfies p(x | ϕ) = p(x). Then under this condition, (21) (the marginal of (23)) becomes Z Z p(z | ϕ, D) = p(z | x, θ) p(θ | D) p(x) dx dθ = p(z | D), (24)

yielding demographic parity. The effective marginal predictor (across different x) for a given θ is R p(z | x, θ) p(x) dx = p(z | θ), recovering the form assumed in (20). There are settings where full marginal independence of test-time covariates from the sensitive attribute does not hold (Condition 1): groups defined by ϕ often present with systematically different covariate distributions due to structural inequities, selection effects, or genuine population differences (e.g. in health settings). When p(x | ϕ) ̸= p(x), full marginal demographic parity may not hold. But crucially, (23) still holds: any residual violation of demographic parity is entirely attributable to the biased test-time covariate distribution p(x | ϕ) and not to the trained model or the data it was trained on. This cleanly isolates the source of unfairness to one identifiable, external component of the pipeline, which is itself a meaningful step forward. Note that settings in which demographic parity is the appropriate fairness criterion are precisely those in which striving for unbiased test-time covariates is a reasonable goal, since the covariate bias reflects structural inequities rather than genuine population differences; in settings where genuine differences do exist (e.g. health settings), alternative fairness metrics are likely more appropriate, but (23) ensures that individual predictions remain ϕ-invariant regardless, so the trained model can be deployed in either regime. The separation of trained-model fairness from test-time covariate fairness established by (23) is particularly valuable in settings where historical data encodes biases that are no longer present at deployment: a model trained naively on such data would inherit those biases and violate demographic parity even when current test-time covariates are unbiased, whereas our approach ensures that the acquired training data does not encode sensitive information, so that when p(x | ϕ) = p(x) holds at test time, full demographic parity is achieved. Sufficient Condition 2 (Conditional covariate independence: x ⊥ ϕ | θ). Sufficient Condition 1 requires marginal independence between x and ϕ. An alternative is to work at the level of a fixed θ. The effective prediction for a given θ is obtained by marginalising over the conditional covariate distribution p(x | θ, ϕ): Z p(z | θ, ϕ) = p(z | x, θ) p(x | θ, ϕ) dx. (25) Now if x ⊥ ϕ | θ, so that p(x | θ, ϕ) = p(x | θ), then Z p(z | θ, ϕ) = p(z | x, θ) p(x | θ) dx = p(z | θ),

(26)

directly recovering an effective predictor of the form p(z | θ). Demographic parity now follows: Z p(z | ϕ, D) = p(z | θ) p(θ | D) dθ = p(z | D). (27) Example: insurance risk. Let θ denote a policyholder’s true risk profile, x their observable characteristics (property age, claim history, driving record), and ϕ their skin pigmentation. Condition 2 asserts that for two individuals with identical risk profiles, their observable characteristics do not additionally depend on pigmentation. The marginal correlation between pigmentation and observables exists, but only because structural inequities cause the distribution of risk profiles to differ across groups. Once we condition on the actual risk level, pigmentation is redundant for predicting what observables we would see. This is plausible when the sensitive attribute’s influence on the observables 17

is fully mediated through the target variable: pigmentation does not cause a house to be older or a driving record to be worse, but it correlates with socioeconomic factors that determine risk, which in turn determine those observables. The condition fails when ϕ has a direct causal effect on x that bypasses θ; for instance, in medical settings where gender affects biomarker levels conditional on disease severity. Remark A.1 (These conditions are sufficient, not necessary). The fundamental requirement is that the effective predictor, after marginalising over test-time inputs, behaves as p(z | θ). Conditions 1 and 2 identify tractable, verifiable cases where this can be confirmed. However, cancellations between ϕ-dependence in p(x | ϕ) (or p(x | θ, ϕ)) and in p(z | x, θ) could yield a ϕ-invariant integral even when neither condition holds.

B

Detailed FairBED objective Ru (πd ) = Edata [IGθ − βIGϕ ] = Edata [IGθ ] − β Edata [IGϕ ]   P (D|ϕ, πd ) P (D|θ, πd ) = EP (θ,ϕ)P (D|θ,ϕ,πd ) log − β log P (D | πd ) P (D | πd )   P (D|θ, πd ) + (β − 1) log P (D | πd ) = EP (θ,ϕ)P (D|θ,ϕ,πd ) log P (D|ϕ, πd )β = EP (θ,ϕ)P (D|θ,ϕ,πd ) [log P (D|θ, πd ) − β log P (D|ϕ, πd ) + (β − 1) log P (D|πd )]

Let A(θ, ϕ, D, πd ) denote the integrand inside the expectation; it depends on πd through the three log-density terms. We compute the gradient used to update designs as follows. If designs are produced by a neural network, we backpropagate from πd to network parameters via the chain rule, and in practice we rely on PyTorch’s automatic differentiation. The expressions below are the true gradients of Ru (πd ). In practice, the marginal log-densities log p(D | θ, πd ), log p(D | ϕ, πd ) and log p(D | πd ) are intractable. Therefore, we instead replace A with estimator (sPCE/sNMC) [28, 29].

Reparameterized form (used in our experiments). In the source location finding experiments the observation likelihood p(D | θ, ϕ, πd ) is reparameterizable: we can write D = g(ε, θ, ϕ, πd ) for some base noise ε ∼ p(ε) that does not depend on πd (here ε collects the Gaussian observation noise). The sampling distribution is then independent of πd and the gradient is purely pathwise: 
 ∇πd Ru (πd ) = Ep(θ,ϕ) p(ε) ∇πd A θ, ϕ, g(ε, θ, ϕ, πd ), πd . This is the form realised by autograd in our implementation. General (score-function) form. When the likelihood is not reparameterizable, the same gradient admits the standard score-function plus direct-derivative decomposition, which we include for completeness: Z  d ∇πd Ru (πd ) = Ep(θ,ϕ) (p(D|θ, ϕ, πd ) A) dD dπd Z  d d = Ep(θ,ϕ) A p(D|θ, ϕ, πd ) + p(D|θ, ϕ, πd ) A dD dπd dπd Z  d d = Ep(θ,ϕ) A p(D|θ, ϕ, πd ) log p(D|θ, ϕ, πd ) + p(D|θ, ϕ, πd ) A dD dπd dπd   d d = Ep(θ,ϕ) p(D|θ,ϕ,πd ) A log p(D|θ, ϕ, πd ) + A . dπd dπd When reparameterization is available, the score-function term can be avoided and the gradient can instead be estimated by differentiating through the reparameterized sample path, yielding the pathwise form above. For the conditional form we have:     p(D | θ, πd ) p(D | θ, ϕ, πd ) Rc (πd ) = Ep(θ) p(D|θ,πd ) log − β Ep(θ) p(ϕ|θ) p(D|θ,ϕ,πd ) log p(D | πd ) p(D | θ, πd ) = Ep(θ) p(ϕ|θ) p(D|θ,ϕ,πd ) [log p(D | θ, πd ) − log p(D | πd ) − β log p(D | θ, ϕ, πd ) + β log p(D | θ, πd )] = Ep(θ) p(ϕ|θ) p(D|θ,ϕ,πd ) [(1 + β) log p(D | θ, πd ) − β log p(D | θ, ϕ, πd ) − log p(D | πd )] . 18

C

Active learning

Active learning (AL) baseline. In AL [10, 37] we choose inputs x whose labels y will be queried so as to reduce the epistemic uncertainty of a classifier with predictive distribution pµ (y | x). The ensuing labeled set accelerates training relative to randomly sampling inputs to label. pµ (y | x) = Epµ (υ) [pµ (y | x, υ)] ,

(28)

Expected predictive information gain (EPIG). The EPIG goal extends this notion. Let the unseen test-time input distribution follow x∗ ∼ p∗ (x∗ ). The information gained about their labels y∗ after querying (x, y) is     IGy∗ (x, y, x∗ ) = H pµ (y∗ | x∗ ) − H pµ (y∗ | x∗ , x, y) . (29) Taking expectations over the random test input x∗ and the unknown queried label y yields the standard EPIG acquisition score.   (30) EPIG(x) = E x∗ ∼p∗ , y∼pµ (·|x) IGy∗ (x, y, x∗ ) . Note that the target inputs are random as at test time we do not know which input will be sampled which is in contrast to the input x which is selected during training time. C.1

FairBED AL Objectives.

Our implementation introduces two predictive distributions: pµ (yθ | x)

(target, e.g. IQ),

qµ (yϕ | x)

(sensitive, e.g. gender).

At test time we observe the same inputs x∗ but obtain two labels (y∗θ , y∗ϕ ). We seek queries that are informative about the target y∗θ yet uninformative about y∗ϕ . Here, we adapt the notation from the general BED case such that θ := (x∗ , y∗θ ) and ϕ := (x∗ , y∗ϕ ). The resulting active-learning objective becomes LFairBED-uncond (x) = Edata [IGθ − βIGϕ ] = Edata [IGy∗ θ − β IGy∗ ϕ ] (31) h    i = E x∗ ,x′∗ ∼p∗ Eyθ ∼pµ (·|x) IGy∗θ (x, yθ , x∗ ) − β Eyϕ ∼qµ (·|x) IGy∗ϕ (x, yϕ , x′∗ ) (32) (33)

= LFairBED-EPIG (x)

Where we rename the objective LFairBED-EPIG for the AL setting. Maximizing the objective chooses data points that sharpen the target predictor while deliberately limiting information about the sensitive predictor. We also note that the conditional variant of FairBED under two distinct predictive models reduces down to the same objective. Consider the following general case, noting the equivalence of EIG and MI. LFairBED-cond (x) = Im (θ; D | x) − β Ep(θ) [Im (ϕ; D | θ, x)]

(34)

In the general AL case we have D := (x, yθ , yϕ ), θ := (x∗ , y∗θ ) and ϕ := (x′∗ , y∗ϕ ). LFairBED-cond (x) = Im ((x∗ , y∗θ ); D | x) − β Ep(θ) [Im (ϕ; D | θ, x)]

(35)

In the case where we use two distinct models to model the yθ and yϕ distributions, the objective then resolves into the FairBED-EPIG objective (14) due to no mechanism to pass θ dependence when under the alternative model qµ . 19

  LFairBED-cond (x) = Ex∗ ∼p∗ Ipµ (yθ ; y∗θ | x, x∗ ) − β Im (ϕ; D | x) (36) h i     = Ex∗ ,x′∗ ∼p∗ Eyθ ∼pµ (·|x) IGy∗θ (x, yθ , x∗ ) − β Eyϕ ∼qµ (·|x) IGy∗ϕ (x, yϕ , x′∗ )

(37) (38)

= LFairBED-EPIG (x)

We also introduce a predictive entropy baseline (39). Note as discussed in Bickford Smith et al. [10], the entropy formulation loses the notion of a test-time distribution p∗ (x∗ ), so is more liable to the pathologies from not considering this setting. RHFE (x) = H[p(yθ | x, D)] − β H[p(yϕ | x, D)]

D

(39)

EIG Estimators

The Nested Monte Carlo (NMC) bound [62] provides an upper bound on the EIG such that, " # p(y|θ0 , ξ) EIG(ξ) ≤ Ep(θ0 ,y|ξ)p(θ1:L ) log 1 PL l=1 p(y|θl , ξ) L with θ0 , y ∼ p(θ0 , y|ξ) and θ1:L ∼ p(θ).

(40)

The sequential Prior Contrastive Estimator (sPCE) bound [29] provides a lower bound on the EIG such that, " # p(y|θ0 , ξ) (41) EIG(ξ) ≥ Ep(θ0 ,y|ξ)p(θ1:L ) log 1 PL l=0 p(y|θl , ξ) L+1 with θ0 , y ∼ p(θ0 , y|ξ) and θ1:L ∼ p(θ).

E

Additional Related Work

Data acquisition and BED. Canonical BED has followed a two-step strategy of posterior inference followed by EIG maximization [51, 59, 63, 66, 74]. BED has recently been extended to: non-myopic objectives that plan over future design steps [13, 29, 40, 41], bounds and surrogates for tractable estimation of EIG [29, 42–44], implicit settings working with samples from the likelihood [8, 12, 44, 47, 50], semi-amortized approaches [36] and moving beyond EIG [46]. Existing work predominantly treats the objective as unconstrained.

F

Experiment Details

F.1

Location Finding

We consider a synthetic source-localization task in which the goal is to infer the unknown locations ψ of K sources (with K treated as known). For any queried design point x, we observe an intensity measurement y which is a noisy version of the log of an underlying signal model. Specifically, the mean log-intensity is determined by K X αk µ(ψ, x) = b + (42) 2. (m + ||ψk − x||) k=1 where b > 0 is a constant background level, m is a fixed offset controlling saturation/maximum signal, and αk denotes the contribution strength of source k. Observations follow a log-normal noise model: log[y | ψ, x] ∼ N (log µ(ψ, x), σ 2 ).

(43)

During training, we place an i.i.d. standard Gaussian prior on the source locations, i.i.d.

ψk ∼ N (ηd , Id ), 20

k = 1, . . . , K.

(44)

F.1.1

Training details

Tables 3 and 4 describe the architecture of the DAD policy network. Tables 5 and 6 describe the model hyper-parameters.

Table 3: Source location finding. Encoder network Eϕ1 , architecture [29] Layer Overview Dimension Activation Design-outcome H1 H2 Output

x, y Fully connected Fully connected Fully Connected

3 64 256 16

RELU RELU -

Table 4: Source location finding. Decoder network Fϕ2 , architecture [29] Layer Overview Dimension Activation Input H1 H2 Output

E(ht ) Fully connected Fully connected x

16 128 16 2

RELU RELU -

Table 5: Source location finding. Parameter Values Parameter Value αk Max signal, m Base signal, b Observation noise scale, σ

1 for all k 10−4 10−1 0.5

Table 6: Source location finding. Parameters for training DAD. Parameter Value Batch size Number of negative samples Number of gradient steps (default) Learning rate (LR) Number of samples for marginalizing a parameter Number of inner samples (evaluation) Number of theta (evaluation)

F.1.2

100 1000 10K 0.00005 1000 100000 100

Location finding Fairness–utility frontiers

We sweep β for both static and DAD parameterizations, showing monotone suppression of EIGϕ with a controlled reduction in EIGθ (Figs. 9a–9b and Tab 1; extended ranges and Pareto visualizations are provided in Figs. 10–12). Achieving comparable reductions in EIGϕ often requires larger β in the policy case. 21

Percentage change in EIG (%)

Percentage change in EIG (%)

0

θ param φ param

−20 −40 −60 −80 −100

0.25

0.50

0.75 1.00 β Coefficient

1.25

1.50

(a) Static, T =10. Increasing β strengthens unlearning of the sensitive coordinate ϕ with some target learning trade-off. EIGϕ exhibits diminishing returns at larger β near saturation, whereas EIGθ remains significant over the same range.

0

θ param φ param

−20 −40 −60 −80 −100

0.25

0.50

0.75 1.00 β Coefficient

1.25

1.50

(b) DAD, T =10. Qualitatively similar trade-offs to the static case, but with a shifted β scale: larger β is typically needed to obtain comparable suppression of EIGϕ .

Figure 9: Fairness–utility frontier induced by β. Comparison of static and DAD acquisition settings. Error bars denote ± 1 standard error (too small to be seen).

F.1.3

Location finding ablation study

0

Percentage change in EIG (%)

Percentage change in EIG (%)

In Figs. 11 and 10 we demonstrate how β informs the trade-off between information gain in θ against information gain for the sensitive attribute ϕ for static designs and policy-based DAD [29]. Increasing β encourages unlearning along ϕ, at the cost of reduced information gain along θ. For the static design, as β increases, unlearning along ϕ plateaus with approximately 98% reduction in EIG, while θ retains significant information. This view extends the experiments for β > 1 and confirms that this plateau persists. Both use objective (8).

θ param φ param

−20 −40 −60 −80 −100

0

2

4 6 β Coefficient

8

10

Figure 10: Sensitivity to β in the Source Location Finding experiment with a static policy (T = 10). Increasing β promotes unlearning along ϕ at the expense of information gain along θ, with ϕ quickly plateauing while θ retains substantial information even for β > 1. Errors ± 1 std err but are too small to be seen.

0

θ param φ param

−20 −40 −60 −80 −100

0

2

4 6 β Coefficient

8

10

Figure 11: Sensitivity to β (DAD, T =10). Extending to larger values of β compared to Fig. 9b. Trained under objective (8). Errors ± 1 std err but are too small to be seen.

In Fig. 12 we demonstrate the choice of β and its impact in preserving the target accuracy of θ in terms of the ratio of EIG in the corresponding parameter for a policy trained under FairBED vs a policy trained to target EIGθ with sPCE objective. Increasing β encourages unlearning along ϕ, at the cost of reduced information gain along θ. 22

θ EIG: FairBED / PCE objective

Unconditional Conditional

1.0

β = 0.1

β = 0.1

0.8

0.6 β=1 β=1

0.4

β = 10

0.2

0.0

β = 10

0.0

0.2

0.4

0.6

0.8

1.0

φ EIG: FairBED / PCE objective Figure 12: Sensitivity to β in the Source Location Finding experiment with a static policy (T = 10) under conditional objective (8) and unconditional objective (9). Increasing β promotes unlearning along ϕ at the expense of information gain along θ. The top left of the chart is the ideal region. Axes denote the ratio of EIG in the corresponding parameter for a policy trained under FairBED vs a policy trained to target EIGθ with sPCE objective. Y = x line denotes points where reduction in EIGθ is equivalent to reduction in EIGϕ . Errors ± 1 std err.

Percentage change in EIG (%)

Sequential horizon In Fig. 13 we evaluate the relationship between the number of experiments T and lower bound EIG estimates for θ and ϕ for the static design when β = 0.5. We observe a similar pattern, with significantly larger relative drops in sensitive-attribute EIGϕ than in target-attribute EIGθ . 0 −20 −40 θ param φ param

−60 3

6 9 12 15 Number of Experiments, T

18

Figure 13: Sensitivity to horizon T (static, β=0.5). Plots show lower-bound EIG estimates. Error bars ± 1 std err.

23

84

Target 1 Test Time Accuracy (%)

Number of Labels 100 400

82 80

β coefficient

10 3 1 0.1 EPIG Random

78 76 0.5

0.6 DP Ratio

0.7

Figure 14: Varying β on Census. Accuracy vs DP ratio F.2

Active Learning: Census

Fig. 15 presents target and sensitive attribute accuracy across an increasing number of acquired labels for varying β and for random, HFE and FairBED acquisition. As label budget increases, predictive accuracy plateaus for each attribute and reaches stability. As β increases, FairBED generally maintains predictive accuracy in the target attribute and reduces accuracy in the sensitive attribute compared to baselines. Performance between HFE and FairBED is generally on par here however, FairBED has greater DP performance (Fig. 16).

24

Random

FairBED

HFE Target 2 (yφ) Mean Accuracy (%)

Target 1 (yθ ) Mean Accuracy (%) β=10.0

80 70 60 50

β=3.0

80 70 60

β=1.0

80 70 60 50

β=0.1

80 70 60 50

β=0.0

80 70 60 0

100

200 300 Number of Labels

400

0

100

200 300 Number of Labels

400

Figure 15: Census Income active learning curves (RF evaluation). Mean test accuracy versus acquisition iteration for the target (income; left column) and the sensitive target (gender; right column), under random label sampling (blue) and FairBED-based acquisition (orange). Each row corresponds to a different β coefficient (with β = 0 recovering standard EPIG). Curves are averaged over 8 seeds with shaded regions indicating ±1 standard error. All other settings follow the Census setup described in Section 7 (label budget 400, batch size 1).

25

In Fig. 16 we evaluate the DP ratio for an increasing number of labels. As β increases, data acquired using FairBED has an improved DP ratio (closer to 1) compared to random acquisition and HFE. Notably data acquired under FairBED significantly improved the DP ratio over HFE when fairness is targeted more strongly (e.g. larger β values). Random

FairBED

HFE

β = 10.0 0.75 0.60 0.45

β = 3.0 0.75

Mean Demographic Parity Ratio

0.60 0.45

β = 1.0 0.75 0.60 0.45

β = 0.1 0.75 0.60 0.45

β = 0.0 0.75 0.60 0.45 0

50

100

150 200 250 Number of Labels

300

350

400

Figure 16: Census. Comparing DP ratio with number of acquired labels for different acquisition strategies. Ratio of 1 is the target. β = 0 Corresponds to EPIG (Orange) and Predictive entropy (Green) baselines.

26

F.3

Active Learning: Student Graduation

Fig. 17 presents target and sensitive attribute accuracy across an increasing number of labels for varying β and for random, HFE and FairBED acquisition. We once again demonstrate that predictive accuracy plateaus for each attribute as the label budget increases. As β increases, FairBED generally maintains predictive accuracy in the target attribute and significantly reduces predictive accuracy for the sensitive attribute. Performance between HFE and FairBED is generally on par here however, FairBED has greater DP performance (Fig. 18). Random

FairBED

HFE Target 2 (yφ) Mean Accuracy (%)

Target 1 (yθ ) Mean Accuracy (%) β=10.0

80 70 60

β=3.0

80 70 60 50

β=1.0

80 70 60 50

β=0.1

80 70 60

β=0.0

80 70 60 50 0

100

200 300 Number of Labels

400

0

100

200 300 Number of Labels

400

Figure 17: Student Graduation active learning curves (RF evaluation). Mean test accuracy versus acquisition iteration for the target (graduated or not; left column) and the sensitive target (gender; right column), under random label sampling (blue), FairBED-based acquisition (orange) and HFE-based acquisition (green). Each row corresponds to a different β coefficient (with β = 0 recovering standard EPIG for FairBED). Curves are averaged over 8 seeds with shaded regions indicating ±1 standard error. All other settings follow the setup described in Section 7 (label budget 400, batch size 1).

27

In Fig. 18 we evaluate the DP ratio for an increasing number of labels. Similarly to Fig. 16, FairBED improves upon the DP ratio of acquired data compared to random and HFE acquisition. Random

FairBED

HFE

β = 10.0 0.8 0.7 0.6 0.5

β = 3.0 0.8

Mean Demographic Parity Ratio

0.7 0.6 0.5

β = 1.0 0.8 0.7 0.6 0.5

β = 0.1 0.8 0.7 0.6 0.5

β = 0.0 0.8 0.7 0.6 0.5

0

50

100

150 200 250 Number of Labels

300

350

400

Figure 18: Student Graduation. Comparing DP ratio with number of acquired labels for different acquisition strategies. Ratio of 1 is the target. β = 0 Corresponds to EPIG (Orange) and Predictive entropy (Green) baselines.

28

F.4

Active Learning: CelebA

Dataset. CelebA [54] consists of N =202,599 aligned and cropped celebrity face images, each annotated with 40 binary attributes. Following standard practice in the CelebA fairness literature, we use the Smiling attribute as the binary prediction target and the Gender attribute as the binary sensitive attribute. Image preprocessing and feature extraction. CelebA is high-dimensional and non-tabular, so we follow the common protocol of representing each image by an embedding from an ImageNetpretrained convolutional network rather than the raw pixels. Images are resized so the shorter side is 224, centre-cropped to 224×224, converted to tensors, and normalised using the standard ImageNet statistics (µ=(0.485, 0.456, 0.406), σ=(0.229, 0.224, 0.225)). Each image is then passed through a ResNet-18 [34] initialised with the default torchvision ImageNet-1k pretrained weights [61, 65]. The final classification head is replaced by an identity mapping, so the network outputs the 512dimensional penultimate-layer embedding. Forward passes are executed in evaluation mode with batch size 256 and no gradient tracking. Features are pre-computed once and cached to disk in chunks; the resulting X ∈ R202,599×512 matrix together with the 40-dim attribute matrix is the input to the active-learning loop. No fine-tuning of the backbone is performed: the backbone is fixed and the active learning operates entirely on top of the cached embeddings, isolating the contribution of the FairBED acquisition rule from any representation learning effects. Active learning protocol. We mirror the protocol used for Census and Student Graduation (Sec. F.2, Sec. F.3), modified only to accommodate the larger dataset. For each seed: • The N =202,599 embeddings are split into a pool and a held-out test set using and stratified on the (Smiling) target. • At each acquisition iteration, two random forest acquisition models (one per target, 100 trees each) are refit from scratch on all labels acquired so far. This is one choice of instantiating the sensitive predictor. • We run with a label budget of 300 and acquisition batch size B=2, repeated over 5 random seeds; reported error bars are ±1 standard error across seeds. The β value used for the headline DP/EO numbers in Tab. 2 is β=10, matching the other AL benchmarks. Compute. ResNet-18 feature extraction over the full 202,599 images takes a single pass on one GPU and is cached for all subsequent runs. Each AL run is then cheap: forward passes are not required during the AL loop because features are pre-computed, and acquisition models are random forests. Accuracy and fairness summary. Tab. 7 reports test-time accuracy on the target (Smiling) and sensitive predictor (Gender), alongside the DP and EO ratios (as in Tab. 2). The FairBED (β=10) trades a small drop in target accuracy for a substantial drop in sensitive predictability. This pattern is characteristic of FairBED’s mechanism: degrading the predictability of the sensitive attribute in exchange for a modest reduction in target accuracy. Table 7: CelebA. Target (yθ ) and sensitive (yϕ ) accuracy together with DP and EO ratios at a label budget of 300, batch size 2, β=10 for HFE and FairBED. Accuracy reported over 5 seeds, fairness ratios reproduced from Tab. 2. A ratio of 1 is optimal; lower yϕ accuracy is desirable.

Method Random Predictive Entropy EPIG HFE FairBED-EPIG

yθ Acc.

yϕ Acc.

DP

EO-TPR

EO-FPR

0.693 ± 0.005 0.679 ± 0.004 0.689 ± 0.004 0.631 ± 0.015 0.645 ± 0.013

0.890 ± 0.002 0.890 ± 0.002 0.890 ± 0.001 0.685 ± 0.037 0.684 ± 0.039

0.65 ± 0.09 0.77 ± 0.08 0.80 ± 0.02 0.82 ± 0.07 0.83 ± 0.03

0.70 ± 0.08 0.80 ± 0.07 0.84 ± 0.02 0.90 ± 0.04 0.90 ± 0.06

0.71 ± 0.10 0.82 ± 0.08 0.89 ± 0.05 0.61 ± 0.03 0.77 ± 0.09

29

G

Fairness definitions

In this section we present fairness definitions in terms of a binary classifier with true labels Y = {0, 1}, predicted labels Ŷ = {0, 1} and sensitive attribute Φ = {ϕ1 , ϕ2 }. Predictive Parity [22] Within supervised learning, predictive parity requires equal precision across sensitive groups, i.e., the probability that the predicted positive instances are actually positive should be equal across sensitive attributes. P (Y = 1 | Ŷ = 1, Φ = ϕ1 ) = P (Y = 1 | Ŷ = 1, Φ = ϕ2 ),

(45)

where Y are the true labels and Ŷ are the predicted labels. This fairness metric is increasingly prioritized within healthcare [30]. Additionally, it is simple to extend this fairness definition to a continuous parameter setting as it implies an invariance across a subset of the sensitive attribute space. Thus, this fairness definition more naturally fits within fair decision making problems compared to fair parameter inference problems. Equalised Odds [32] A predictor satisfies equalised odds if its true positive rate and false positive rate are equal across sensitive groups. Formally, for binary outcomes Y ∈ {0, 1} and sensitive attribute Φ: P (Ŷ = 1 | Y = y, Φ = ϕ1 ) = P (Ŷ = 1 | Y = y, Φ = ϕ2 )

for y ∈ {0, 1}

(46)

Equalised odds asks for parity in error behavior, not parity in selection rate. It is about fairness of mistakes relative to the ground truth. Equal Opportunity [32] Equal opportunity is a relaxation of equalized odds, focusing only on the true positive rate. Formally, P (Ŷ = 1 | Y = 1, Φ = ϕ1 ) = P (Ŷ = 1 | Y = 1, Φ = ϕ2 ).

(47)

This ensures that individuals who truly belong to the positive class have equal chances of being correctly predicted as such, irrespective of group membership.

H Impossibility result: Trade-off between Demographic Parity and Equalized Odds. Let Φ = {ϕ1 , ϕ2 } denote a binary sensitive attribute, Y ∈ {0, 1} the true label, and Ŷ ∈ {0, 1} the prediction of a classifier. Demographic parity requires that predictions be independent of the sensitive attribute: Ŷ ⊥ Φ,

i.e.,

P (Ŷ = 1 | Φ = ϕ1 ) = P (Ŷ = 1 | Φ = ϕ2 ).

Equalized odds requires that predictions be conditionally independent of the sensitive attribute given the true label: Ŷ ⊥ Φ | Y,

i.e.,

P (Ŷ = 1 | Y = y, Φ = ϕ1 ) = P (Ŷ = 1 | Y = y, Φ = ϕ2 )

∀y ∈ {0, 1}.

In general, these two fairness criteria are incompatible when base rates differ across groups, i.e., P (Y = 1 | Φ = ϕ1 ) ̸= P (Y = 1 | Φ = ϕ2 ). Specifically, except in degenerate cases (trivial classifiers, or equal base rates), no nontrivial classifier can simultaneously satisfy both demographic parity and equalized odds [38]. This incompatibility reflects a fundamental trade-off between equalizing prediction rates across groups and equalizing error rates across groups. 30

Table 8: Census ROC post-processing on randomly acquired data ( N = 400, 8 seeds, AIF360 default band [−0.05, 0.05]). FairBED at β = 10. Ratio of 1 is optimal. Method Acc (t1) DP Ratio EO Ratio - TPR EO Ratio - FPR Random ROC ([−0.05, 0.05]) FairBED - EPIG (14)

I

0.82 ± 0.00 0.75 ± 0.01 0.76 ± 0.01

0.55 ± 0.02 0.88 ± 0.05 0.68 ± 0.06

0.88 ± 0.01 0.90 ± 0.01 0.93 ± 0.02

0.72 ± 0.04 0.56 ± 0.06 0.86 ± 0.03

Reject Option Classification (ROC)

ROC [45] is a post-processing fairness intervention. We include it not as a competitor to FairBED but as a complementary intervention at a different point in the pipeline: FairBED shapes which data is collected, ROC re-labels predictions after a model has been fit. The two are orthogonal and can be composed. Method. Given a probabilistic classifier f (x) = p(ŷ=1 | x), ROC defines a critical region of low-confidence predictions, |f (x) − t| < δ, for threshold t ∈ (0, 1) and margin δ > 0. Inside this region the prediction is overridden using the sensitive attribute: instances from the unprivileged group are assigned the favourable label, instances from the privileged group the unfavourable one. Outside the region, the standard decision 1{f (x)>t} is used. The pair (t, δ) is selected on a held-out set to maximise balanced accuracy subject to a fairness constraint. In this work we target Demographic Parity (DP) with the AIF360-default band [9] [−0.05, 0.05] on the DP difference. ROC conflates model defects with real-world differences. DP is the constraint p(z=1 | ϕ=ϕ1 ) = p(z=1 | ϕ=ϕ2 ), which expands to Z p(z=1 | ϕ) = p(z=1 | x, ϕ) p(x | ϕ) dx. A DP gap can therefore arise from two distinct sources: (i) the conditional predictor p(z=1 | x, ϕ) depending on ϕ, a genuine biased model, or (ii) the input distribution p(x | ϕ) differing across groups, reflecting a real-world distributional difference that the predictor has correctly captured. These are fundamentally different problems and demand different responses, but ROC cannot distinguish them: it operates on the marginal p(z=1 | ϕ) and applies the same group-conditional flip regardless of which source the gap originates from. In doing so it conflates correcting a biased model with overwriting an accurate one, destroys the diagnostic that would let a practitioner tell the two apart, and leaves no record of which (if any) was actually wrong. The intervention is opaque by construction. Upstream methods like FairBED avoid this failure mode entirely by acting on the data-generating process, where the locus of fairness is explicit and inspectable. Design choices. We integrate ROC into the FairBED active-learning setting as follows. The labelled budget (N =400 samples) is the only data with sensitive labels available, so ROC’s validation set must be carved out of it: we use a stratified 70/30 split on the joint (yθ , yϕ ) label, giving roughly 280 train and 120 val samples. The base classifier is a Random Forest with 100 trees, matching the rest of the pipeline so that any difference between RF and RF+ROC isolates the post-processing effect rather than the choice of estimator. We search (t, δ) on a 100 × 50 grid over t ∈ [0.01, 0.99] and δ ∈ [0, min(t, 1 − t)], the AIF360 reference grid [9]. Drawbacks. Two further limitations of ROC are worth noting. First, it requires sensitive labels on a validation set at fit time. Second, ROC only adjusts the decision threshold on a fixed score function: it can flip predictions near the boundary but cannot recover from a base classifier whose scores are themselves miscalibrated or insufficiently informative for the unprivileged group. When the base classifier is uninformative on a group, ROC’s flips trade off accuracy without restoring useful predictive structure, a regime in which upstream interventions like FairBED are necessary rather than optional.

Comparison to a downstream baseline (Census, ROC). Table 8 situates FairBED alongside Random+ROC at the AIF360-default constraint band [−0.05, 0.05]. These results are to demonstrate that approaches at the data acquisition stage can still be performant for downstream notions of fairness. ROC’s validation set is drawn from a stratified split of the labelled subset. 31

J

Learned Fair Representations (LFR)

LFR [79] is a pre-processing fairness intervention. As with ROC, we include it not as a competitor to FairBED but as a complementary intervention at a different point in the pipeline: FairBED shapes which data is collected, LFR re-encodes the features of an already-collected dataset before a downstream model is fit. The two are orthogonal and can be composed (Fig. 8). Method. LFR maps each input x to a probability distribution over K learned prototypes {vk }K k=1 via a softmax of negative distances, P (Z=k | x) ∝ exp(−d(x, vk )), and then to a label via a learned linear head w on the soft assignments. Three terms are jointly optimised: a reconstruction loss Lx (the prototypes should preserve information about x), a prediction loss Ly (the head should predict P the target), and a fairness loss Lz = k | Mk+ − Mk− | penalising any difference between the average prototype-assignment vector for the privileged and unprivileged groups. The fairness loss enforces statistical parity of the representation itself, with weight Az controlling the strength of the constraint. Downstream models are then trained on the soft-assignment vectors instead of the raw features. Design choices. We integrate LFR into the FairBED active-learning setting as follows. The labelled budget (N =400 samples) is the only data with sensitive labels available, and LFR is fit on this set alone. We use K=5 prototypes and the LFR-default hyperparameters Ax =0.01, Ay =1.0, with Az swept across {0.1, 1, 10}. Optimisation uses PyTorch L-BFGS with autograd over the joint objective and runs for up to 5000 steps; inputs are standardised before prototype fitting. Downstream models are then trained on the resulting soft-assignment vectors, matching the rest of the pipeline so that any difference between RF and LFR+RF isolates the representation effect rather than the choice of estimator.

32