Actions Have Consequences: Detecting Outcome Performativity using Intervention Testing

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Actions Have Consequences: Detecting Outcome Performativity using Intervention Testing

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

ACTIONS H AVE C ONSEQUENCES : D ETECTING O UTCOME P ERFORMATIVITY U SING I NTERVENTION T ESTING - E XTENDED V ERSION

arXiv:2607.26908v1 [cs.LG] 29 Jul 2026

Brandon Gower-Winter and Georg Krempl Utrecht University 8 Hiedelberglaan, Utrecht, 3584 CS, NL [email protected], [email protected]

A BSTRACT In many domains such as Palliative Care, Credit Assignment and Recommender Systems, predictions may causally influence the outcomes they predict. This phenomena is known as Outcome Performativity. This paper formalises an approach for detecting Outcome Performativity using prediction intervention called Outcome Performativity A/B Detection (OPAB). OPAB enables the detection of Outcome Performativity by assessing the dissimilarity in outcome distributions produced by different predictions groups (interventions). If that dissimilarity is significant, Outcome Performativity is detected. We derive sample complexity bounds for OPAB under various Outcome Performative assumption classes which we empirically validate. Results show that detecting Outcome Performativity using OPAB is achievable in numerous cases. Results also show the presence of regions of indistinguishability which describe settings where the allotted number of interventions are insufficient for detecting Outcome Performativity. The results of which have broader practical implications for the detectability of Outcome Performativity in settings where samples are scarce, cost-prohibitive or potentially unethical to obtain. The paper concludes with a case study on the efficacy of OPAB on the Open Bandits dataset, and provides directions for future work.

Introduction

It is often assumed that predictions made by Machine Learning (ML) models have no effect on the outcomes they observe, or the data distributions they encounter in the future. However, a growing body of research under the term Performative Prediction [1] has shown that this is not the case and that many problem domains such as Credit Assignment [2] or Patient Care prediction [3, 4] may exhibit these performative phenomena. In the former, assigning high interest rates to high risk debtors may increase the likelihood that they default on their loans. In the latter, assigning palliative care to a patient (instead of curative care) may ultimately lead to their death. This work focuses on a specific type of Performativity known as Outcome Performativity [5] where predictions made by ML models causally affect the outcomes observed (Fig. 1a). The state-of-art often assumes the Outcome Performativeness of a setting is known a priori. However, it has been shown that ML algorithms can induce performative feedback loops inconspicuously and may produce unwanted phenomena such as an increased False Positive Rate (FPR) over time [3]. This motivates the need to detect such phenomena. If performativity is undetected, practitioners may unknowingly deploy models that exacerbate bias or suffer from performance decay. Despite the existence of online methods for identifying performative effects [2, 6], there has been little work studying the identifiability of Outcome Performativity in an offline setting (i.e., during the data collection process). If such an offline detection method exists, it would allow practitioners to identify Outcome Performativity before a predictive model is deployed, which may be more ethically and financially beneficial to an online approach which requires a trained and deployed model. Taking inspiration from Randomized Controlled Trials (RCTs) [7] and work on A/B (Intervention) testing to detect performative feedback loops [6], we apply intervention testing [8] to detect Outcome Performativity (Fig. 1b). By

Anonymous

do(Ŷ)

(a)

(b)

Figure 1: Causal diagrams of Outcome Performative with (a) no interventions and (b) OPAB (our method) applied. Here, the outcome (label) Y of a prediction is causally dependent on both the features X and prediction Ŷ itself. observing the outcome distributions that arise from randomly assigning predictions to instances, we causally assess the dissimilarity of said distributions and, if significant, detect Outcome Performativity. This approach is called Outcome Performativity A/B Detection (OPAB). In contrast to contemporary methods, OPAB is an offline method which enables the detection of Outcome Performativity prior to a ML model’s training and deployment (i.e., detection occurs during the data labelling process). Given this, we are interested in evaluating the sample efficiency of OPAB. i.e., the ability to detect Outcome Performativity in as few samples as possible which is critical in settings where label acquisition is impractical. We investigate this by analysing the sample complexity of OPAB on several assumption classes of Outcome Performativity [3, 5], for which we derive sample complexity bounds. This work can be summarized by the following research questions: 1. Under what conditions can intervention testing be used to detect Outcome Performativity in binary classification tasks? For this, we explore three assumption classes of Outcome Performativity in various configurations. 2. To what degree can the number of instances (sample complexity) needed to detect Outcome Performativity be estimated? This is important for domains where intervention testing is costly or has to be minimized for ethical reasons. 3. Using the derived sample complexity bounds, can we identify regions of indistinguishability? These are settings where required sample complexity is prohibitive for detecting Outcome Performativity in a practical setting. The rest of the paper follows: Sect. 2 introduces related work, Sect. 3 describes the implementation of OPAB, Sect. 4 derives the sample complexity bounds of OPAB under different Outcome Performativity assumption classes, and Sect. 6 reflects on the strengths and limitations of OPAB. Sect. 7 concludes the paper. The source code for this work can be found at: https://edu.nl/mhh9d

Background and Related Work

Outcome Performativity was introduced by Mendler-Dünner et al. [9] and Kim and Perdomo [5]. Outcome Performativity is an extension of Performativity [1] whereby the realised outcome for some prediction task is causally dependent on both the features of an instance and the prediction given to that instance by some predictor (e.g., human expert, or ML model). Outcome Performative tasks fall into subset of traditional supervised learning tasks where a prediction must be made before the true outcome is observed. For example, in a credit assignment task, one might want to predict whether an applicant would become insolvent in the near future. In this setting, predicting the likelihood of insolvency may affect whether that state of insolvency is realised. If a loan is granted, the applicant may have enough funds to stay operational in the near future. If the applicant is not granted a loan, that may increase the likelihood that they go bankrupt in that same time period. Formally, a binary classification Outcome Performative task is described as: p∗ : X × Yb → [0, 1]

(1)

Here, an outcome y from outcome space Y = {0, 1} is determined by a Bernoulli distribution parametrized by p∗ (x, ŷ) for every instance x ∈ X and prediction ŷ ∈ Y pair. p∗ , also known as Nature [5], describes the true outcome distribution of the system. In practice, the dynamics of p∗ are unknown but some assumption classes exist in literature. Adam et al. [3, 10] adopt misclassification-based dynamics where an outcome y is only subject to Outcome Performativity if the instance x is misclassified. Their model requires that the notion of a desirable outcome can be described before a prediction is made. In the palliative care problem, it is desirable to offer palliative care to patients who would pass away even if curative interventions were administered because it maximises the quality of the patient’s 2

Detecting Outcome Performativity Using Intervention Testing

end-of-life care. If a curative intervention would save a patient’s life, then it is the desirable outcome because life is preferable to the risk of death. In Mendler-Dünner et al. [9], they use model-based dynamics of p∗ which adopts the notion that the Outcome Performative behaviour of instances close together in the feature space should behave similarly when given the same prediction assuming no hidden confounders. Outcome Performative phenomena are highly relevant in many practical domains. A review from Pagan et al. [11] categorized Outcome Performativity under the term Outcome Feedback Loops and highlighted the prevalence of Outcome Performativity in both adversarial and non-adversarial domains. Liley et al. [12] showed that naive updating of ML models can cause performance degradation in Outcome Performative settings which is particularly relevant in healthcare settings where even an accurate model can produce unwanted phenomena [4]. Furthermore, Adam et al. [3, 10] investigated the subtle consequences that arise if practitioners are not aware of the feedback loops induced by their ML models. They show that an over-reliance on the predictions made by a ML model results in an increased False Positive Rate if the model later trains on data it made predictions on.

Methodology

Before introducing OPAB, we motivate why intervention testing is important, and useful, when detecting Outcome Performativity. In an binary classification Outcome Performative setting, an instance x is given a prediction ŷ which results in the observation of some outcome y. Formally, we say that outcome y is sampled from a Bernoulli distribution parametrised by p∗ : y ∼ Bernoulli(p∗ (x, ŷ)) where p∗ , Nature, is defined as a mapping of instances and predictions to some probability of realising an outcome y = 1: p∗ : X × Yb → [0, 1]

(2)

While the former definitions neatly capture the underlying mechanisms of Outcome Performativity, they do not describe how one should seek to identify if a setting is Outcome Performative. Given this, we can view Outcome Performativity through the lens of the Bayes Theorem: P (Yb | X, Y )P (Y |X) P (Y | X, Yb ) = P (Yb | X)

(3)

where X, Yb , and Y are the observed instances, predictions and outcomes resp. To detect Outcome Performative effects, we look for the following conditions: H(P (Y | X, Yb = 0), P (Y | X, Yb = 1)) ≤ δ

(4)

where H is a hypothesis test that calculates the dissimilarity of the two distributions given some threshold δ which controls the type I error rate. Informally, Eq. 4 describes testing to see if the predictions Yb in some setting X, affect the realised outcomes Y . However, this is not sufficient to detect Outcome Performativity as we first need to examine from where the predictions Yb originate. During outcome data acquisition, or the training of a predictive model, initial predictions about outcomes will come from either a different predictive model or some domain expert ϕ (i.e., ŷ = ϕ(x)) Referring back to Eq. 4, this would mean that our naive test for Outcome Performativity would be: H(P (Y | X, ϕ(X) = 0), P (Y | X, ϕ(X) = 1)) ≤ δ

(5)

Herein lies the problem. Eq. 5, at most, allows one to tell if predictor ϕ’s predictions give rise to different outcome distributions, not if the setting itself is Outcome Performative. This is because the outputs of ϕ are themselves dependent on the instances X (Fig. 1a). To remedy this and break the dependency of Yb on X, we can take inspiration from A/B testing and randomised control trials (RCTs) and perform a do-intervention [8] on the predictions themselves. Note that this is equivalent to replacing the expert predictor ϕ with a random sampling strategy: ϕ(x) = U{0, 1} which decouples the dependency of Yb on X (Fig. 1b). This gives us the following: H(P (Y | X, do(Yb = 0)), P (Y | X, do(Yb = 1))) ≤ δ

(6)

Eq. 6 is enough to allow one to detect Outcome Performativity, it however requires the ability to learn some reliable approximation of P (Y | X , do(Yb )) using methods, such as T-learner [13] or Double ML [14], which are sensitive to 3

Anonymous

intervention imbalances, hyper-parameter and model choices [15]. Given that one of the paper’s goals is to take the first steps towards estimating the sample complexity bounds for Outcome Performative settings, we make an assumption that the Outcome Performative effects in a given setting are uniform across the feature space. This reduces Eq. 6 to: H(P (Y | do(Yb = 0)), P (Y | do(Yb = 1))) ≤ δ

(7)

which is achievable by simply performing interventions (i.e., random sampling predictions), and statistically evaluating the outcome distributions Y using an appropriate statistical test such as a Chi-Squared or Fisher’s Exact Test. As we show later, the assumption that Outcome Performativity is uniform across the feature space is reasonable for several assumption classes. We do, however, elaborate on the limitations of this assumption in Sect. 6. Sect. A provided an additional perspective on why intervention testing is necessary to detect Outcome Performativity. 3.1

Outcome Performative A/B Testing

This section describes the execution logic of Outcome Performative A/B testing (OPAB). Recall that in order test for Outcome Performativity, we need to perform do-interventions on the predictions Yb and observe if there are statistical differences in the outcome distributions of both interventions (Eq. 7). Formally, given a set of T instances X for some binary classification task Y = {0, 1} and a significance threshold δ for controlling the type I error, OPAB will randomly assign predictions ŷ ∈ Yb = {0, 1} ∀ x ∈ X sampled from a Bernoulli distribution (ŷ ∼ Bernoulli(0.5)) [5] (i.e., do-intervention). This results in two groups of size N . For each group of instances XYb =0 and XYb =1 (the A and B groups in A/B testing or the assigned groups in RCTs), OPAB observes the outcome y ∈ Y of each instance-prediction pair (x, ŷ), and then evaluates the dissimilarity of the label distributions P (Y | Yb = 0) and P (Y | Yb = 1) using Eq. 7. If H(P (Y | Yb = 0), P (Y | Yb = 1)) ≤ δ, then Outcome Performativity is detected. Alg. 1 provides the pseudocode. Algorithm 1 Pseudocode of OPAB. Here we assume that H is either a Chi-Squared or Fisher’s Exact Test. Require: Instances X, Function H, Threshold δ 1: A ← [0, 0] 2: B ← [0, 0] 3: for x ∈ X do 4: ŷ ← Bernoulli(0.5) 5: y ← observe(x, ŷ) 6: if ŷ = 0 then 7: A[y] ← A[y] + 1 8: else 9: B[y] ← B[y] + 1 10: end if 11: end for 12: return H(A, B) ≤ δ

▷ Construct frequency tables ▷ for groups A and B ▷ For each instance ▷ Assign random prediction ▷ Observe outcome of prediction ŷ on instance x ▷ Update frequency tables

▷ Return result of hypothesis test

Sample Complexity Analysis

Recall that in an Outcome Performative setting, an outcome y is sampled from a Bernoulli distribution parametrised by p∗ : X × Yb → [0, 1] where p∗ is Nature (i.e., y ∼ Bernoulli(p∗ (x, ŷ))). Nature is the true conditional outcome distribution that describes the underlying dynamics of an Outcome Performative setting. In reality, the actual dynamics of p∗ are unknown, but several assumption classes describing p∗ exist. Each subsequent subsection derives the sample complexity bounds for three of those assumption classes and empirically validates them using OPAB. Deriving sample complexity bounds for Outcome Performative settings is of practical importance because it allows practitioners to calculate the minimum sample size needed to reliably detect Outcome Performativity, which is vital in high-stakes domains like healthcare where the cost of intervention testing is either ethically ambiguous or prohibitively expensive. In order to empirically evaluate two of the assumption classes: Model-based (Sect. 4.2) [9], and Misclassification-based (Sect. 4.3) [3], real-data which is imputed with Outcome Performativity is required. The datasets used in this work are: breast cancer [16], diabetes [17], adult census [18], kickstarter [19], titanic [20], and loan [21]. They are all binary classification tasks. Unless stated otherwise, results reported in this section for every parameter combination are averaged over 100 replicates and a pseudorandom number generator is used to ensure reproducibility. We use the Chi-Squared Test to detect Outcome Performativity and δ = 0.05. A description of each dataset, the 4

Detecting Outcome Performativity Using Intervention Testing

source code for all experiments, and additional experimental results such as using different statistical tests to determine significance or varying the parameter δ, are can be found in the Appendix. 4.1

Simple Outcome Performativity

The first assumption class we investigate is a simplified model of Outcome Performativity. In this setting, we drop p∗ ’s dependence on the instance space X describing it as follows: p∗ =

α0 α1

if ŷ = 0 if ŷ = 1

(8)

where αc = P (y = 1 | ŷ = c) is the probability that the outcome is y = 1 when the prediction is ŷ = c ∈ {0, 1} (binary classification task). In this setting, a realized outcome y is only dependent on the prediction ŷ. We acknowledge that this assumption class is unrealistically simple, but we include it because it allows us identify key behavioural dynamics of Outcome Performativity as well as derive sample complexity bounds which are used as building blocks for deriving sample complexity bounds for more sophisticated Outcome Performative settings. In order to derive sample complexity bounds, we need to make assumptions about the hypothesis test H and the significance threshold δ, Assuming H is a Chi-Squared Test, the estimated minimum number of interventions per prediction group to detect Outcome Performativity is given by the following Eq. (see Sect. B for proof): N = χ2δ

2(α0 + α1 ) − (α0 + α1 )2 2(α0 − α1 )2

(9)

where χ2δ is the critical χ2 value for some significance threshold δ (e.g., this work uses δ = 0.05, therefore the χ2δ=0.05 with one degree of freedom is 3.841). Eq. 9 reveals several important findings: the first is that fewer instances are needed to detect Outcome Performativity as |α0 − α1 | → 1.0. This makes sense as the resulting outcome distributions will be increasingly different as the performative response of α0 and α1 differ greatly. Secondly, the number of instances required per intervention N → ∞ when α0 ≈ α1 . This means that for arbitrarily small differences in Outcome Performative responses per intervention, detecting Outcome Performativity becomes increasingly prohibitive. From a practical perspective, the derivation of sample complexity bounds for an Outcome Performative setting introduces another concept: regions of indistinguishability. Formally, given an allotted budget per intervention N̄ , confidence threshold δ, and assumption class A, the regions of indistinguishability are defined as the set of parametrisations of a ∈ A where the allotted interventions per group are less than the minimum required number of instances defined by the complexity bound Nδ,a : {a | a ∈ A and N̄ < Nδ,a }

(10)

Note that Nδ,a is proxy notation for a sample complexity bound calculation. For Simple Outcome Performativity, Nδ,a is Eq. 9 and a is some 2-tuple that parametrises the setting (i.e., a = (α0 , α1 )). However, the concept of the regions of indistinguishability in an Outcome Performative setting are both assumption class, and sample complexity bound agnostic. 4.1.1

Results

To empirically validate these findings, we run simulations of a Simple Outcome Performative setting over various α0 and α1 ∈ [0, 1] using OPAB (δ = 0.05) to detect the presence of Outcome Performativity. Fig. 2 reports the detection rate of OPAB over values of N̄ = {10, 100, 1000, 10000}. The detection rate reported in each cell is the number of times OPAB was able to detect Outcome Performativity divided by the total number of repeated runs (25 in this case). These results confirm the presence of regions of indistinguishability (Fig. 2, dark blue regions): As N̄ increases, the number of parameter combinations (α0 , α1 ) decreases proportionally. In the simple Outcome Performativity setting, when |α0 − α1 | is large, few instances (N̄ = 10) are required to reliably detect Outcome Performativity. However, when α0 ≈ α1 , even a large amount of instances (N̄ = 10 000) cannot detect Outcome Performativity. Interestingly, despite assuming a Chi-Squared test is used to detect Outcome Performativity, the estimated N produced by Eq. 9 is applicable when using other statistical tests to detect Outcome Performativity (Results using Fishers Exact Test and Mann-Whitney U Test are shown in Sect. D.3). 5

Anonymous

1.0

N = 10

1.0

N = 100

N = 1000

1.0

N = 10000

1.0

0.6

Detection Rate

0.8 0.6

0.6

0.4 0.2

0.2

0.8

0.4

0.0

0.8

0.4

0.0

0.8

0.4

0.0

0.8

0.4

0.0

0.0 0

Figure 2: shows the detection rate of OPAB in the Simple Outcome Performativity setting as the allotted interventions per group N increases. Dark blue values indicate regions of indistinguishability where, for the given N , Outcome performativity cannot be reliably detected. fθ

Feature 2

X + β.θ

Feature 1

Figure 3: A visualization of the Model-based Outcome Performativity Model on a two dimensional feature space. fθ is a linear model trained on a labelled dataset and its decision boundary is represented by the dotted line. The red region indicates where fθ (x) = 0 and the green region indicates where fθ (x) = 1. To simulate Outcome Performativity an instance x is moved in the feature space using the linear model’s weights θ and Outcome Performative strength β. The result of fθ (x + β.θ) is then the observed outcome Y . In this example, the instance would initially have an observed outcome of Y = 0, but because of the simulated Outcome Performativity, the observed outcome is Y = 1 instead. 4.2

Model-based Outcome Performativity

Recall that in Outcome Performative settings, the outcome of an instance-prediction pair is determined by p∗ : X × Yb → [0, 1]. The previous assumption class abstracted away X . This section reintroduces X with Model-based Outcome Performativity. In Model-based Outcome Performativity [9], an instance’s position in the feature space is taken into consideration before applying any Outcome Performative Effects. More specifically, Model-based Outcome Performativity assumes that p∗ can be represented by a linear model fθ → {0, 1} and its weights θ. Using fθ and θ, Outcome Performativity is modelled as: p∗ = fθ (x + βŷ .θ)

(11)

where βŷ is the strength of the Outcome Performativity for prediction ŷ = c. If βc < 0.0, it biases instances towards an outcome of y = 0. If βc > 0.0, it biases instances towards an outcome of y = 1. If βc = 0.0, predictions are nonOutcome Performative. A visualization of Model-based Outcome Performativity is shown in Fig. 3. Sample complexity analysis shows that Model-based Outcome Performativity behaves similarly to Simple Outcome Performativity (Eq. 9,) despite the introduction of the feature space X (see Sect. B for proof): 6

Detecting Outcome Performativity Using Intervention Testing

Breast Cancer (N = 28) Diabetes (N = 38) Adult Census (N = 1628) Kickstarter (N = 5406) 1.0 1.0 1.0

-1.0

0.2 0.0

-0.

-1.

-0.

-1.

-0.

-1.

-0.

-1.

Detection Rate

-1.0

0.4

1.0

-0.5

0.5

-0.5

0.6

5 0.0

-0.5

1.0 0.8

-0.5

1.0

0.0

5 0.0 0.5

0.0

1.0

0.0

5 0.0 0.5

0.5

1.0

0.5

5 0.0 0.5

0.5

1.0

Figure 4: shows the detection rate of OPAB in Model-based Outcome Performativity settings for different parameter combinations (β0 , β1 ) over several datasets. Dark blue parameter combinations reveal regions of indistinguishability for given datasets given the allotted interventions per group N .

N = χ2δ

2(pβ0 + pβ1 ) − (pβ0 + pβ1 )2 2(pβ0 − pβ1 )2

(12)

where pβc = P (Y = 1 | Yb = c , θ) is the proportion of instances that would have an outcome of y = 1 given a prediction of class ŷ = c when using model fθ . When |β0 − β1 | → ∞, the total number of instances N needed per intervention group → 0.0. Conversely, when |β0 − β1 | → 0.0, N → ∞. The caveat is that the rate at which this happens depends on the parametrisation of fθ , which we show in the next section. 4.2.1

Results

In order to simulate Model-based Outcome Performativity, a predictive model fθ and its parameters θ are needed. We achieve this by using non-Outcome Performative binary classification dataset (e.g., Diabetes), and train fθ on that dataset. This enables one to impute the non-Outcome Performative dataset with Model-based Outcome Performativity. This approach preserves class imbalances, and feature densities from the original non-Outcome Performative dataset. In order to simulate Outcome Performativity, one samples an instance x from the original dataset, assigns a prediction ŷ and applies Eq. 11 to observe the final outcome y. To evaluate the sample complexity bounds derived in Eq. 12, we evaluate the detection rate of OPAB on the datasets described in Sect. 4 imputed with Model-based Outcome Performativity. When evaluating OPAB we randomly sample 10% of the instances from each dataset without replacement. We do this because it is unreasonable in many cases to assume that some large percentage of the total available instances would undergo intervention testing (i.e., intervention testing could be cost prohibitive or unethical at large scales). Fig. 4 illustrates the results of this experiment where we find that as N increases, the detection rate of OPAB increases. This is most clearly shown across the breast cancer and adult census datasets where for the latter, OPAB reliably detects Outcome Performativity across a range of β0 and β1 parameter values. For the former, the range of β0 and β1 parameters that OPAB detects Outcome Performativity for decreases (see Sect. D.5). Results also confirm non-uniform Outcome Performativity responses across the datasets. The breast cancer dataset illustrates this where a class imbalance in the training dataset (bias towards Y = 1) causes non-uniformity in the number of interventions needed per class N . As βc → 1.0 (the biased class), N increases as the outcome distributions rapidly converge. The original breast cancer dataset is imbalanced favouring benign samples (Y = 1) which means that as β0 and β1 → 1.0, fθ more rapidly produces identical distributions across both prediction groups. This effect is seen less intensely in the diabetes setting where the original dataset is slightly imbalanced favouring Y = 0. These results further support the presence of regions of indistinguishability (Fig. 4, dark blue regions). Unlike the Simple Outcome Performative setting (Sect. 4.1), Model-based Outcome Performative settings reveal how imbalances in the Outcome Distributions and feature space density can significantly increase the number of instances required to detect Outcome Performativity. This is shown in Fig. 4 where certain combination of (β0 , β1 ) are trivially easy to detect, but others are practically infeasible with the allotted interventions (N̄ ). 4.3

Misclassification-based Outcome Performativity

The final assumption class we evaluate is Misclassification-based Outcome Performativity [3, 10] where p∗ , Nature, is described as: 7

Anonymous

1.0

Breast Cancer (N = 28) Diabetes (N = 38) Adult Census (N = 1628) Kickstarter (N = 5406) 1.0 1.0 1.0

1.0

Detection Rate

0.6 0.4

0.8

0.2

0.4

0.0

0.8

0.2

0.4

0.2

0.0

0.2

0.8

0.2

0.4

0.6

0.0

0.6

0.8

0.6

0.4

0.6

0.0

0.8

Figure 5: shows the detection rate of OPAB in Misclassification-based Outcome Performativity settings. Black cells indicate parameter combinations (γ0 , γ1 ) in the regions of indistinguishability for the allotted interventions per class N .

 1.0    0.0 p∗ = γ 1   1 − γ0

if ŷ = 1 and ȳ = 1 if ŷ = 0 and ȳ = 0 if ŷ ̸= 1 and ȳ = 1 if ŷ ̸= 0 and ȳ = 0

(13)

Unlike the previous two assumption classes, Misclassification Outcome Performativity only occurs when some prediction ŷ is not equal to some desirable outcome ȳ = c. What constitutes a desirable outcome is domain specific. In the palliative care problem, it is desirable to give curative care to patients who are likely to survive, whereas palliative is desirable when patients will pass away regardless of any type of medical intervention. In the credit assignment problem, it is desirable to grant credit to entities who will be able to pay the debt back. In Misclassification Outcome Performativity, when a mismatch between a prediction ŷ and desirable outcome ȳ occur, the probability the desired outcome is realized despite misclassification (i.e., y = c) is defined by the probability γc ∈ [0, 1]. When γc = 0, desired outcomes are never realized upon misclassification and when γc = 1, the desired outcome is always realized despite misclassification (i.e. no Outcome Performativity). To derive sample complexity bounds for Misclassification Outcome Performativity, we again assume H is the ChiSquared Test. The estimated number of interventions needed per class N is: N = −χ2δ

(γ0 p0 + γ1 p0 − γ1 − 1)(γ0 p0 + γ1 p0 − γ1 + 1) 2 × (γ0 p0 − γ1 p0 + γ1 − 1)2

(14)

Where p0 is the probability of encountering a desired outcome ȳ = 0 when randomly selecting an instance and desired outcome pair (x, ȳ). See Sect. B for proof deriving Eq. 14 and visual aid for different p0 estimates. Eq. 14 shows that as γ0 and γ1 → 1.0, N → ∞ meaning that weaker Outcome Performativity is harder to detect. This makes sense because as γc → 0.0, more instances will not realize their desired outcomes when misclassified which creates increasingly different outcome distributions. Conversely, as γc → 1.0 fewer instances will not realize their desired outcomes resulting in increasingly similar outcome distributions. 4.3.1

Results

We empirically validate Eq. 14, by applying OPAB to the datasets described before imputed with Misclassification Outcome Performativity. To achieve this, the set of desirable outcomes Y is determined by the dataset’s original labels prior to being imputed with Outcome Performativity. If a prediction mismatches the desired outcome, we simulate Misclassification Outcome Performativity using Eq. 13. Fig. 5 shows the results of four datasets (others are included in the appendix). We allocate 10% of the total dataset for detecting Outcome Performativity with each instance being randomly sampled without replacement. Empirical results confirm the theoretical findings of Eq. 14. When N is large, OPAB can detect Outcome Performativity over most γ0 and γ1 values investigated. The exception being when γ0 = γ1 = 1.0 which is expected as this parameter combination is non-Outcome Performative. Interestingly, regions of indistinguishability (Fig. 5, black regions) seem less common in the Misclassification-based Outcome Performativity setting. This is expected because there is only one 8

Detecting Outcome Performativity Using Intervention Testing

Table 1: Detection Rate of OP-AB across varying sample sizes (N ) on the OBD datasets. These results show that OP-AB has a low false detection rate as shown on the non-Outcome performative women dataset and a increased efficacy as N increases as shown the Outcome Performative men dataset dataset / N

102

103

104

105

women men

0.0 0.0

0.001 0.0

0.042 0.05

0.042 0.094

0.016 0.523

Table 2: Detection Rate of OP-AB across varying sample sizes (N ) on balanced variants of the OBD datasets. The results show that OP-AB is more sample efficient when the outcome distribution is balanced. However, the detectability of Outcome Performativity in ultimately dependent on performative strength (effect size) which is negligible in these datasets. dataset / N 10 50 100 500 1000 women men

0.046 0.044

0.059 0.068

0.06 0.074

0.051 0.159

0.032 0.234

parameter combination (γ0 = 1.0, γ1 = 1.0) which is truly Outcome Performative as opposed to the infinite parameter combinations in the Simple (α0 = α1 ) and Model-based (β0 = β1 ) Outcome Performative settings. Given this, our analyses suggest that detecting Misclassification-based Outcome Performativity (using OPAB or otherwise) requires fewer interventions than the other assumption classes.

Case Study: Open Bandits Dataset

To the best of our knowledge, no dataset exists which explicitly studies Outcome Performativity. However, Outcome Performativity can be inferred if there are instances for which (1) predictions are recorded and (2) these predictions are randomly assigned. The Open Bandits Dataset (OBD) [22] meets these criteria. OBD was constructed using multi-armed bandits on the fashion e-commerce platform ZOZOTOWN for the off-policy evaluation of recommender systems. Each instance represents a user impression containing feature values, item recommendations and their placement (clothing items to choose from and where they appear on the store webpage), and click indicators (was the recommended clothing item clicked on) as an outcome. For a subset of the data, the placement of each recommended item on the store page (left, centre, or right) was determined using random sampling. We use this subset and repurpose the placement of item recommendation as predictions Yb and click indications as outcomes Y . The dataset is also divided into sub-datasets for both men and women. Applying OP-AB to all the instances in both of these sub-datasets we find Outcome Performativity in men (p = 0.014), but not in women (p = 0.87). We cannot speak to gender-based purchasing habits, but these results would indicate that the order in which recommended items appear on the ZOZOTOWN platform are not necessarily relevant for women (not outcome performative), but are relevant for men (outcome performative). Nevertheless, we use the men dataset to study the efficacy of OP-AB and the women dataset to study the prevalence of false positives produced by OP-AB across various N . Sect. C give a complete description of the dataset and all pre-processing steps. We first evaluated OP-AB (δ = 0.05) on the OBD datasets by taking random samples (without replacement) of size N = {10, 102 , 103 , 104 , 105 } over 1000 replicates. The detection rate reported in Table 1 shows that OP-AB has a low false positive of about [0.0, 0.05] in the women dataset. Conversely, OP-AB seemingly struggles to detect Outcome Performativity in the men dataset only being 50% accurate at the largest N = 105 . This occurs for two reasons: (1) the Outcome Performative effect in this dataset is quite small and thus harder to detect which requires exponentially more instances with random predictions. This claim is supported by the low Phi coefficient (Φ = 0.004) which indicates a negligible effect size and is a limitation of this dataset (2) the class imbalance of the dataset makes it harder for Outcome Performative effects to be observed. The OBD datasets are heavily imbalanced with only about 0.05% of all item recommendation resulting in a click in our setup. To investigate the potential class imbalance further, we created balanced datasets of both the women and men datasets by randomly downsampling the majority class (without replacement) over 1000 replicates and repeating the experiment over N = {10, 50, 100, 500, 1000} (sample size N must decrease as the total dataset decreases in size). The detection rates are reported in Table 2 and confirm that OP-AB is more effective in settings where the outcome distribution P (Y ) 9

Anonymous

is balanced. For example, OP-AB was four times more effective at detecting Outcome Performativity when N = 103 on the balanced men dataset.

Discussion

This work formalised the act of performing intervention testing on predictions in order to detect Outcome Performativity. This method, OPAB, was evaluated on several theoretical models (assumption classes) of Outcome Performativity for which we also derived sample complexity bounds (i.e., the minimum number of interventions per class needed to detect Outcome Performativity). To the best of our knowledge, there exist no other works which explicitly derive sample complexity bounds for detecting Outcome Performativity. Given this, we believe our work provides a practical contribution to the state-of-the-art. The implications of these findings are threefold: First, by formally describing the process of performing intervention testing on predictions for binary classification tasks, OPAB enables practitioners to detect Outcome Performativity in real-world domains. This contrasts with related works which have only looked at the identifiability of Outcome Performativity in purely theoretical or semi-synthetic domains. To support the claim that OPAB can be used to detect Outcome Performativity in real-world settings, we conducted a case study on the Open Bandits Dataset [22]. Second, OPAB is an offline algorithm, This enables practitioners to detect Outcome Performativity before a model is deployed which can be more cost effective and prevent unwanted performative phenomena from arising once a model is deployed (Appendix includes a demonstration of why training a classifier in an Outcome Performative setting is different than in the classical setting). The complexity analysis of Eq.s 9, 12, and 14 provide guidance on the number of instances needed per intervention group. Lastly, the derivation of sample complexity bounds provides practical insight into the general applicability of intervention testing in various domains. In particular, the introduction of regions of indistinguishability which describe Outcome Performative settings for which the allotted number of interventions cannot reliably detect Outcome Performativity. Unsurprisingly, more interventions are needed to detect subtler Outcome Performative effects. This suggests that there may be settings for which detecting Outcome Performativity is practically infeasible. The consequences of such a finding are domain-specific. If the Outcome Performative effects of some setting are subtle (requiring a large number of intervention), but interventions are costly or unethical, it may be inappropriate to deploy ML models in that domain unless other, non-interventional methods, can be used to detect Outcome Performativity. The caveat being that non-interventional methods tend to be less sample efficient than intervention tests [15] which poses additional challenges around data availability. This work has several limitations. First we did not consider the verification latency [23] that might occur before an outcome is realized. In such settings, one would either have to wait until all the instances given to OPAB are realized (which could take a considerable amount of time) or the hypothesis testing would need to be done with the subset of instances whose outcomes have been realized. Our work also assumes that the labels recorded when observing an outcome are reliable (i.e. no noisy labels [24]). If both of these assumptions do not hold, it may increase the chance that OPAB produces a type I or type II error. Lastly, this work assumes that detecting Outcome Performativity is reliably achievable by observing the difference in outcome distributions of each group of interventions (i.e., OPAB tests if P (Y | do(Yb = 0)) ̸= P (Y | do(Yb = 1))). If this assumption does not hold for a given setting where Outcome Performativity is only detectable through P (Y | X, do(Yb = 0)) ̸= P (Y | X, do(Yb = 1)), OPAB is able to identify if a setting is Outcome Performative, but it cannot identify is a setting is not Outcome Performative. OPAB is sensitive to non-uniform Outcome Performative behaviour. However, this work has shown that OPAB and more specifically analysis of the outcome distributions under prediction interventions can reliably detect Outcome Performativity in a myriad of settings. Future work will address this limitation by relaxing the uniformity assumption.

Conclusions and Future Work

In this work we formalise the process of detecting Outcome Performativity using prediction interventions. This method: Outcome Performativity A/B Detection (OPAB) is empirically evaluated across several models of Outcome Performativity for which we also derive sample complexity bounds. This work also introduces the concept of regions of indistinguishability which describe Outcome Performative settings where given an allotted number of interventions, Outcome Performativity cannot be reliably detected. Broadly speaking, maximising the number of interventions minimises regions of indistinguishability. These results have implications for settings where interventions are cost prohibitive or potential unethical. Future work will look to address the limitations of this paper. Namely, settings with 10

Detecting Outcome Performativity Using Intervention Testing

verification latency (when outcomes take some amount of time to be observed) and imperfect / noisy labelling will be considered.

OP-AB Motivation

Recall that OP-AB uses intervention testing to detect Outcome Performativity. The motivations for this decision are described using Figure 6. Naively, it might be tempting to simply compare whether the outcome distributions conditioned on a classifier’s predictions ŷ are dissimilar: H(P (Y | ŷ = 0), P (Y | ŷ = 1)) ≤ δ

(15) 1

Here, H is a function that calculates the dissimilarity of the two distributions (e.g. a hypothesis test) and δ is the threshold that needs to be reached in order to determine if that dissimilarity is significant. However, this is not sufficient, as these results are only meaningful if the instances that make up the two groups come from the same distribution (i.e. P (X | ŷ = 0) ≈ P (X | ŷ = 1)). In contradiction, the very aim of fitting a classifier is to maximize the dissimilarity between P (X | ŷ = 0) and P (X | ŷ = 1). This is shown in the second row of Figure 6. Using these predictions will often falsely detect Outcome Performativity. To combat this phenomena, OP-AB takes inspiration from Randomized Controlled Trials by using random sampling to assign predictions, ensuring P (X | ŷ = 0) ≈ P (X | ŷ = 1), removing the effects of confounding variables, and preventing false detections in settings with no Outcome Performativity. This is shown in row 3 of Figure 6.

Proofs

In this section we provide proofs for Equations 9, 12, and 14. All proofs start with same assumption about estimating the number of samples needed per intervention group N : Assumption: The task is a binary classification task such that an outcome y ∈ Y = {0, 1}. Assumption: Predictions ŷ ∈ Yb = {0, 1} are assigned to instances x ∈ X at random. Assumption: A Chi-Squared Test will be used to determine if two outcome distributions are significantly dissimilar. If these assumptions hold, the χ2 value of the calculated using the following Equation: χ2 =

X (Oi − Ei )2 Ei

(16)

where Ei is the expected outcome and Oi is the observed outcome of a prediction-outcome pair i = (y, ŷ). These values are calculated by creating a contingency / frequency table from the prediction-outcome pairs that are observed. B.1

Simple Model of Outcome Performativity

In the Simple Model of Outcome Performativity (Sect. 4.1), outcomes are determined by: p∗ =

α0 α1

if ŷ = 0 if ŷ = 1

(17)

where αc = P (y = 1 | ŷ = c) is the probability that the outcome is y = 1 for when the prediction is ŷ = c ∈ {0, 1} (binary classification task). Using Equation 17, we can populate a contingency table that would be generated (on average) for a given number of samples per intervention group N and α0 , α1 ∈ [0.0, 1.0] as shown in Table 3 We can then calculate the Expected Contingency Table that would occur if the Null Hypothesis of the Chi-Squared Test was true as shown in Table 4. We then plug the values from Tables 3 and 4 into Equation 16 which will give us: χ2 = 2N 1

(α0 − α1 )2 2(α0 + α1 ) − (α0 + α1 )2

If a distance metric is used, the equation will then instead be: H(P (Y | ŷ = 0), P (Y | ŷ = 1)) ≥ δ.

(18)

Anonymous

Visualization of the True Distribution P(X) 1.0

P(X) - Decision Boundary P(X | Y = 0) P(X | Y = 1)

Density

0.8 0.6 0.4 0.2 0.0 2.0

1.5

1.0

0.5

0.0 X

0.5

1.0

1.5

2.0

1.5

2.0

Visualization of the Distributions produced by 1.0

P(X) - Decision Boundary P(X |Y = 0) (Group A) P(X |Y = 1) (Group B)

Density

0.8 0.6 0.4 0.2 0.0 2.0

1.5

1.0

0.5

0.0 X

0.5

1.0

Visualization of the Distributions produced by Intervention Testing 0.40

P(X) P(X |Y = 0) (Group A) P(X |Y = 1) (Group B)

0.35 0.30 Density

0.25 0.20 0.15 0.10 0.05 0.00 2.0

1.5

1.0

0.5

0.0 X

0.5

1.0

1.5

2.0

Figure 6: A visualization of the motivation behind OP-AB. The first row shows the true distribution P (X) of an example binary classification dataset X. The dataset is then further split per class (P (X | Y ∈ {0, 1})). θ is the model trained on that dataset. In the second row, we show the distributions that would be produced (P (X | Ŷ ∈ {0, 1})) if θ were used to determine the intervention where Ŷ ∈ {0, 1}. As shown, the two distributions (Group A and B) are very different, and thus cannot be used to determine if the label / outcome distributions (P (Y )) are significantly different. In the third row, we show the distributions that would be produced (P (X | Ŷ ∈ {0, 1})) if intervention testing (randomized predictions) were used to determine the intervention. As shown, the two distributions are nearly identical and thus can be used to determine if the outcome / label distributions (P (Y )) are significantly different (and therefore Outcome Performative). 12

Detecting Outcome Performativity Using Intervention Testing

ŷ = 0 ŷ = 1 Totals y = 0 (1 − α0 )N (1 − α1 )N N (2 − α0 − α1 ) y=1 α0 N α1 N N (α0 + α1 ) Totals N N 2N Table 3: Average observed Contingency Table in the Simple Model of Outcome Performativity given N , α0 and α1 . ŷ = 0 ŷ = 1 y = 0 N2 (2 − α0 − α1 ) N2 (2 − α0 − α1 ) N N y=1 2 (α0 + α1 ) 2 (α0 + α1 ) Table 4: Expected Contingency Table in the Simple Model of Outcome Performativity if the Null Hypothesis for the Chi-Squared Test is true for a given N , α0 and α1 .

which we can rearrange to estimate N : N = χ2

2(α0 + α1 ) − (α0 + α1 )2 2(α0 − α1 )2

(19)

Here χ2 is the Chi-Squared statistic that you’d want to achieve. In the work, we set χ2 = 3.841 which is the threshold required to get a significant result for p = 0.05 with one degree of freedom. Figure 7 shows visualization of Equation 19. B.2

Model-based Outcome Performativity

The model-based model [9] takes into account an instance’s position in the feature space. Given a set of labelled instances (X, Ȳ ), a linear model fθ → {0, 1} is trained and its weights θ observed. Using θ, we simulate Outcome Performativity using: y = fθ (x + βŷ .θ)

(20)

where βŷ is the strength of the Outcome Performativity for prediction ŷ = c. If βc < 0.0, it biases instances towards an outcome of y = 0. If βc > 0.0, it biases instances towards an outcome of y = 1. If βc = 0.0, predictions are non-Outcome Performative. Using Equation 20, we can populate a contingency table that would be generated (on average) for a given number of samples per intervention group N and β0 , β1 ∈ [−1.01.0] as shown in Table 5 ŷ = 0 ŷ = 1 Totals y = 0 (1 − pβ0 )N (1 − pβ1 )N N (2 − pβ0 − pβ1 ) y=1 pβ0 N pβ1 N N (pβ0 + pβ1 ) Totals N N 2N Table 5: Average observed Contingency Table in the Model-based assumption class of Outcome Performativity given N , β0 and β1 . In Table 5, the number of instances that would have an outcome of y = 1 given a prediction of class ŷ = c when using model fθ is given by pβc = P (Y = 1 | Yb = c , θ). This contingency table has the same form as Table 3. The derivation for which we know is: N = χ2

2(pβ0 + pβ1 ) − (pβ0 + pβ1 )2 2(pβ0 − pβ1 )2

(21)

The visualization (which looks the same as Figure 7) is shown in Figure 8 where the two main findings, as is the case in the Simple Model of Outcome Performativity presented are that Outcome Performativity is undetectable when pβ0 ≈ pβ1 and only a few samples are needed to detect Outcome Performativity when |pβ0 − pβ1 | → 1.0. Of course, this derivation is not necessarily useful unless you know pβ0 and pβ1 . This can be done for any arbitrary set of features X and training labels Y ′ where fθ trains on X, and Y ′ , and then, for the desired β0 and β1 pair, you calculate pβ0 and pβ1 . 13

Anonymous

1.0

104

0.9

Samples Needed per Group(N)

0.8 0.7

103

0.6 0.5

102

0.4 0.3 0.2

101

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

0.1

Figure 7: The estimated number of instances needed per group (N ) to detect Outcome Performativity (Equation 9) across varying α0 and α1 values. The two main findings presented in this figure are that Outcome Performativity is undetectable when α0 ≈ α1 and only a few samples are needed to detect Outcome Performativity when |α0 −α1 | → 1.0.

B.3

Misclassification-based Outcome Performativity

Recall the definition of misclassification-based Outcome Performativity: Given a set of T instances and desirable outcomes (X, Y ), the outcome y ∈ Y = {0, 1} of an instance x ∈ X, given desirable outcome ȳ ∈ Y = {0, 1} and prediction ŷ ∈ Yb = {0, 1} is defined as:  1.0    0.0 p∗ =  γ   1 1 − γ0

if ŷ = 1 and ȳ = 1 if ŷ = 0 and ȳ = 0 if ŷ ̸= 1 and ȳ = 1 if ŷ ̸= 0 and ȳ = 0

(22)

where λc is the likelihood that a desired outcome is realized despite being misclassified: λc = P (y = c | ȳ = c and ŷ ̸= c)

(23)

Applying the same process as before, we can populate a contingency table that would be generated (on average) for a given number of samples per intervention group N and γ0 , γ1 ∈ [0.0, 1.0] as shown in Table 6 ŷ = 0 ŷ = 1 Totals y = 0 N (γ1 p0 − γ1 + 1) N γ0 p 0 N (γ0 p0 + γ1 p0 − γ1 + 1) y=1 N γ1 (1 − p0 ) N (−γ0 p0 + 1) N (−γ0 p0 − γ1 (p0 − 1) + 1) Totals N N 2N Table 6: Average observed Contingency Table in misclassification-based Outcome Performativity given N , p0, γ0 and γ1 .

Detecting Outcome Performativity Using Intervention Testing

Samples Needed per Group (N)

1.0

104

0.8

103

0.6

102

0.4

0.8

0.6

0.4

101

0.2

0.0

0.2

Figure 8: The estimated number of instances needed per group (N ) to detect Outcome Performativity (Equation 21) across varying pβ0 and pβ1 values. This figure is shows that model-based Outcome Performativity behaves similarly to Simple Model Outcome Performativity.

Note that in this model we have to also include p0 which describes the likelihood that a randomly drawn sample x ∈ X will have a desirable outcome ȳ = 0. We can then calculate the Expected Contingency Table that would occur if the Null Hypothesis of the Chi-Squared Test was true as shown in Table 7. ŷ = 0 ŷ = 1 N N y=0 (γ p + γ p − γ + 1) (γ p + γ1 p0 − γ1 + 1) 0 0 1 0 1 0 0 2 2 y = 1 N2 (−γ0 p0 − γ1 p0 + γ1 + 1) N2 (−γ0 p0 − γ1 p0 + γ1 + 1) Table 7: Expected Contingency Table in a misclassification-based Outcome Performative settings if the Null Hypothesis for the Chi-Squared Test is true for a given N , p0 , γ0 and γ1 .

We then plug the values from Tables 6 and 7 into Equation 16 which will give us:

χ2 = −

2N × (γ0 p0 − γ1 p0 + γ1 − 1)2 (γ0 p0 + γ1 p0 − γ1 − 1)(γ0 p0 + γ1 p0 − γ1 + 1)

(24)

which we can rearrange to estimate N :

N =−

χ2 × (γ0 p0 + γ1 p0 − γ1 − 1)(γ0 p0 + γ1 p0 − γ1 + 1) 2 × (γ0 p0 − γ1 p0 + γ1 − 1)2

(25)

Note that getting to Equation 14 required quite a few steps which we had to validate using Sympy. Figure 9 shows visualization of Equation 14. 15

Anonymous

1.0 0.9 104

Samples Needed per Group(N)

0.8 0.7 1

0.6

103

0.5 0.4

102

0.3 0.2

101

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

0.1

101

105

103 102 101

104

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

(p0 = 0.99) 106

Samples Needed Per Group (N)

102

(p0 = 0.75) Samples Needed Per Group (N)

101

103

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

105 104 103 102 101

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

102

104

103

(p0 = 0.5)

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

101

104

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

Samples Needed Per Group (N)

102

105

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

103

104

(p0 = 0.25)

105

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

Samples Needed Per Group (N)

106

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

(p0 = 0.01)

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Figure 9: The estimated number of instances needed per group (N ) to detect Outcome Performativity (Equation 14, p0 = 0.5) across varying γ0 and γ1 values. This figure shows that it is increasingly difficult to detect Misclassificationbased Outcome Performativity as γ0 and γ1 → 1.0.

Figure 10: Visualizations of Equation 14 across varying values of p0 . The values on these heatmaps report the estimated number of samples needed per intervention group (N ) to detect Outcome Performativity when χ2 = 3.841. B.3.1

Visualizations of Varying Initial Label Distributions

Given that Equation 14 requires that p0 be stipulated to know the estimated number of samples per intervention needed N to detect Outcome Performativity, we include plots for various values p0 as shown in Figure 10. Here we have assumed a χ2 = 3.841. The plots clearly show that as p0 → 0.0, the number of instances N required per intervention increases exponentially predominantly dependent on the misclassification response of class 0 (γ0 ). Conversely, as p0 → 1.0, the number of instances N required per intervention increases exponentially predominantly dependent on the misclassification response of class 1 (γ1 ). These results make sense given that the detecting Outcome Performativity in these settings becomes increasingly dependent on the prediction response of the minority class. This is also supported by the two orders of magnitude more instances that are required to detect subtle (γ0 , γ1 close to 1.0) Outcome Performativity.

Experimental Design

In this work, we made of the following datasets for our misclassification and model-based assumption classes of Outcome Performativity. Note that in order to get the datasets to work with the model-based Outcome Performativity Model, we could only make use of numerical features which are listed in the description below: 16

Detecting Outcome Performativity Using Intervention Testing

1. breast cancer [16]: The Breast Cancer Wisconsin (Diagnostic) Data Set. It consists of 569 instances of which 357 are benign and 212 malignant. We use all 30 features. 2. diabetes [17]: The Pima Indians Diabetes Database. It consists of 768 instances of which 268 belong to Class 1 (The others Class 0). We use all 8 features. 3. adult census [18]: The 1994 Adult Census Income dataset. It has ≈ 32 000 instances of which 76% are of Class ≤ 50k which the others are of class > 50k. We make use of the age, fnlwgt, education.num, capital.gain, capital.loss, and hours.per.week features. 4. kickstarter [19]: The Funding Successful Kickstarter Projects dataset. It contains 108 129 instances of which 73 568 are Class 0. The others are Class 1. We make use of the goal and backers\_count features. 5. titanic [20]: The Titanic Survival Prediction dataset. It consists of 891 instances of which 549 are Class 0 while the others are Class 1. We make use of the Age, SibSp, Parch, and Fare features. 6. loan [21]: A Loan Approval Dataset. It consists of 4269 instances of which 62% are labelled Approved. The others are labelled Rejected. We use all 9 numerical features. For all of the Experiments conducted in sections 4.3 and 4.2, each parameter set was evaluated over 100 replicates. The hypothesis test H given to OP-AB was the Chi-Squared test. In the case where the outcome contingency table contained a 0, Fisher’s Exact test was used instead. A successful detection meant that the p value returned by the hypothesis test was < δ = 0.05. A pseudorandom number generator was used to ensure reproducibility. All experiments were run in Google Colab using the Python 3 Google Compute Engine Backend. C.1

Open Bandits Dataset

For our real-world case study, we made use of the Open Bandits Dataset (OBD [22]). OBD was constructed using multiarmed bandit policies on the fashion e-commerce platform ZOZOTOWN for the off-policy evaluation of recommender systems. Each instance represents a user impression containing feature values, item recommendations and their placement (clothing items to choose from and where they appear on the store webpage), and click indicators (was the recommended clothing item clicked on) as an outcome. For a subset of the data, the placement of each recommended item on the store page (left, centre, or right) was determined using random sampling. We use this subset and repurpose the placement of item recommendation as predictions and click indications as outcomes. For preprocessing, we take all of the randomly sampled impressions and extract the position and click values. The former is our set of predictions Ŷ and the latter the set of outcomes Y . To ensure that both variables are binary, we filter out any recommendations with position == 2 which corresponds to centre on the webpage. The rest of the preprocessing proceeds as described in the Sect. 4, we first test to see if Outcome Performativity is present in each sub-dataset (men, women). We then perform random subsampling procedures over 1000 replicates to evaluate OP-AB.

Additional Results

In this section we report on the results for the Titanic and Loan datasets for both the misclassification and featurebased Models of Outcome Performativity. Additionally, we report on the additional experiments that we conducted. Namely: The efficacy of OP-AB when using different statistical tests, relationship between sample complexity N and δ, and demonstrating the importance N plays in determining the detection capabilities of OP-AB. The section concludes with a comparison of OP-AB and the Online Adam detection method described in [3]. D.1

Misclassification-based Outcome Performativity

Figure 11 reports on the results of applying OP-AB to detect misclassification-based Outcome Performativity on the Titanic and Loan datasets. These results are in keeping with our main findings. OP-AB exhibits a low false detection rate (i.e. γ0 = γ1 = 1.0). If N is larger, OP-AB can detect subtler Outcome Performative Effects. D.2

Model-based Outcome Performativity

Figure 12 reports on the results of applying OP-AB to detect model-based Outcome Performativity on the Titanic and Loan datasets. These results are in keeping with our main findings. OP-AB exhibits a low false detection rate when β0 = β1 ) because the outcome distributions per intervention are identical and thus non-Outcome Performative. If N is larger, OP-AB can detect subtler Outcome Performative Effects. Figure 12 also shows the limitation of the model-based assumption class of Outcome Performativity as features with extremely skewed Gaussian or Bimodal distributions (as 17

Anonymous

Loan (N = 213)

1.0

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

Detection Rate

0.8 0.6 0.4 0.2

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Titanic (N = 35)

Figure 11: Results of applying OP-AB to detect misclassification-based Outcome Performativity. These results are in keeping with our main findings. OP-AB exhibits a low false detection rate (i.e. γ0 = γ1 = 1.0) and if N is larger, OP-AB can detect subtler Outcome Performative Effects.

1.0

Detection Rate

0.8 0.6 0.4 0.2

1.0

0.8

0.6

0.4

0.2

0.0

-0.

0.0

1.0

0.8

0.6

0.4

0.2

2 0.0

-0.

0 -1.

Loan (N = 213)

1.0 0.8 0.6 0.4 0.2 0.0 -0.2 -0.4 -0.6 -0.8 -1.0

-1.

Titanic (N = 35)

1.0 0.8 0.6 0.4 0.2 0.0 -0.2 -0.4 -0.6 -0.8 -1.0

-0.

in the case of the Loan dataset) can cause significant portions of the features to change outcomes at small βn values. This is not a limitation of OP-AB, but rather of the assumption class of Outcome Performativity.

Figure 12: Results of applying OP-AB to detect model-based Outcome Performativity. These results are in keeping with our main findings. OP-AB exhibits a low false detection rate when β0 = β1 ) and if N is larger, OP-AB can detect subtler Outcome Performative Effects.

D.3

Using OP-AB with Other Statistical Tests

In this work we primarily used the Chi-Squared Test to determine if the outcome distributions per intervention were significant. OP-AB is hypothesis test agnostic so we wanted to test the efficacy of OP-AB over different Statistical Tests. Figure 13 reports the detection rate of OP-AB applied to the Simple Model of Outcome Performativity using the Chi-Squared, Fisher’s Exact of Mann-Whitney U tests. Each cell is averaged over 100 replicates. We conducted these experiments with N = 50. The results reported in Figure 13 clearly show that across the statistical tests evaluated, the detection rate of OP-AB remains approximately the same. The Mann-Whitney U test seems slightly more unreliable 18

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

Fisher's Exact Test

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

Mann-Whitney U test

1.0 0.8 Detection Rate

Chi-Squared Test

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

Detecting Outcome Performativity Using Intervention Testing

0.6 0.4 0.2

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

0.0 0

Figure 13: Results of using various Hypothesis Tests with OP-AB to detect Outcome Performativity. These results show that OP-AB is agnostic to the H used to detect Outcome Performativity (as long as H is appropriately applied). when the Outcome Performativity response is low (i.e. small |α0 − α1 |. These results reveal two additional insights: First, N is without a doubt the most important parameter in OP-AB. If too few instances are used in the intervention testing, OP-AB will not detect Outcome Performativity. Second, the trends estimated by Eq. 9, 14 and 12 in the main paper capture the statistical test-agnostic dynamics of the Outcome Performativity Models in this work. D.4

Varying Detection Threshold δ

In OP-AB, the detection threshold δ corresponds to the necessary p-value required by Hypothesis test H to raise a detection event. Naturally, one would assume that as δ increases, the OP-AB becomes less restrictive and will more likely flag Outcome Performativity. This should also be doable with fewer random samples. Conversely, as δ decreases, OP-AB will become more restrictive and less likely to flag Outcome Performativity. A detection event, on overage, will require more samples as δ decreases. The trade-off when choosing δ is between sample complexity (number of samples N needed per group) and the reliability of a detection event (a higher δ will produce more false positives). Figure 14 showcases the sample complexity dynamics across various commonly used statistical thresholds. These results confirm our suspicions, as δ increases, sample complexity N decreases. D.5

Increased N on Breast Cancer Dataset

For the final set of additional experiments, we wanted to empirically demonstrate the importance that N plays in being able to detect Outcome Performativity. Figure 15 shows these results where we apply OP-AB to the Breast Cancer dataset imputed with Feature-based Outcome Performativity over an increasing N . The results clearly show that as N increases, OP-AB is able to detect Outcome Performativity across a wider range of β0 and β1 values. Again, we want to note that our results suggest that as the difference in Outcome Performativity responses tend to non-Outcome Performativity (|β0 − β1 | → 0.0), the number of instances required per intervention N → ∞. D.6

Comparison with Adam Detection

To the best of our knowledge, the only other explicit Outcome Performativity detection method is described in [3]. In short, this method (which we will call Adam Detection) monitors the change in True Positive Rate (TPR) of a deployed model compared to its TPR that it achieved on its historical (training) data. If the change in TPR is determined to be statistically significant, Outcome Performativity is detected. Figures 16 summarizes the results of Adam Detection on several datasets imputed with the Misclassification Outcome Performativity described earlier. In all figures we can see that the detection rates of Adam Detection are lower and more inconsistent when compared to the results obtained by OP-AB. This occurs because Adam Detection requires a deployed model. Under Misclassification Outcome Performativity, the more accurate the deployed model classifier, the harder it is to detect said Outcome Performativity as only misclassified instances are susceptible to having their observed outcomes altered. This is most evident in Figure 16a where the deployed model achieves high accuracy, rarely misclassifying instances. Adam Detection is also less sample efficient because it requires historical labelled data to initially train a model, and labelled data after the model has been deployed. This is shown in Figures 16b and 16c where Adam Detection exhibits the same detection rate 19

N = 100

N = 1000

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

N = 10000

1.0 0.8 Detection Rate

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

N = 10

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

Anonymous

0.6 0.4 0.2

N = 10

N = 100

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

0.0 0

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

N = 10000

1.0 0.8 Detection Rate

N = 1000

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

(a) δ = 0.01

0.6 0.4 0.2

N = 10

N = 100

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

0.0 0

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

N = 10000

1.0 0.8 Detection Rate

N = 1000

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

(b) δ = 0.05

0.6 0.4 0.2

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

0.0 0

Figure 14: Results from exploring various detection thresholds δ on the Simple Outcome Performativity Model. These results show that as δ decreases, sample complexity N increases. The trade-off being that detection events at a smaller δ is more reliably than a detection event at a larger δ.

1.0 0.8 Detection Rate

Breast Cancer (N = 213)

1.0 0.8 0.6 0.4 0.2 0.0 -0.2 -0.4 -0.6 -0.8 -1.0

0.6 0.4 0.2

-1. 0 -0. 8 -0. 6 -0. 4 -0. 2 0.0 0.2 0.4 0.6 0.8 1.0

Breast Cancer (N = 142)

1.0 0.8 0.6 0.4 0.2 0.0 -0.2 -0.4 -0.6 -0.8 -1.0

-1. 0 -0. 8 -0. 6 -0. 4 -0. 2 0.0 0.2 0.4 0.6 0.8 1.0

Breast Cancer (N = 71)

1.0 0.8 0.6 0.4 0.2 0.0 -0.2 -0.4 -0.6 -0.8 -1.0

-1. 0 -0. 8 -0. 6 -0. 4 -0. 2 0.0 0.2 0.4 0.6 0.8 1.0

Breast Cancer (N = 28)

1.0 0.8 0.6 0.4 0.2 0.0 -0.2 -0.4 -0.6 -0.8 -1.0

-1. 0 -0. 8 -0. 6 -0. 4 -0. 2 0.0 0.2 0.4 0.6 0.8 1.0

Detecting Outcome Performativity Using Intervention Testing

Figure 15: Results demonstrating the importance the number of instances per intervention (N ) plays on the efficacy of OP-AB. A higher N allows OP-AB to detect Outcome Performativity across a wider range of β0 and β1 values .

(a)

(b)

(c)

1.0 0.8 0.6 0.4 0.2 0.0

0.9 1.0

0.8

0.7

0.6

0.5

0.4

0.0

0.3

0.2

0.4

Detection Rate

0.6

0.1

0.8

1.0 1.0 1.0 1.0 0.96 0.96 0.92 0.68 0.52 0.4 0.04 0.0 0.9 1.0 1.0 0.96 0.92 0.92 0.84 0.72 0.32 0.16 0.0 0.08 0.8 1.0 0.92 0.96 0.92 0.84 0.84 0.44 0.16 0.04 0.08 0.12 0.7 0.92 1.0 0.88 0.92 0.64 0.68 0.36 0.16 0.08 0.08 0.24 0.6 0.96 0.88 0.76 0.8 0.56 0.52 0.4 0.08 0.08 0.08 0.32 0.5 0.88 0.76 0.8 0.72 0.76 0.52 0.12 0.08 0.16 0.2 0.28 0.4 0.92 0.84 0.72 0.56 0.44 0.28 0.08 0.12 0.12 0.24 0.6 0.3 0.92 0.76 0.64 0.64 0.48 0.08 0.16 0.24 0.32 0.36 0.48 0.2 0.72 0.68 0.56 0.56 0.28 0.2 0.08 0.16 0.36 0.52 0.56 0.1 0.52 0.36 0.56 0.32 0.24 0.24 0.24 0.44 0.56 0.52 0.68 0.0 0.64 0.52 0.2 0.2 0.2 0.2 0.36 0.28 0.4 0.6 0.6

0.0

0.9 1.0

0.8

0.7

0.6

0.5

0.4

0.3

0.1

0.2

0.1

0.3

Detection Rate

0.4

0.0

0.5

Kickstarter Dataset

1.0 0.96 0.88 0.52 0.64 0.4 0.36 0.12 0.12 0.08 0.16 0.04 0.9 0.84 0.64 0.68 0.44 0.2 0.24 0.44 0.16 0.12 0.08 0.08 0.8 0.84 0.64 0.64 0.44 0.32 0.2 0.12 0.0 0.08 0.0 0.04 0.7 0.64 0.56 0.4 0.36 0.4 0.16 0.16 0.04 0.08 0.0 0.04 0.6 0.72 0.52 0.76 0.4 0.24 0.2 0.12 0.08 0.16 0.16 0.04 0.5 0.64 0.28 0.24 0.36 0.24 0.08 0.16 0.16 0.12 0.2 0.36 0.4 0.52 0.32 0.32 0.32 0.24 0.28 0.12 0.08 0.16 0.28 0.24 0.3 0.4 0.36 0.44 0.24 0.16 0.12 0.2 0.4 0.24 0.44 0.32 0.2 0.56 0.36 0.2 0.32 0.12 0.2 0.36 0.48 0.36 0.48 0.68 0.1 0.32 0.12 0.32 0.28 0.32 0.28 0.36 0.52 0.68 0.6 0.72 0.0 0.4 0.4 0.36 0.32 0.28 0.56 0.48 0.52 0.48 0.76 0.76

Detection Rate

Adult Census Dataset

0.9 1.0

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

Breast Cancer Dataset 1.0 0.12 0.24 0.24 0.32 0.4 0.16 0.28 0.24 0.16 0.2 0.12 0.9 0.16 0.32 0.16 0.2 0.24 0.28 0.28 0.24 0.36 0.28 0.24 0.8 0.2 0.32 0.2 0.16 0.16 0.24 0.12 0.4 0.16 0.16 0.24 0.7 0.4 0.32 0.2 0.16 0.16 0.12 0.24 0.28 0.08 0.24 0.12 0.6 0.16 0.24 0.36 0.12 0.24 0.12 0.32 0.16 0.2 0.28 0.28 0.5 0.2 0.28 0.36 0.28 0.32 0.4 0.2 0.2 0.08 0.24 0.2 0.4 0.36 0.28 0.32 0.28 0.24 0.28 0.2 0.28 0.12 0.24 0.16 0.3 0.32 0.08 0.28 0.2 0.28 0.12 0.16 0.2 0.24 0.32 0.28 0.2 0.32 0.12 0.16 0.12 0.16 0.24 0.24 0.12 0.28 0.2 0.28 0.1 0.2 0.24 0.36 0.24 0.28 0.2 0.2 0.28 0.44 0.32 0.36 0.0 0.44 0.28 0.24 0.4 0.28 0.56 0.12 0.32 0.36 0.4 0.12

Figure 16: Adam Detection performed on the Breast Cancer (a), Adult Census (b) and Kickstarter (c) datasets. Overall, the detection rate of Adam Detection is inconsistent. This is partly due to the accuracy of the deployed classifiers which when high, rarely produce misclassifications making detection difficult. The number of instances in each dataset also play a role with more instances increasing the likelihood of detection such as in the Kickstarter dataset. Our method, OP-AB, achieves higher and more consistent detection rates with only 10% of the samples used by Adam Detection.

trends as OP-AB (although less consistent), but it required the whole of the Adult Census and Kickstarter datasets whereas OP-AB achieved better and more consistent detection rates with only 10% of the datasets’ instances. Adam Detection is an online algorithm, which is susceptible to both Intrinsic [25] and non-Outcome Performative [6] Concept Drift phenomena. These results put the primary advantages of OP-AB into perspective. Because OP-AB is an offline algorithm, it is (1) not susceptible to Concept Drift, and (2) it can be used already during the labelling process of a ML model’s development lifecycle. This has the added benefit of requiring fewer instances (i.e. better sample efficiency) and it eliminates any potential negative impacts that might occur from a deployed model that is used before the Outcome Performativeness of a problem domain is detected.

D.6.1

Implementing Adam Detection

Unfortunately, we were not able to obtain the source code for Adam detection so we had to remake it using the text-only description provided in the referenced paper [3]. This was slightly problematic as the paper did not describe the Online Updating procedure for the deployed model which made it challenging to know if we replicated Adam detection exactly. In short, our implementation of Adam will take in an entire static dataset, shuffle it and partition it into 11 approximately equal sized chunks. The 11th chunk is used as the test set. The first 5 of the remaining sets are used as historical data to determine the deployed model’s True Positive Rate (TPR) before deployment and the last final 5 sets are used to iteratively update the deployed model in an Online fashion. These Online sets are susceptible to the Adam Outcome Performativity described in Sect. 4.3. At the end of each update, the model’s new TPR is recorded. Once training is complete, a Mann-Whitney U test is used to determine if the difference in TPR achieved on the historical datasets are statistically different from the TPR achieved on the iterative (deployed) sets. For these experiments, reported results are the average over 25 replicates and a δ = 0.05 was used to determine statistical significance. 21

Anonymous

Difference in Average Accuracy obtained by Linear Model with and without Prediction Conditioning over different P(Y = 0 |Y)

P(Y = 0 |Y = 0)

(a)

(b)

(c)

0.16 0.14 Average Accuracy

0.12 0.10 0.08 0.06 0.04 0.02 0.00

0.9 1.0

0.8

0.7

0.6

0.5

0.0

0.4

0.2

0.3

0.4

1.0 0.0 0.01 0.01 0.02 0.04 0.07 0.07 0.06 0.04 0.01 0.0 0.9 0.0 -0.0 -0.0 0.01 0.02 0.04 0.04 0.03 0.01 0.0 0.01 0.8 0.01 0.0 -0.0 -0.0 0.01 0.02 0.01 0.01 0.0 0.03 0.08 0.7 0.01 0.0 0.01 0.01 -0.0 0.0 0.0 0.0 0.02 0.08 0.13 0.6 0.09 0.04 0.01 0.01 -0.0 -0.01 0.0 0.02 0.06 0.11 0.16 0.5 0.14 0.09 0.06 0.03 0.01 0.0 0.0 0.04 0.06 0.1 0.14 0.4 0.17 0.12 0.05 0.02 0.0 -0.0 0.0 -0.0 0.01 0.03 0.08 0.3 0.14 0.09 0.03 0.0 0.0 -0.01 0.01 -0.0 0.01 0.01 0.03 0.2 0.09 0.03 0.0 0.01 0.02 0.02 0.01 0.0 0.0 0.0 0.0 0.1 0.02 0.0 0.01 0.03 0.04 0.04 0.03 0.01 -0.0 0.0 -0.0 0.0 0.0 0.02 0.04 0.07 0.08 0.06 0.05 0.03 0.01 0.0 0.0

0.2

0.6

P(Y = 1 |Y = 1)

Average Accuracy

0.8

0.1

1.0

0.0

0.9 1.0

0.8

0.7

0.6

0.5

0.0

0.4

0.2

0.3

0.4

0.2

0.6

P(Y = 1 |Y = 1)

Average Accuracy

0.8

0.1

1.0

0.9 1.0

0.8

0.7

0.6

0.5

0.4

0.3

0.0

0.2

1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.99 0.99 0.98 0.98 1.0 0.9 0.9 0.9 0.89 0.89 0.89 0.89 0.88 0.88 0.87 0.91 0.95 0.8 0.81 0.79 0.79 0.79 0.79 0.78 0.77 0.78 0.81 0.87 0.96 0.7 0.7 0.7 0.71 0.69 0.69 0.67 0.68 0.71 0.77 0.86 0.98 0.6 0.6 0.6 0.59 0.57 0.55 0.57 0.6 0.67 0.76 0.88 0.99 0.5 0.5 0.48 0.46 0.48 0.47 0.5 0.57 0.67 0.77 0.9 1.0 0.4 0.39 0.37 0.34 0.35 0.4 0.48 0.56 0.68 0.78 0.9 1.0 0.3 0.29 0.28 0.27 0.29 0.37 0.46 0.57 0.67 0.8 0.9 1.0 0.2 0.18 0.17 0.19 0.28 0.37 0.47 0.57 0.7 0.8 0.88 1.0 0.1 0.06 0.11 0.18 0.27 0.37 0.47 0.58 0.69 0.78 0.91 1.0 0.0 0.0 0.08 0.19 0.29 0.39 0.5 0.59 0.7 0.8 0.9 1.0

0.1

Heatmap of Average Accuracy obtained by Linear Model exploring different P(Y|Y) using Prediction Conditioning

1.0 1.0 0.99 0.99 0.96 0.95 0.94 0.92 0.92 0.94 0.96 1.0 0.9 0.9 0.9 0.89 0.88 0.86 0.84 0.83 0.84 0.87 0.91 0.94 0.8 0.8 0.79 0.79 0.79 0.77 0.75 0.75 0.77 0.81 0.84 0.88 0.7 0.68 0.69 0.7 0.69 0.69 0.67 0.68 0.71 0.75 0.78 0.85 0.6 0.54 0.55 0.57 0.57 0.55 0.57 0.6 0.65 0.7 0.77 0.84 0.5 0.37 0.39 0.4 0.46 0.47 0.5 0.55 0.64 0.72 0.8 0.85 0.4 0.24 0.25 0.29 0.33 0.4 0.49 0.56 0.68 0.77 0.87 0.92 0.3 0.14 0.2 0.25 0.29 0.36 0.47 0.56 0.68 0.79 0.89 0.98 0.2 0.09 0.14 0.19 0.28 0.35 0.45 0.56 0.7 0.8 0.88 0.99 0.1 0.04 0.11 0.18 0.24 0.33 0.43 0.56 0.68 0.79 0.91 1.0 0.0 0.0 0.06 0.14 0.22 0.32 0.44 0.54 0.67 0.78 0.89 1.0

P(Y = 1 |Y = 1)

Heatmap of Average Accuracy obtained by Linear Model exploring different P(Y|Y) using Standard Supervised train-test loop

Figure 17: Figure (a) shows the average accuracy achieved by a Linear Model (trained using standard Supervised Learning) over various parameters in the Simple Model of Outcome Performativity. Figure (b) plots the same using prediction conditioning whereby the Linear Model can predict the outcomes (Y ) that will be observed given the prediction it makes (Ŷ ). Figure (c) shows the difference in accuracy achieved by each model ((b) − (a)) with prediction conditioning performing better, particularly around areas of label imbalance in the original dataset.

Training Machine Learning Models in Outcome Performative Settings

Although it is not the focus of this paper, we do want demonstrate why training models in an Outcome Performative setting is different than in the traditional Supervised setting. For this, we will use the breast cancer [16] dataset and the Simple Model of Outcome Performativity described in the paper. All results shown are average of 25 runs. The first challenge in the Outcome Performative setting is gathering data. This has to come from historical data where a prediction Ŷ has already been made and an outcome Y observed. To simulate this in these demonstrations, we train a base classifier on the breast canncer dataset, and use it’s predictions to simulate outcomes for our real model θ to learn from. Assuming that we just use a standard Supervised training-test process, we would get results shown in Figure 17a. Overall, the results are about what we expected, the accuracy of the model increases when the probability of encountering a label P (Y | Ŷ ) → 1.0. When P (Y | Ŷ ) → 0.0 the accuracy decreases as the model will never make a correct prediction. These results are unsurprising, but the model θ is far from achieving the maximum achievable accuracy of max(P (Y | Ŷ = 0), P (Y | Ŷ = 1)). We can somewhat solve this problem by introducing prediction conditioning. That is when the model can be conditioned on the prediction it would make in order to predict the final outcome it will observe. In practice, this is just adding an additional feature to the model Ŷ = {0, 1}. This introduces a new problem where we must now choose between the probability scores produced for each class (e.g. θŷ=0 (x) vs. θŷ=1 (x) for a binary classification task). If we want to maximize accuracy as we did in the standard Supervised setting, we add a decision rule to determine our final prediction ŷ: ŷ =

0 if Pθ (Y = 0 | θŷ=0 (x) = 0) > Pθ (Y = 1 | θŷ=1 (x) = 1) 1 else

(26)

where Pθ is the estimated probability score of outcome Y when θ is conditioned on prediction ŷ. In short, this decision rule will predict ŷ = 0 if the estimated probability of observing outcome y = 0 (given ŷ = 0) is greater than observing outcome y = 1 (given ŷ = 1). The results of using a model with prediction conditioning is shown in Figure 17b. Overall, the results look similar to that of the model without prediction conditioning, but if we observe Figure 17c which plots the difference in accuracy of the model with prediction conditioning − the accuracy of the model without, we can see that that prediction conditioning can improve the accuracy of the model dramatically. For the breast cancer dataset, the greatest benefits are at parameter ranges around that correlate with the data imbalances in the original dataset. Naively, these results imply that prediction conditioning is good, and sufficient for solving Outcome Performative problems. This may not always be the case. Yes, prediction conditioning does allow one to gain insight into how their predictions may affect observed outcomes, but it relies heavily on the decision rule you use. Is accuracy all you really care about, then the aforementioned decision rule works well. Consider the palliative care problem, we don’t necessarily just want want to be accurate, we actually want to maximize an outcome (a patient’s quality or life). If our deployed model is maximizing accuracy, it may over or under-prescribe interventional care. Fortunately, changing the model’s behaviour is (in this abstract context) quite simple, we just need to change the decision-rule. We demonstrate that in Figure 18 where we show how the same prediction conditioned model can be used to maximize different outcomes (Y = 1 in this case) just by changing the decision rule to: 22

Detecting Outcome Performativity Using Intervention Testing

Likelihood of achieving an Outcome (Y = 1)

1.0 0.8 0.6 0.4 0.2 0.0

0.9 1.0

0.8

0.7

0.6

0.5

0.4

0.3

0.2

1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.9 0.88 0.9 0.89 0.9 0.89 0.9 0.9 0.91 0.9 1.0 0.89 0.79 0.78 0.8 0.8 0.8 0.8 0.81 0.81 0.8 1.0 0.9 0.79 0.7 0.7 0.7 0.71 0.71 0.71 0.68 0.7 1.0 0.91 0.79 0.68 0.59 0.6 0.6 0.59 0.6 0.61 0.61 1.0 0.9 0.81 0.68 0.59 0.48 0.49 0.48 0.49 0.5 0.49 1.0 0.91 0.82 0.71 0.59 0.46 0.39 0.38 0.4 0.4 0.4 1.0 0.89 0.8 0.71 0.61 0.49 0.4 0.31 0.27 0.3 0.3 1.0 0.9 0.81 0.69 0.59 0.5 0.41 0.29 0.21 0.19 0.21 1.0 0.89 0.79 0.7 0.62 0.52 0.41 0.29 0.2 0.09 0.11 1.0 0.9 0.8 0.7 0.6 0.49 0.41 0.3 0.2 0.1 0.0

0.1

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

0.0

P(Y = 1 |Y = 1)

Average Likelihood of achieving an Outcome (Y = 1) using a Linear Model exploring different P(Y|Y)

P(Y = 0 |Y = 0)

Figure 18: Figure showing how the same model used in Figure 17b can be used to produce varying results over the same parameters. In this case, a simple change in the decision rule (from maximizing accuracy to maximizing observing an outcome of y = 1) produced the following figure. This is meant to highlight the difficulty and potential danger of naively deploying predictive models in settings that are Outcome Performative.

ŷ =

0 1

if Pθ (Y = 1 | θŷ=0 (x) = 1) > Pθ (Y = 1 | θŷ=1 (x) = 1) else

(27)

The results clearly show that in the Outcome Performative setting, how you use your deployed model can have massive effect on the types of outputs you get. Before our model struggled to accurately predict outcomes in regions it is now achieving perfect scores in. This further acts as motivation for using OP-AB. Being aware of Outcome Performativity is the first step in understanding how to deploy a model in an Outcome Performative setting. These results also show how using prediction conditioning and different decision rules can produce vary different results, which further leads into ethical concerns that could arise given that these models will be deliberately steering outcomes towards results specified by their desired outcomes (i.e. decision rule).

Applying OP-AB to Monitoring Tasks

We introduced OP-AB as a "one-and-done" algorithm. That is, you give OP-AB T instances to perform intervention testing on and the output of the algorithm is a "yes/no" answer to whether the setting is Outcome Performative. Practically, you may want to repeatedly evaluate a setting (perhaps the Outcome Performativeness of setting itself is dynamic or subject to Concept Drift). This would then be a monitoring task we you would have to repeatedly have to re-evaluate the Outcome Performativeness of your problem domain. Algorithm 2 details how you might do this. Simply put, every Z instances, you would dedicate T instances to OP-AB and use the result to inform future decision making.

References [1] Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt. Performative prediction. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 7599–7609. PMLR, 2020. [2] Gary Cheng, Moritz Hardt, and Celestine Mendler-Dünner. Causal inference out of control: estimating performativity without treatment randomization. In Forty-first International Conference on Machine Learning, 2024. [3] George Alexandru Adam, Chun-Hao Kingsley Chang, Benjamin Haibe-Kains, and Anna Goldenberg. Hidden risks of machine learning applied to healthcare: unintended feedback loops between models and future data causing model degradation. In Machine Learning for Healthcare Conference, pages 710–731. PMLR, 2020. 23

Anonymous

Algorithm 2 Simple Pseudocode for using OP-AB in a monitoring task. The code is intentionally vague about what should be done when a positive test is found, and what is done with data batch X. This will be task specific. Require: Data stream X, Test Frequency Z, Test Size T 1: x = X.next_batch() 2: T = 0 3: while x ̸= ∅ do 4: condition = T mod Z = 0 and OP AB(x, T ) 5: if condition then 6: Report Results 7: end if 8: Perform operations on x 9: x = X.next() 10: T+ = 1 11: end while

[4] Wouter AC van Amsterdam, Nan van Geloven, Jesse H Krijthe, Rajesh Ranganath, and Giovanni Ciná. When accurate prediction models yield harmful self-fulfilling prophecies. Patterns, 6(4), 2025. [5] Michael P Kim and Juan C Perdomo. Making decisions under outcome performativity. In 14th Innovations in Theoretical Computer Science Conference (ITCS 2023), pages 79–1. Schloss Dagstuhl–Leibniz-Zentrum für Informatik, 2023. [6] Brandon Gower-Winter, Georg Krempl, Sergey Dragomiretskiy, Tineke Jelsma, and Arno Siebes. Identifying predictions that influence the future: Detecting performative concept drift in data streams. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025. [7] Nancy Cartwright. What are randomised controlled trials good for? Philosophical studies, 147(1):59–70, 2010. [8] Judea Pearl. Causality. Cambridge university press, 2009. [9] Celestine Mendler-Dünner, Frances Ding, and Yixin Wang. Anticipating performativity by predicting from predictions. Advances in neural information processing systems, 35:31171–31185, 2022. [10] George Alexandru Adam, Chun-Hao Kingsley Chang, Benjamin Haibe-Kains, and Anna Goldenberg. Error amplification when updating deployed machine learning models. In Machine Learning for Healthcare Conference, pages 715–740. PMLR, 2022. [11] Nicolò Pagan, Joachim Baumann, Ezzat Elokda, Giulia De Pasquale, Saverio Bolognani, and Anikó Hannák. A classification of feedback loops and their relation to biases in automated decision-making systems. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–14, 2023. [12] James Liley, Samuel Emerson, Bilal Mateen, Catalina Vallejos, Louis Aslett, and Sebastian Vollmer. Model updating after interventions paradoxically introduces bias. In International Conference on Artificial Intelligence and Statistics, pages 3916–3924. PMLR, 2021. [13] Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10):4156–4165, 2019. [14] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), 2018. [15] Yingrong Wang, Haoxuan Li, Minqin Zhu, Anpeng Wu, Baohong Li, Keting Yin, Ruoxuan Xiong, Fei Wu, and Kun Kuang. Causal inference with complex treatments: A survey. ACM Computing Surveys, 58(9):1–36, 2026. [16] Matjaz Zwitter and Milan Soklic. Breast Cancer, 1988. Accessed: April 2025. [17] Jack W Smith, James E Everhart, William C Dickson, William C Knowler, and Robert Scott Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, page 261, 1988. Accessed: April 2025. [18] Barry Becker and Ronny Kohavi. Adult census income, 1996. Accessed: April 2025. [19] Lathwal. Funding successful projects on kickstarter, 2017. Accessed: April 2025. [20] Will Cukierski. Titanic - machine learning from disaster, 2012. Accessed: April 2025. 24

Detecting Outcome Performativity Using Intervention Testing

[21] Kai. Loan approval prediction dataset, 2023. Accessed: August 2024. [22] Yuta Saito, Aihara Shunsuke, Matsutani Megumi, and Narita Yusuke. Large-scale open dataset, pipeline, and benchmark for bandit algorithms. arXiv preprint arXiv:2008.07146, 2020. [23] Georg Krempl, Indre Žliobaite, Dariusz Brzeziński, Eyke Hüllermeier, Mark Last, Vincent Lemaire, Tino Noack, Ammar Shaker, Sonja Sievi, Myra Spiliopoulou, et al. Open challenges for data stream mining research. ACM SIGKDD Explorations Newsletter, 16(1):1–10, 2014. [24] Zeyu Ren, Shuihua Wang, and Yudong Zhang. Weakly supervised machine learning. CAAI Transactions on Intelligence Technology, 8(3):549–580, 2023. [25] João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):1–37, 2014.

Record · ID 411058 · SHA-256 2d8da885bfd529e4
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.