Rule of Thumb: Explaining Artificial Intelligence Systems using Partial Information

Rule of Thumb: Explaining Artificial Intelligence Systems using Partial Information Kaivalya Rawal∗1 , Daria Onitiu2 , Brent Mittelstadt1,3 , Sandra Wachter1,2 , Chris Russell1 1

University of Oxford ∗

2

Hasso Plattner Institute

3

Weizenbaum Institute

address correspondence to: kaivalya.rawal@oii.ox.ac.uk

arXiv:2608.10766v1 [cs.AI] 11 Aug 2026

Abstract Explainable Artificial Intelligence (XAI) seeks to explain how an Artificial Intelligence (AI) system arrived at a particular decision. We propose “Rule of Thumb” (ROT) explanations, a new approach to XAI based upon a novel formulation that identifies the most relevant features for predicting the behaviour of an AI system, for a particular datapoint. We show how ROT is well-suited to enable XAI in: (a) zero-shot classification using large language models (LLMs), (b) auditing of opaque AI systems without model access, and (c) the use of AI in scientific discovery. Additionally, ROT meets specific requirements from leading AI regulations, provides a familiar interface and visualisations for XAI practitioners, is model-agnostic, and is substantially faster than alternatives. Code available at: https://github.com/KaiRawal/Rule-of-Thumb-Explaining -Artificial-Intelligence-Systems-using-Partial-Information

1

Introduction

Artificial intelligence (AI) in the form of large language models has become ubiquitous. One of the key components that has made it disruptive and widespread is its ability to perform zero-shot tasks. Given an applicant’s resume, it can say if they are a good fit for a job, or identify an animal in a photo. But does it work? As it stands, the only way to find out if a large language model (LLM) can perform a task, is to label a significant subset of data, and measure how consistent the LLM is with these labels. This data labeling process substantially decreases the disruptive nature of AI. While the AI itself may be zero-shot, the process of deciding if we should deploy it is not. Explainable AI (XAI) has long been promised as a tool to help with this kind of deployment decision. Better understanding of how, and in some cases, why AIs make decisions can increase confidence in deployment and help users identify and correct errors [1, 2]. However, precisely when you would expect XAI to be making a resurgence, it is nowhere to be seen. In no small part, this is because of fundamental limitations of the methods. Existing Explainable AI methods try to either peer inside the algorithmic black-box and use access to the internals of models (grad-cam [4], integrated gradients [5], counterfactuals [6], or mechanistic interpretability [7]) which is inapplicable to many widely used closed source LLMs that only allow access via an application programming interface (API), or 1

GT: Cat | API: Cat | RoT: Cat

GT: Dog | API: Dog | RoT: Dog

GT: Cat | API: Dog | RoT: Dog

(a) ROT explanation for GPT-4o-mini classification label ‘Cat’

(b) ROT explanation for GPT-4o-mini classification label ‘Dog’

(c) ROT explanation for an image with both classes

GT: Cat | API: Cat | RoT: Cat

GT: Dog | API: Dog | RoT: Dog

GT: Dog | API: Cat | RoT: Dog

(d) ROT explanation for GPT-4o-mini classification label ‘Cat’

(e) ROT explanation for GPT-4o-mini classification label ‘Dog’

(f) ROT explanation for an image with neither class

Figure 1: Examples of local feature importances for zero-shot image classification performed via the OpenAI GPT-4o-mini API. The model is prompted to perform zero-shot classification of pet images into labels ‘Cat’ and ‘Dog’ from a public dataset[3]. ROT is orchestrated to produce positive importances for input features indicative of ‘Cat’, highlighted red, and negative importances for features indicative of ‘Dog’, highlighted blue. ROT explanations can be computed without access to model weights, and correctly handle cases ((c) and (f)) where GPT-4o-mini disagrees with the provided labels. Here the explanation indicates that supporting regions for both classes can be found in (c), while no regions significantly support either class in (f). are combinatorially enumerative, and rely on passing a large amount of synthetic data to the model (SHAP [8], LIME [1], Counterfactual explanations [6] or Recourse [9]), which is either prohibitively expensive or simply infeasible. Feature importance explanations, such as those from LIME or SHAP, are post-hoc explanations of individual predictions in the form of a signed vector of scores, denoting the importance of each input feature to the prediction1 . Prior techniques have been variants of sensitivity analysis, operationalising the intuition that important input features are those which, upon synthetic perturbation, cause the greatest changes in model outputs. We focus on post-hoc, local (per-datapoint) feature importance explanations because they 1

LIME and SHAP have together been cited over 100,000 times, and are the most common XAI used in practice.

2

are a popular and easy to use form of XAI, do not require retraining AI systems, and can be implemented separately from the predictions they seek to explain.

2

Rule-of-Thumb: Explanations for Emerging XAI Applications

We propose “Rule-of-Thumb” (ROT), a novel form of XAI that considers important inputs to be those that are most predictive of the outputs of an AI system. Rather than formulating XAI in terms of how altering an input value would alter the system output, we ask how knowledge of particular feature values should alter our predictions of the AI system’s behaviour. The outputs from ROT are standard feature importance explanations, which can be analysed using existing visualisations such as pixel or token level saliency maps that highlight positive or negative contributions towards an output (Figures 1 and 2). These figures show the unique ability to use ROT even when model access is available only via API, where other explainers are unviable2 . For tabular data where model access

Ebony Robinson is an experienced IT professional with over two decades of expertise in systems and network administration, help desk support, and project management within academic and legal institutions. Currently serving as an Information Technology Administrator at UNH School of Law since 2014, she manages infrastructure hardware and software, including servers, network systems, security, and campus-wide applications, supporting a diverse user base of faculty, staff, and students. Her previous role as Associate Director of Computer Services at the same institution involved supporting multi-platform environments, leading campus network initiatives, and transitioning infrastructure from Novell to Microsoft systems. Ebony possesses extensive knowledge of various operating systems, server platforms, networking protocols, and hardware, with strong skills in system security, virtualization, and content management systems. She holds a Bachelor of Science in Exercise and Sports Sciences and has a proven track record of self-motivation, organization, and creative problem-solving in evolving technological landscapes.

Matthew Walsh is an experienced Biology educator and laboratory professional with a strong background in science instruction, curriculum development, and laboratory management. He has over 10 years of teaching experience, including implementing STEM, NGSS, and problem-based learning strategies, and co-teaching with special education staff. He is a Level 1 Google Certified Educator and has led professional development sessions on Google applications. In addition to teaching, Matthew has extensive laboratory management experience, including training residents, performing molecular diagnostics, and maintaining laboratory quality standards. His educational background includes a B.S. in Biological Sciences, a Post-Baccalaureate Teaching Certification from the University of Washington, and additional coursework in geosciences and education. Key skills include curriculum design, differentiated instruction, laboratory procedures, molecular techniques( PCR, DNA extraction), and educational technology. He is committed to student success, professional growth, and integrating technology into science education.

Figure 2: Resume summaries [10] from an IT-worker (Ebony Robinson, top) and a biology teacher (Matthew Walsh, bottom), that respectively pass and fail our AI filter. The zero-shot resume classification system [10] uses the GPT-4.1-nano API to find suitable candidates for an information technology (IT) role. ROT explanations indicate the predictive value of individual text tokens, visualised here by using deeper colour for greater importances. Extra information passed to ROT but found to be relatively unimportant, such as race, gender, and political preferences is not displayed. Tokens with positive contributions are green, and with negative contributions are red. 2

Sensitivity analysis via perturbation [1, 8] requires prohibitively expensive API calls for many synthetically perturbed datapoints, whereas sensitivity analysis using gradient based methods [4, 5, 11, 12] requires access to weights

3

Negative Contribution (Non-Diabetic)

0.50

=

83

5

0.45

na nc

ies

n=

= se Glu

(a) Rule of Thumb (ROT) explanation for a datapoint. Glucose and age (in blue) contribute most to reducing the predicted probability

0.40

99

0.35

co

e= Ag

SHAP baseline

Pre g

0.30

30

0.25

uli

0.20

Ins

model prediction

34

0.15

I=

99 se co

0.45

Glu

ies na nc

=

=

34 I= Pre g

0.40

BM

0.35

Positive Contribution (Diabetic)

30

0.30

RoT baseline

e=

0.25

5

0.20

BM

ure Bl = ood 54

0.15

Pre ss

Negative Contribution (Non-Diabetic)

model prediction

Ag

Positive Contribution (Diabetic)

(b) SHAP explanation for the same datapoint. Glucose (in blue) is still most important in reducing output probability, but here age (in red) increases it

Figure 3: Examples of local feature importance explanations from a Random Forest diabetes prediction system. The AI is trained on the Pima dataset [13] and for a datapoint with feature values: glucose level = 99, age = 30, BMI = 34, etc, predicts a diabetes probability of 0.235. ROT and SHAP explain this differently, but glucose is the most important input feature for both. Standard force plots visualise how individual feature importances contribute to the prediction. is common, we are able to compare with explainers like SHAP and LIME using standard force-plots for a single datapoint (Figure 3) or swarmplots for a collection of datapoints (Figure 7). In these latter comparisons, we note that applying sensitivity analysis implicitly requires several nontrivial assumptions: (i) that it is possible to query the AI system with new datapoints; (ii) the system responds similarly to synthetically perturbed data in the same way it does to real-world data, and (iii) The outputs of a system are continuous, and typically vary when a limited number of features are changed 3 . ROT makes none of these assumptions, conferring specific advantages in the following three emerging XAI application areas: 1. Explaining Zero-shot Classification with LLMs: Classification with LLMs presents two practical challenges. First, weights and gradients are inaccessible for closed systems accessed via API, and second, even when the AI system is open-source, producing each explanation requires computing many additional LLM inferences, making explanations too cumbersome and slow to compute. We experiment with a movie sentiment prediction task and a judicial case outcome prediction task to reflect each of these settings. In both cases, we find ROT explanations are aligned with expected explanations produced by human annotators. We also find ROT feasible to compute quickly on consumer hardware, and as ROT explanations do not require the training a model for each datapoint explained, our computational benefits increase with the number of data points explained. When explaining an open-weight LLM, asymptotically, 13 million new ROT explanations can be computed in the time taken for a single additional SHAP explanation. We also conduct an additional experiment filtering resume texts, where we demonstrate the unique ROT ability to explain predictions in terms of non-input features. 2. Auditing Proprietary Models: Auditing AI systems such as e-commerce recommendation engines can be challenging because access to the underlying AI is often restricted. A standard workaround is to first construct an additional mimic model to replicate the AI system, and then explain the mimic model in turn as a proxy for the restricted AI system itself. We go on to show that explanations from this commonly employed 3

This is particularly problematic when modern AI systems answer yes/no questions e.g. ”Is this a picture of a cat?”

4

workaround can be contested by varying the mimic model used; whereas ROT directly provides explanations without depending on mimic models. 3. Scientific Discovery using XAI: We identify an emerging strategy to generate scientific hypotheses, which first collects data about a phenomenon of interest, then trains an AI system using the data, and finally explains its predictions in order to provide insights into the phenomenon itself. Using sensitivity analysis based explainers here is challenging because sometimes they generate incorrect hypotheses. We analyse datasets about diabetes diagnoses, criminal justice, and lending; comparing hypotheses from ROT, SHAP, and LIME explanations. Through a carefully constructed scenario where a specific input feature is unpredictive of model behaviour, and another where a specific input feature is predictive, we demonstrate that only ROT explanations correctly identify these features as unimportant and important respectively.

2.1

Formulation

We wish to determine which features are most informative in estimating a given prediction from an AI system. Using additive functions that model the AI system outputs from individual features, we can both estimate the prediction and determine the relative importance of each feature value. We make some simplifying assumptions to answer this efficiently. Given a model C(·) defined over features J, and dataset X, we want to find the optimal set of additive functions such that for any subset of features J ⊆ J and datapoint x ∈ X we estimate C(x) using their sum.  C(x) ≈ F 

 X

fθj (xj ) + G ∀J ∈ P(J)

(1)

j∈J

where F is a sigmoid function (for classification) or an identity function (for regression), fθj (xj ) is a learnt function representing the information provided by knowing that the j th feature takes value xj , and G a global bias term that indicates what prediction should be made without knowing anything about the datapoint x. We refer to fθj (xj ) as the importance of a feature j taking value xj . This corresponds to how much a prediction of C(x) should update, on average, after discovering this information. Given a random subset of features selected using uniform probability p, we can approximate C(x) as a the sum of importances. This idea of always having a rough estimate given any amount of limited information gives rise to the name Rule of Thumb. To find the optimal weights θj ∀j ∈ J for our model, we minimize the following objective L.

5



loss between C(x) and the estimate using only features in J



  z  }| {    X  |J|  p (1 − p)(|J|−|J|) · ℓ C(x), F     fθj (xj ) + G |  {z }   j∈J  probability of selecting    feature subset J

all datapoints 

min L = min θ

θ

X

z}|{ X

J⊆J

x∈X

|{z}

all subsets of features

(2)

h P i Here the term ℓ C(x), F f (x ) + G is a standard training loss such as log loss or squared j∈J θj j error that penalizes mispredictions of C(x). This loss is not only marginalized over all datapoints, as is common in standard machine learning, but also all possible masks of the data. Although computing L exactly is expensive and requires computing the standard loss over the entire dataset 2|J| times, it can be efficiently optimized using dropout[14] applied to the feature importances fθj (xj ). Using dropout, the training time for fitting is almost the same as fitting the simple additive form of equation (1) that does not consider subsets of features. After fitting the model, the feature j with the largest absolute value |fθj (xj )| is the one that should most change our confidence in the prediction, and can be considered the most important input feature. Comparison with existing methods Gradient-based methods[4, 11, 12] are computationally efficient but require direct access to model weights, which are often restricted. With LLMs ,it is common to provide access only via API access. Gradient-based approaches cannot be applied to any model that only offers API access. Sensitivity-based approaches LIME and SHAP optimise a per-datapoint objective. While ROT masks information and fits one model for the entire dataset, these approaches peturb datapoints and fit one linear model per datapoint. This creates substantial differences in applicability – not only is sensitivity analysis badly suited for analyzing opaque AI systems that only return yes/no decisions (such as those provided by an LLM API), but it also requires many times (typically 500 for SHAP to 5000 for LIME) more inferences or API calls to explain each prediction. Finally, by definition, sensitivity analysis can explain AI systems only in terms of their inputs. ROT can utilise non-input features by fitting on extended datasets if required. This enables explaining model predictions using external models such as MobileNet[15] or BERT[16] to generate more informative explanations. See Appendix A for additional details situating ROT with respect to prior XAI approaches.

3

Experiments

Explanations obtained through sensitivity-based XAI techniques often serve as an aid for users and developers in understanding and debugging model behaviour. However, they fall short in key emerging application areas: zero shot classification with LLMs, auditing black-box AI, and scientific discovery using AI. ROT confers key advantages here: it can be used even if models are slow, expensive, accessible only via API, or inaccessible for additional inferences entirely. It can generate model explanations using input and non-input

6

features, allowing explanations to supplement inputs with other data to explain model behaviour. Finally, ROT makes no causal assumptions and is invulnerable to adversarial attacks that make other explainers ill-suited to proposing scientific hypotheses or auditing black-box systems. Compared to SHAP, ROT reveals how informative it is to know on average, that a feature takes a certain value, while SHAP reveals the average change that occurs upon altering a feature to a certain value from its mean. This is valuable because it allows us to visualise ROT explanations using the same familiar plots as SHAP explanations (eg. the force-plots in Figure 3). This relationship makes SHAP a natural basis for experimental comparisons with ROT .

3.1

Model Evaluation and Zero-Shot Classification with LLMs

LLMs are often used for zero-shot classification when there is no labelled dataset to train a classifier model. For each datapoint, an LLM is prompted to output a classification. Since there are no target labels to compare predictions with, typical model performance metrics such as accuracy, precision or recall cannot be readily computed. XAI offers a complementary path to performance evaluation, allowing us to inspect and verify that the “correct” features are predictive of the model’s behaviour. A key challenge is that new predictions can be difficult to obtain for arbitrarily modified model inputs. When using commercial models, this manifests as increased costs (by default, SHAP requires 500 additional API calls to explain a single API prediction, while LIME requires 5000). Even with an open source model, this causes explanations to be prohibitively slow to compute. By contrast ROT does not require additional API calls for explanation. To improve ROT’s predictive power on text, we generate per token embeddings and treat these as features, performing dropout at the token level. We replicate three uses of LLMs for zero-shot classification: the use of commercial LLM APIs to filter resumes [10], the use of API based LLMs to classify movie reviews [18, 19], and the use of a finetuned language model to predict the outcomes of judicial appeals from a court summary[17]. Examples

Kapur, J. 1. This appeal by special leave is directed against the judgment and order of the High Court of Judicature at Madras. The appellant is ... in support of the contention several cases were relied upon, 6. In our opinion on the findings as given the payment in dispute does not ... it did not affect the quantum of salary of the recipient. The two cases strongly relied upon by the appellant Company were J. P. Hancook v. General Reversionary and Investment Company Ltd. (1919) ... that was the ground for holding the amount to be a proper deduction. 7. In our opinion the proper test to apply in this case is ... therefore the amounts claimed is not a deductible item under S. 10 (2)(xv).

Kapur, J. 1. This appeal by special leave is directed against the judgment and order of the High Court of Judicature at Madras. The appellant is ... in support of the contention several cases were relied upon, 6. In our opinion on the findings as given the payment in dispute does not ... it did not affect the quantum of salary of the recipient. The two cases strongly relied upon by the appellant Company were J. P. Hancook v. General Reversionary and Investment Company Ltd. (1919) ... that was the ground for holding the amount to be a proper deduction. 7. In our opinion the proper test to apply in this case is ... therefore the amounts claimed is not a deductible item under S. 10 (2)(xv).

(a) A datapoint with text segments highlighted as per ROT importances

(b) The same datapoint with segments highlighted as per SHAP importances

Figure 4: Token level explanations for zero-shot text classification with LLMs. A judicial appeal is rejected by an AI system (fine-tuned RoBERTa LLM)[17] making zero-shot predictions. This prediction is explained by ROT and SHAP, with more important text segments highlighted with a deeper colour. Human annotation of the segments is indicated in bold, and aligns better with ROT than with SHAP. We present abridged texts here, with unabridged results from ROT, SHAP, LIME, and Integrated Gradients presented in Figures S2, S3, S4, S5, and S6.

7

Table 1: Measuring alignment between human annotations and token explanations. For legal case outcomes predicted by an LLM[17], binary annotations indicate which text segments from the judgement are important. Token importance explanations from an explainer can be considered to “predict” these annotations, which we evaluate using the area under the receiver-operating-characteristic curve, weighted by the length per evaluated text segment. ROT and SHAP explanations are the most aligned with human annotations.

Avg. wAUROC

R OT

SHAP (default 500 samples)

Integrated Gradients

LIME (default 5000 samples)

Random

0.77

0.74

0.65

0.62

0.47

of explanations from the resume filtering task can be seen in Figure 2. A word-cloud constructed from token-level importances across the corpus is visualised in Figure 5. Since this experiment was conducted entirely via APIs, no other explainers can be used to provide baseline comparisons. We run into the same problem with our movie review sentiment classification experiment, but we are able to compare ROT generated explanations against “ground-truth” token importances from human annotators. This provides us with a quantitative measure of the agreement between humans explaining movie review sentiments and ROT explaining an APIs classification of the same sentiments. For Judicial case outcome prediction, we take advantage of access to the model weights to compare ROT explanations with other explainers, and visualise the agreement competing explanations have with human annotations in Figure 4. Overall, we find ROT and SHAP explanations to be better aligned with human annotators, and ROT explanations fastest to compute. 3.1.1

Explaining Judicial Case Predictions from a Fine-Tuned RoBERTa Model

To test ROT explanations on LLMs, we study an AI system[17] to predict appeal outcomes in Indian courts. Figure 4 presents an abridged example of explanations from one prediction. The data includes annotation labels for individual segments in each case. These indicate which portions of the text were found important to the judgement by human annotators[17], allowing us to measure explanation quality4 by computing the area under the reciever operating characteristic (AUROC) between ROT explanation importances and known ground-truth importances per text segment. As a fine-tuned RoBERTa classifier is used here, we are able to use other explainers, and Table 1 presents AUROC scores from different explainers. A detailed analysis of program runtimes is provided in Figure 9, but it is worth noting that ROT is the only explainer able to produce results in reasonable time on consumer hardware. Details about data preparation, explanation generation, and weighted average AUROC evaluation can be found in Appendix D.2, and individual case explanations can be perused at: https://kairawal.github.io /Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems-using-Partial -Information/JudicialCaseOutcomePrediction/Code/results.html .

4

Quality here refers only to alignment with the judgement of law students, not necessarily to better explanations

8

Figure 5: Word-cloud summary of token importances for zero-shot resume classification. A zeroshot resume filter system uses the GPT-4.1-nano LLM API to find suitable candidates for an information technology (IT) role. ROT explanations indicate the importance of individual text tokens, which we visualise using an importance-weighted word cloud. Alongside IT workers which pass the filter, our corpus contains resumes of bankers, accountants, teachers, and others which our filter rejects. Tokens which increase the likelihood of making a positive prediction are in green, and those that decrease the likelihood are in red. 3.1.2

Explaining Movie Review Classifications from an LLM API

We conduct a similar experiment with zero-shot movie review sentiment classification, using an AI system built upon OpenAI’s GPT-4.1-nano LLM, accessible only via API. When compared with ground truth importance annotations[18, 19] for review sentiments, ROT achieves a weighted AUROC of 0.72, while a random baseline only achieves 0.50. Gradient based explainers (Integrated Gradients) cannot be compared, as access to GPT-4.1-nano weights is restricted, and perturbation based explainers (LIME, SHAP) are infeasible to apply even at this modest scale. This showcases a unique ROT ability, where we can explain discrete decisions without either access to its weights, ors without making additional API calls, simply by looking at predictions already made by the AI. Additional details are in Appendix D.3. 3.1.3

Explaining Resume Filtering from Proprietary LLMs

In addition to classifying movie reviews, we replicated a zero-shot resume classification system to filter candidates for information-technology (IT) jobs using the GPT-4.1-nano API [10]. ROT was then used to generate explanations identifying which parts of the resume text were important to selecting candidates as suitable for an IT position. We augment our text-based features with three attributes – race, gender, and political orientation, utilising ROT’s unique ability to produce explanations using non-input features. We verify findings from the original study indicating that, on this dataset, GPT ascribes negligible importance to race, gender, and political orientation5 when making hiring decisions. We evaluate the system with a corpus of resumes containing both IT workers and other occupations. The word-cloud in Figure 5 presents the text identified as predictive of selection and rejection across the resume corpus, and Figure 2 shows examples of an IT worker’s and a teacher’s resume, with tokens highlighted according to the ROT importances. Additional data and implementation details can be found in Appendix D.4.

3.2

Auditing Proprietary Black-Box AI Systems

We replicate an audit of Amazon’s e-commerce recommendation system conducted by The Markup [20]. Lacking access to Amazon’s AI system, we adopt the standard practice[21–24] of using a dataset of scraped 5

This was evaluated as per[10] by artificially varying names and political orientation on otherwise static resumes

9

0

Scraped Feature

(a) Explanations from The Markup’s Random Forest mimic model. Using SHAP, they report “brand is amazon” and “product reviews” to be important

40

20

20

0

0

80 60 40 20 0

random noise

0

60

shipped by amazon

25

40

100

product stars

20

80

120

top clicked

50

100

Most Important Second Most Important Third or Lesser Importance

140

sold by amazon

75

40

60

120

Relative Importance per Datapoint

160

product reviews

100

140

abs( RoT importance )

125

60

160

brand is amazon

150

Relative Importance per Datapoint

Most Second Most Third or Lesser Important Important Importance SHAP SHAP SHAP LIME LIME LIME

80

abs( LIME importance )

175

abs( SHAP importance )

200

brand is amazon product reviews sold by amazon top clicked product stars shipped by amazon random noise

abs( LIME importance )

80

Most Second Most Third or Lesser Important Important Importance SHAP SHAP SHAP LIME LIME LIME

brand is amazon product reviews sold by amazon top clicked product stars shipped by amazon random noise

abs( SHAP importance )

Relative Importance per Datapoint

100

Scraped Feature

Feature Name

(b) Explanations from the scikit-learn default Random Forest mimic model differ, but continue to indicate the importance of “brand is amazon”

(c) ROT importances obtained without mimic models: “brand is amazon” and “sold by amazon” are important, “product reviews” is unimportant

Figure 6: Auditing Amazon’s proprietary recommendation system through mimic models (Table 2), and directly with ROT. We visualize explanations via their absolute feature importances, summed over all datapoints. We also indicate the proportion of the importance sum where the feature was the first (in green) or second (in pink) most important for each datapoint. With SHAP, Markup’s mimic model shows brand and product reviews as important features (a). But the default Random Forest mimic model shows different feature importances (b), demonstrating that sensitivity based explainers produce different explanations for different mimics. This undermines the veracity of the audit. By contrast, ROT (c) does not need any mimic models or access to the proprietary model. Extended results are in Figures S8, S9, S10 and S11. listings from Amazon to train a mimic AI that predicts product rankings. We then explain mimic predictions using SHAP and LIME, and find these to vary significantly depending on the choice of mimic. This ambiguity is missing from competing ROT explanations, which explain the AI recommendation system directly without using any mimic AI. Our analysis using ROT is also able to surface insights missed by mimic based explanations. When explaining a proprietary AI system, restrictions can prevent access to model weights as well as the ability to query outputs for synthetically perturbed inputs. This usually disqualifies sensitivity based explainers, but a common workaround is to collect input-output pairs from real-world deployments of the AI system, and use this dataset to build a “mimic” AI system. With unrestricted access to the mimic, predictions can be obtained on new synthetically-perturbed inputs, allowing typical post-hoc explanations such as those from SHAP or LIME to be computed[21–24]. This is distinct from XAI that consists of an interpretable Table 2: Model accuracies for different mimic models. Amazon’s product recommendation system is not publicly accessible, making audits difficult. Different models are trained to mimic Amazon predictions, enabling audits by providing query-able interfaces for explainers. We train Random Forest models with hyperparameters from The Markup’s audit or from scikit-learn defaults, and various Logistic Regression models. They all have similar accuracies, forming a Rashomon set of models that make similar predictions.

Test Accuracy

RF (Markup)

RF (Default)

LogReg

LogReg (L1)

LogReg (L2)

0.69

0.69

0.71

0.72

0.69

10

“glass-box” surrogate model, which approximates and AI system holistically as a global explanation (different from local feature importance explanations) itself. Building a mimic AI system and then using it to compute post-hoc feature importance explanations is a distinct explanation category that involves two potential sources of error: the overall approximation of the AI system by the mimic AI system, and the local approximation of the mimic using sensitivity analysis, like SHAP. Mimics can however suffer from the “Rashomon effect” where they can make predictions identical to the original AI system, but through different internal mechanisms [25–27], yielding different local sensitivity and consequently different explanations [28, 29]. In our audit experiment, the SHAP or LIME explanations from TheMarkup’s mimic AI (Figure 6 panel a) reveal the most sensitive input feature to be whether the product brand is Amazon (“brand is amazon”), followed by “product reviews”. However, switching to a different, equally performant mimic AI (Figure 6 panel b) alters these feature importances. We find many equally performant (see Table 2) possible mimics in the Rashomon set of roughly equally performant AI systems, which seem to have different internal decision mechanisms (see Table S6). This ‘different local behaviour but equivalent performance’ casts doubt on the mimic approach. An audit using a particular mimic can be undermined by adversarially selecting a different mimic. Further, AI audit results can be contested because insights from any explanation can be attributed to the choice of mimic rather than the underlying AI system. By contrast, ROT explanations (Figure 6 panel c) can be computed without access to the original AI system or the ability to obtain outputs for new inputs. This makes ROT valuable for enforcing regulations which do not grant access to third-party AI systems or source code. For example under Article 74(12) of the EU AI Act[30], regulators are granted access to observe AI outputs on fixed input data, but not the ability to obtain new predictions on arbitrary inputs. Like other explainers, ROT also uncovers “brand is amazon” to be the most important feature. However, critically, it also shows “sold by amazon” to be an important feature, something that remained hidden in preceding mimic based analysis. This is a new insight signalling potentially self-preferencing behaviour where the Amazon platform does not just promote its own brands, but also third party product sold directly by Amazon over the same product sold by third-party merchants. This finding is harder to undermine because the explanation is independent of idiosyncrasies arising from the choice of mimic. As such, ROT explanations can be useful in identifying potentially self-preferencing practices in regulatory contexts such as the EU Digital Markets Act, where establishing causal links is not strictly necessary[31].

3.3

Scientific Discovery using XAI

We consider a diabetes prediction scenario where XAI needs to award low feature importance scores to unimportant features, and recidivism and loan approval scenarios where XAI needs to award high importance scores to important features in the presence of misleading alternatives. In both situations, ROT is good at hypothesis generation because it correctly identifies the unimportant and important input features. These explanations are being sought not to explain the behaviour of a specific AI system, but to understand the underlying data it was trained on. XAI has been advocated as a means of scientific discovery [32–36] in diverse fields like medicine[37], engineering[38], chemistry[39], and many more [40–47] Upon analyzing 1151 research articles at the intersection of XAI and science, we found 4.9% of them used 11

explanations of AI systems as a means of proposing novel scientific hypotheses. Of these, 70% used SHAP. We detail the methodology behind these statistics in Appendix D.6. XAI use in scientific discovery suffers the same challenges as AI auditing – like mimics, AI systems trained to replicate scientific phenomena are not guaranteed to faithfully duplicate underlying causal mechanisms. This can be problematic when using SHAP, because sensitivity analysis generates explanations by making predictions on synthetic, out-of-distribution datapoints, which may include physically implausible values. The behaviour of the AI system on non-realistic datapoints could be anything, and consequently the feature importances could be arbitrary [48–50].By contrast, ROT importances are inspired by feature “predictiveness” rather than sensitivity, overcoming these limitations without implicitly introducing harder problems such as learning causal models [51] or density estimation [52]. ROT also satisfies leading AI regulations better, and its advantage over explainability methods such as SHAP and LIME can even inform (future) best practices in drug discovery. Specifically, the European Medicines Agency (EMA) and US Food and Drug Administration (FDA) recently published joint guidance noting a set of principles whereby interpretability and explainability are enumerated as best model design and development practices[53]. The principles do not mandate specific XAI methods, but the guidance makes clear that transparency, reliability and accountability are key to acceptable deployment. 3.3.1

Correctly Identifying Uninformative Features

If an explainer indicates that a specific feature impacts a phenomenon of interest, but that feature is extraneous to the phenomenon, then the explainer is poorly suited for scientific discovery. But sensitivity based explainers like SHAP and LIME consider any input feature important if applying any perturbations to it leads to changes in the AI-modeled output phenomenon[48] , even if the changes are not meaningful. Uninformative features

Age 0.25 0.00 0.25 0.50 0.75 1.00 1.25

Low

RoT importance (Contribution to Model Output)

(a) ROT finds age to be unimportant

BMI

BMI

Age

Age

Diabetes Pedigree Func. Blood Pressure Pregnancies Skin Thickness Insulin 0.4

0.2

0.0

0.2

0.4

Low

High

Glucose

Feature value

Feature value

Glucose Diabetes Pedigree Func. Blood Pressure Insulin Skin Thickness Pregnancies

High

Glucose

Feature value

High

BMI

Diabetes Pedigree Func. Blood Pressure Pregnancies Skin Thickness Insulin 0.2 0.1 0.0 0.1 0.2 0.3

Low

SHAP importance (Contribution to Model Output)

LIME importance (Contribution to Model Output)

(b) SHAP finds age to be somewhat important

(c) LIME finds age to be somewhat important

Figure 7: ROT explanations do not erroneously find unimportant features to be important. An AI system for diabetes prediction is trained on “Pima Indians data”[13] with a synthetic transformation such that age is uncorrelated with diabetes. For this transformed data, XAI should ideally consider age to be unimportant. Swarmplots display feature importances across the dataset. While ROT correctly ascribes negligible importance to age, sensitivity based explainers SHAP and LIME find it to be somewhat important. When using XAI for scientific discovery, ROT thus provides clear benefits over SHAP and LIME.

12

can inadvertently seem important this way. We consider the same diabetes prediction task as in Figure 3, using a dataset specially collected to study the disease [13]. Using a predictive AI system in such a manner is one way to study diabetes [54]. However, for our experiment, we manually transform the dataset using FairPCA[55] to ensure that the “age” input feature is statistically uncorrelated with the “diabetes” target feature being predicted. Since age has been deliberately rendered uninformative, an XAI technique useful for scientific discovery should correctly identify that it does not affect diabetes. Figure 7 displays feature importances from ROT, SHAP, and LIME on transformed diabetes data. ROT is the only explainer to consider age unimportant. SHAP and LIME both lead to spurious hypotheses that indicate the importance of age in diabetes prediction. However, if our interest is in learning about the laws and relations that govern the data and phenomenon of diabetes, ROT is the better method because it correctly identifies age as unpredictive. Model details and our use of FairPCA[55] to ensure statistical independence between age and diabetes are detailed in Appendix D.7. 3.3.2

Identifying Important Features in the presence of Misleading Features

We have seen that explainers can be vulnerable to arbitrary out-of-distribution behaviour from an AI system. Adversarial attacks have been proposed to exploit this vulnerability by modifying AI systems to deliberately poison out-of-distribution outputs [56]. This effectively presents different behaviour to SHAP or LIME than to real-world inputs, masking the actual importances of “sensitive” features and manipulating explainers into selecting misleading “foil” features as highly important instead. In black-box AI audits, adversial

LIME

Foil Fea ture (s)

All Other Features

80%

100% 0%

0% 100%

(a) Distribution of feature importances from SHAP and ROT explanations across “sensitive”, “foil” or other features for an adversarial model

re

60%

atu

40%

40%

80%

20% 20%

e Fe

SHAP

80%

60%

60%

re

40%

atu

60%

40%

e Fe

60%

80%

sitiv Sen

sitiv Sen

40%

100% 0%

20%

100% RoT

80%

RoT

20%

0%

100%

Foil Fea ture (s)

0%

20% 20%

40%

60%

All Other Features

80%

0% 100%

(b) Distribution of feature importances from LIME and ROT explanations across “sensitive”, “foil” or other features for an adversarial model

Figure 8: ROT explanations are immune to manipulation from certain adversarial attacks. The adversarial attack misleads explainers into ascribing importance to spurious “foil” features when the underlying AI system actually makes predictions using a “sensitive” demographic feature. We plot the distribution of feature importances across the “sensitive”, “foil”, and other features, and find that both SHAP and LIME are misled by the adversary, but ROT correctly ascribes importance only to the “sensitive” feature.

13

attacks cause “fairwashing” by allowing malicious actors to deliberately hide AI systems with discriminatory behaviour behind benign-looking explanations [56]. For scientific discovery, the same thing can cause candidate hypotheses generated using XAI to be misleading. To verify that ROT does not generate such spurious explanations, we replicate adversarial AI systems for credit-scoring and criminal justice, which deliberately hide the importance of an important “sensitive” feature by introducing one or more misleading “foil” features. We find that this is ineffective at misleading ROT, which is able to correctly identify the sensitive features (race for recidivism prediction, and gender for loan approvals) as important and the foil features (simulated, extraneous features) as unimportant. In Figure 8 we show the distribution of SHAP, LIME, and ROT importances over three feature categories: sensitive, adversarial, and other, from AI systems subjected to an adversarial attack [56]. Details about the datasets used, model behaviours introduced, “sensitive” input features masked and adversarially misleading “foil” features introduced are available in Appendix D.8. We report detailed statistics about the proportion of explanations that identified the sensitive feature as most important (success), or the foil feature (adversarial failure) or other features (non-adversarial failure) in Table S7. Across 10 experiments, for LIME or SHAP the rate of recovering the sensitive feature as most important does not exceed 5% except once, where adversarial failures (foil feature is most important) still outnumber success (sensitive feature is most important). ROT has a success rate always greater than 89%, and in 6 of the 10 experiments ROT shows a perfect 100% recovery rate, always correctly identifying the sensitive feature to be the most important.

3.4

Computational Efficiency and Environmental Impacts

Of all the experiments listed so far, generating ROT, SHAP, LIME, and Integrated Gradients explanations for Judicial Case Outcome Prediction with a finetuned LLM was the largest in terms of dataset size and computations involved for prediction. For this experiment, generating the first ROT explanation took 1160 seconds, and each additional explanation took less than 0.1 milliseconds. By contrast, the average time taken for each SHAP explanation was 13 million times longer at 1320 seconds. This was without GPU acceleration, to simulate consumer hardware conditions. In practice, it is impractical to compute SHAP or LIME explanations for LLM predictions, except if only a handful of datapoints need explaining. Gradient methods are quicker, but require access to model weights when most commercial LLMs can only be accessed via API. ROT is the first explainer that can actually be deployed in practice, bringing feature importance XAI to LLMs. ROT was also found to be orders of magnitude faster than SHAP and LIME in our benchmark XAI comparison, detailed in Appendix D.9, which we document in Tables S4 and S5. This increase in computational efficency, evidenced by faster program running times, is a direct result of ROT’s unique formulation that does not require obtaining additional predictions for each explanation. Local feature importance explainers often function by collecting data about an AI system’s behaviour and then fitting an interpretable model. Both SHAP and LIME need to fit a new interpretable model for each prediction they explain. This requires data collection by making arbitrarily time-consuming calls to the AI system for each additional explanation. By contrast, ROT uses dropout to fit a single interpretable ensemble on the entire unperturbed dataset, and each 14

106

Cumulative Runtime (log)

Time Taken (log)

104

105

102

104

RoT SHAP Int. Grad. LIME RoT average SHAP average I.G. average

100

10 2 10 4 0

20

40

60

103 102 101

80 100

Case Number

1 7 19

40

Time Taken (Quickest First) RoT SHAP Int. Grad. LIME 60 80 100

Number of Cases Explained

(a) Explanation Runtimes: Judicial Case Outcome Prediction with LLMs

(b) Cumulative Explanation Runtimes (Quickest First): Case Prediction

Figure 9: Comparing runtimes for ROT and other explainers from the Judicial Case Outcome Prediction Experiment. While the first ROT explanation takes time, additional explanations are virtually instantaneous, unlike SHAP, LIME, or Integrated Gradients. The time-complexity of SHAP thus appears linear in number of cases, while ROT appears to be constant, and 13 million RoT explanations can be computed for each additional SHAP explanation, after the first. Even the 7 fastest SHAP explanations take as long as all 101 ROT explanations. explanation is generated using per-feature importance scores from this model. Since this inference is quicker than continuous training for each explanation, this makes ROT extremely fast once fit. When explaining predictions from large AI systems such as computationally intensive LLMs, the efficiency gains from ROT would translate directly to reduced environmental impacts.

4

Regulations Concerning Emerging XAI Applications

Compared to other XAI methods, ROT has a range of strengths that make it an ideal tool for making existing AI systems auditable and accountable. Its narrow focus on directly predicting the behaviour of an existing system on real data, gives it robustness against adversarial attacks, and removes the need for mimic models. This gives it a clear advantage in meeting the EU AI Act’s requirements for accuracy, robustness, and adversarial testing of high-risk AI systems under Article 15 and for robustness and resilience under Article 15 (4) – (5) [30]. This in turn, supports transparency and instructions of use requirements under Article 13 (1) and (3) (b) (ii). ROT’s computational efficiency compared to approaches requiring construction of a mimic model give it clear advantages in meeting environmental sustainability goals [57]. Most importantly, it allows auditing proprietary models, overcoming inherent transparency limitations found in many AI regulatory frameworks [58]. ROT enables participatory governance of both narrow and general-purpose AI systems, facilitating “independent external testing” and adversarial testing of general-purpose systems with systemic risk, as required under Recital 114; Article 55 (1) (a) of the AI Act [59]. In this regard, ROT may prove particularly

15

valuable for enforcement of the AI Act. While Article 74(13) allows regulators to access testing datasets and model outputs, it does not always allow full model access or the ability to study model responses on new data, as would be required for typical sensitivity based explainers. RoT can also be a powerful tool to identify and challenge anti-competitive practices of major technology companies that are increasingly embedding AI systems across their platforms. ROT can help identify selfpreferencing practices, such as search result manipulation, that may be illegal under horizontal competition laws and the EU Digital Markets Act (DMA) that require establishing a link between model behaviour and anti-competitive practices [60, 61]. We elaborate further on the specific methods and limitations of ROT in Appendix C. In addition to auditing, explainability and interpretability are also important safeguards for AI use in drug development[62] and related scientific discovery[63, 64], with each fulfilling distinct expectations depending on context. For instance, the EMA notes that for the review and monitoring of black box models, explainability metrics, such as SHAP and LIME should be used[64]. The EMA–FDA joint guidance provides a broad view on the use of AI in evidence generation and monitoring across all phases of a medicine including early research. While this recent document does not recommend a specific explainability metric, it aims to promote a baseline on the use of transparent models that are reliable, robust, generalisable and utilises data that is ‘fit-for-purpose’ [53]. Here we see the added value of ROT, clarifying why SHAP and LIME would not be sufficient for ensuring accountability, transparency and reliability, and providing the more appropriate baseline for future policy guidance and reform.

5

Conclusion

This paper presents a new approach to explainable AI based on the predictiveness of individual feature values, rather than their sensitivity to alteration. This Rule of Thumb approach allows us to approximate any decision making process by summing the importance response over any subset of feature values. The most useful feature values for a prediction are naturally those with largest importance, and their predictive power overrides the contribution of others. As such, ROT can be understood as a post-hoc and per-datapoint explainer with more in common with ablation studies than sensitivity analysis. ROT addresses several long-standing concerns in the community about the validity and practical utility of XAI, and importantly does so without making additional assumptions. It is flexible and useful in a range of emerging XAI applications areas where other forms of XAI simply can not be used – evaluating LLMs and explaining their behaviour in zero-shot classification, auditing proprietary AI systems where we can only observe their predictions and not alter input features, and using XAI to understand real-world phenomena and propose novel scientific hypotheses. As a local, post-hoc explanation method, ROT can be used as a drop in replacement for other explainers in existing XAI codebases, and visualised the same way, providing a fast and computationally efficient (and therefore environmentally friendly) method of explaining AI.

16

Acknowledgments The work of all the authors has been supported through research funding provided by the Wellcome Trust (grant nr 223765/Z/21/Z), Sloan Foundation (grant nr G-2021-16779), Department of Health and Social Care, EPSRC (grant nr EP/Y019393/1), and Luminate Group. Their funding supports the Trustworthiness Auditing for AI project and Governance of Emerging Technologies research programme at the Oxford Internet Institute, University of Oxford. Kai Rawal is funded by an EPSRC doctoral scholarship. The work of Sandra Wachter, Daria Onitiu, Chris Russell, and Kai Rawal has also been supported by the Alexander von Humboldt Foundation in the framework of the Alexander von Humboldt Professorship (Humboldt Professor of Technology and Regulation) endowed by the Federal Ministry of Education and Research via the Hasso Plattner Institute. Data and materials availability: All code used in our experiments can be found at https://github .com/KaiRawal/Rule-of-Thumb-Explaining-Artificial-Intelligence-Systems -using-Partial-Information . The scripts rely on permissively licensed publicly available datasets, and instructions for downloading these, where applicable, are included with the code.

References [1] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ”why should i trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10.1145/2939672.2939778. URL https://doi.org/10.1145/2939672.2939778. [2] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 618–626, October 2017. [3] Bhavik Jikadara. Cats and Dogs Classification Dataset. https://www.kaggle.com/datasets/ bhavikjikadara/dog-and-cat-classification-dataset, 2024. Kaggle dataset. [4] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014. URL http://arxiv.org/abs/1312.6034. [5] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/sundararajan17a.html. 17

[6] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2018. [7] Leonard Bereska and Stratis Gavves. Mechanistic Interpretability for AI Safety - A Review. Transactions on Machine Learning Research, 2024. URL https://mlanthology.org/tmlr/2024/ bereska2024tmlr-mechanistic/. [8] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 4768–4777, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964. [9] Kaivalya Rawal and Himabindu Lakkaraju. Beyond individualized recourse: Interpretable and interactive summaries of actionable recourses. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12187–12198. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/ paper/2020/file/8ee7730e97c67473a424ccfeff49ab20-Paper.pdf. [10] Akshaj Kumar Veldanda, Fabian Grob, Shailja Thakur, Hammond Pearce, Benjamin Tan, Ramesh Karri, and Siddharth Garg. Investigating hiring bias in large language models. In R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023. URL https://openreview. net/forum?id=erl90pLIH0. [11] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, abs/1706.03825, 2017. URL http://arxiv.org/abs/ 1706.03825. [12] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 3145–3153. JMLR.org, 2017. [13] Jack W Smith, James E Everhart, William C Dickson, William C Knowler, and Robert Scott Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the annual symposium on computer application in medical care, page 261, 1988. [14] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a. html. [15] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.

18

[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423. [17] Shubham Kumar Nigam, Anurag Sharma, Danush Khanna, Noel Shallum, Kripabandhu Ghosh, and Arnab Bhattacharya. Legal judgment reimagined: PredEx and the rise of intelligent AI interpretation in Indian courts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 4296–4315, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.255. URL https://aclanthology.org/2024.findings-acl.255/. [18] Omar Zaidan, Jason Eisner, and Christine Piatko. Using ”annotator rationales” to improve machine learning for text categorization. In Proceedings of the conference of the North American chapter of the Association for Computational Linguistics (NAACL), pages 260–267, 2007. [19] Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. Eraser: A benchmark to evaluate rationalized nlp models, 2019. [20] Adrianne Jeffries and Leon Yin. Amazon puts its own “brands” first above better-rated products. The Markup, October 2021. URL https://themarkup.org/amazons-advantage/2021/10/ 14/amazon-puts-its-own-brands-first-above-better-rated-products. [21] Krishna Gade, Sahin Geyik, Krishnaram Kenthapadi, Varun Mithal, and Ankur Taly. Explainable ai in industry: Practical challenges and lessons learned. In Companion Proceedings of the Web Conference 2020, WWW ’20, page 303–304, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450370240. doi: 10.1145/3366424.3383110. URL https://doi.org/10.1145/ 3366424.3383110. [22] Arize AI. Surrogate Model. URL https://docs.arize.com/arize/machine-learning/ how-to-ml/explainability/surrogate-model. Accessed: 2024-08-02. [23] Fiddler AI.

Surrogate Models.

URL https://docs.fiddler.ai/ui-guide/

explainability-ui-giude/surrogate-models. Accessed 2024-08-02. [24] Microsoft.

Model

interpretability

-

Azure

Machine

Learning,

2023.

URL

https://learn.microsoft.com/en-us/azure/machine-learning/ concept-model-interpretability. Accessed 2024-08-02. [25] Lucas Monteiro Paes, Rodrigo Cruz, Flavio P. Calmon, and Mario Diaz. On the inevitability of the rashomon effect. In 2023 IEEE International Symposium on Information Theory (ISIT), pages 549–554, 2023. doi: 10.1109/ISIT54713.2023.10206657. 19

[26] Cynthia Rudin, Chudi Zhong, Lesia Semenova, Margo Seltzer, Ronald Parr, Jiachang Liu, Srikar Katta, Jon Donnelly, Harry Chen, and Zachery Boner. Amazing things come from having many good models, 2024. URL https://arxiv.org/abs/2407.04846. [27] Xavier Renard, Thibault Laugel, and Marcin Detyniecki. Understanding prediction discrepancies in classification. Machine Learning, Aug 2024. ISSN 1573-0565. doi: 10.1007/s10994-024-06557-4. URL https://doi.org/10.1007/s10994-024-06557-4. [28] Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6765–6774. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/marx20a.html. [29] Sebastian Müller, Vanessa Toborek, Katharina Beckh, Matthias Jakobs, Christian Bauckhage, and Pascal Welke. An empirical evaluation of the rashomon effect in explainable machine learning. In Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Elena Baralis, and Francesco Bonchi, editors, Machine Learning and Knowledge Discovery in Databases: Research Track, pages 462–478, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-43418-1. [30] European Union. Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence and amending certain union legislative acts (artificial intelligence act). Official Journal of the European Union, L 202, 12 July 2024, pp. 1–241, 2024. URL https://eur-lex.europa.eu/legal-content/EN/TXT/?uri= CELEX%3A32024R1689. OJ L 202, 12.7.2024, p. 1–241. [31] European Union. Regulation (eu) 2022/1925 of the european parliament and of the council of 14 september 2022 on contestable and fair markets in the digital sector and amending directives (eu) 2019/1937 and (eu) 2020/1828 (digital markets act). https://eur-lex.europa.eu/eli/ reg/2022/1925/oj, 2022. OJ L 265, 12.10.2022, p. 1–66. [32] Ribana Roscher, Bastian Bohn, Marco F. Duarte, and Jochen Garcke. Explainable machine learning for scientific insights and discoveries. IEEE Access, 8:42200–42216, 2020. doi: 10.1109/ACCESS.2020. 2976199. [33] Jacques A. Esterhuizen, Bryan R. Goldsmith, and Suljo Linic. Interpretable machine learning for knowledge generation in heterogeneous catalysis. Nature Catalysis, 5(3):175–184, Mar 2022. ISSN 2520-1158. doi: 10.1038/s41929-022-00744-z. URL https://doi.org/10.1038/ s41929-022-00744-z. [34] Richard Dybowski. Interpretable machine learning as a tool for scientific discovery in chemistry. New J. Chem., 44:20914–20920, 2020. doi: 10.1039/D0NJ02592E. URL http://dx.doi.org/10. 1039/D0NJ02592E.

20

[35] Quinn Dickinson and Joel G. Meyer. Positional shap (poshap) for interpretation of machine learning models trained from biological sequences. PLOS Computational Biology, 18(1):e1009736, 2022. doi: 10.1371/journal.pcbi.1009736. URL https://journals.plos.org/ploscompbiol/ article?id=10.1371/journal.pcbi.1009736. [36] Mateusz Garbulowski, Klev Diamanti, Karolina Smolińska, Nicholas Baltzer, Patricia Stoll, Susanne Bornelöv, Aleksander Øhrn, Lars Feuk, and Jan Komorowski. R.rosetta: an interpretable machine learning framework. BMC Bioinformatics, 22(1):110, Mar 2021. ISSN 1471-2105. doi: 10.1186/ s12859-021-04049-z. URL https://doi.org/10.1186/s12859-021-04049-z. [37] Chukwuebuka Joseph Ejiyi, Zhen Qin, Joan Amos, Makuachukwu Bennedith Ejiyi, Ann Nnani, Thomas Ugochukwu Ejiyi, Victor Kwaku Agbesi, Chidimma Diokpo, and Chidinma Okpara. A robust predictive diagnosis model for diabetes mellitus using shapley-incorporated machine learning algorithms. Healthcare Analytics, 3:100166, 2023. ISSN 2772-4425. doi: https://doi.org/10.1016/j. health.2023.100166. URL https://www.sciencedirect.com/science/article/pii/ S2772442523000333. [38] Liangliang Hu, Xiaolin Meng, Yilin Xie, Craig Hancock, George Ye, and Yan Bao. Examination of load-deformation characteristics of long-span bridges in harsh natural environments based on real-time updating artificial neural network. Engineering Structures, 308:118022, 2024. ISSN 0141-0296. doi: https://doi.org/10.1016/j.engstruct.2024.118022. URL https://www.sciencedirect.com/ science/article/pii/S0141029624005844. [39] Qiming Fu, Tao Xu, Chenggong He, Daomiao Wang, Meiling Liu, and Chao Liu. Machine learningassisted study of ren(x)c(6-x)-doped graphene as potential electrocatalysts for oxygen electrode reactions. Langmuir, 40(20):10726–10736, 2024. doi: 10.1021/acs.langmuir.4c00803. Epub 2024 May 8. [40] Gian Maria Campedelli. Explainable machine learning for predicting homicide clearance in the united states. Journal of Criminal Justice, 79:101898, 2022. ISSN 0047-2352. doi: https://doi. org/10.1016/j.jcrimjus.2022.101898. URL https://www.sciencedirect.com/science/ article/pii/S0047235222000186. [41] Xinyi Niu, Chengpeng Lu, Ying Zhang, Yong Zhang, Chengcheng Wu, Ebrima Saidy, Bo Liu, and Longcang Shu. Hysteresis response of groundwater depth on the influencing factors using an explainable learning model framework with shapley values. Science of The Total Environment, 904:166662, 2023. ISSN 0048-9697. doi: https://doi.org/10.1016/j.scitotenv.2023.166662. URL https://www. sciencedirect.com/science/article/pii/S0048969723052877. [42] Andrés Cremades, Sergio Hoyas, Rahul Deshpande, Pedro Quintero, Martin Lellep, Will Junghoon Lee, Jason P. Monty, Nicholas Hutchins, Moritz Linkmann, Ivan Marusic, and Ricardo Vinuesa. Identifying regions of importance in wall-bounded turbulence through explainable deep learning. Nature Communications, 15(1):3864, May 2024. ISSN 2041-1723. doi: 10.1038/s41467-024-47954-6. URL https://doi.org/10.1038/s41467-024-47954-6. 21

[43] Zhen Sun, João Santos, Elsa Caetano, and Catarina Oliveira. Interpreting cumulative displacement in a suspension bridge with a physics-based characterisation of environment and roadway/railway loads. Journal of Civil Structural Health Monitoring, 13(2):387–397, Mar 2023. ISSN 2190-5479. doi: 10.1007/s13349-022-00647-4. URL https://doi.org/10.1007/s13349-022-00647-4. [44] Shuo Wang, Tianzhuo Zhang, Ziheng Li, and Jinglan Hong. Exploring pollutant joint effects in disease through interpretable machine learning. Journal of Hazardous Materials, 467:133707, 2024. ISSN 03043894. doi: https://doi.org/10.1016/j.jhazmat.2024.133707. URL https://www.sciencedirect. com/science/article/pii/S0304389424002863. [45] Zicheng Cao, Zekai Qiu, Feng Tang, Shiwen Liang, Yinghan Wang, Haoyu Long, Cai Chen, Bing Zhang, Chi Zhang, Yaqi Wang, Kang Tang, Jing Tang, Junhong Chen, Chunhui Yang, Yuzhe Xu, Yulin Yang, Shenglan Xiao, Dechao Tian, Guozhi Jiang, and Xiangjun Du. Drivers and forecasts of multiple waves of the coronavirus disease 2019 pandemic: A systematic analysis based on an interpretable machine learning framework. Transboundary and Emerging Diseases, 69(5):e1584–e1594, Sep 2022. doi: 10.1111/tbed.14492. URL https://doi.org/10.1111/tbed.14492. Epub 2022 Mar 13. [46] Xinyuan Cao, Jisi Huang, Kexin Du, Yawen Tian, Zhixin Hu, Zhu Luo, Jinlong Wang, and Yanbing Guo. Machine-learning-assisted descriptors identification for indoor formaldehyde oxidation catalysts. Environmental Science & Technology, 58(19):8372–8379, 2024. doi: 10.1021/acs.est.4c01691. Epub 2024 May 1. [47] Ruikun Gou, Jinshu Chi, Jiangong Liu, Yunpeng Luo, Ankit Shekhar, Lidong Mo, and Guanghui Lin. Atmospheric water demand constrains net ecosystem production in subtropical mangrove forests. Journal of Hydrology, 630:130651, 2024. ISSN 0022-1694. doi: https://doi.org/10.1016/j. jhydrol.2024.130651. URL https://www.sciencedirect.com/science/article/pii/ S0022169424000453. [48] I. Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle Friedler. Problems with shapley-value-based explanations as feature importance measures, 2020. URL https://arxiv. org/abs/2002.11097. [49] Dominik Janzing, Lenon Minorics, and Patrick Blöbaum. Feature relevance quantification in explainable ai: A causal problem, 2019. URL https://arxiv.org/abs/1910.13413. [50] Muhammad Faaiz Taufiq, Patrick Blöbaum, and Lenon Minorics. Manifold restricted interventional shapley values, 2023. URL https://arxiv.org/abs/2301.04041. [51] Leland Gerson Neuberg. Causality: Models, reasoning, and inference, by judea pearl, cambridge university press, 2000. Econometric Theory, 19(4):675–685, 2003. doi: 10.1017/S0266466603004109. [52] Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998. ISBN 978-0-471-03003-4.

22

[53] European Medicines Agency and U.S. Food and Drug Administration. Guiding principles of good AI practice in drug development. Technical report, European Medicines Agency and U.S. Food and Drug Administration, January 2026. URL https://www.ema.europa.eu/en/documents/ other/guiding-principles-good-ai-practice-drug-development_en.pdf. Principle 7: Model design and development practices. [54] Victor Chang, Jozeene Bailey, Qianwen Ariel Xu, and Zhili Sun. Pima indians diabetes mellitus classification based on machine learning (ML) algorithms. Neural Comput Appl, pages 1–17, March 2022. [55] Matthäus Kleindessner, Michele Donini, Chris Russell, and Muhammad Bilal Zafar. Efficient fair pca for fair representation learning. In International Conference on Artificial Intelligence and Statistics, pages 5250–5270. PMLR, 2023. [56] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20, page 180–186, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371100. doi: 10.1145/3375627.3375830. URL https: //doi.org/10.1145/3375627.3375830. [57] Philipp Hacker. Sustainable ai regulation. Common Market Law Review, 61(2), 2024. [58] Sandra Wachter. Limitations and Loopholes in the EU AI Act and AI Liability Directives: What This Means for the European Union, the United States, and Beyond. Yale Journal of Law & Technology, 26(3), 2024. doi: 10.2139/ssrn.4924553. URL https://ssrn.com/abstract=4924553. Available at SSRN: https://ssrn.com/abstract=4924553 or http://dx.doi.org/10.2139/ssrn.4924553. [59] European Commission.

Contents of the code of practice on generative ai.

https://

digital-strategy.ec.europa.eu/en/policies/contents-code-gpai, 2024. Accessed: 2025-09-10. [60] European Court of Justice. Google LLC and Alphabet Inc v European Commission, 2024. URL https: //eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62022CJ0048. Case C-48/22. [61] General Court of the European Union (Ninth Chamber, Extended Composition). Alphabet v. Commission (Google Shopping), 2021.

Google and

URL https://curia.europa.eu/

juris/document/document.jsf?text=&docid=250881&pageIndex=0&doclang= en&mode=lst&dir=&occ=first&part=1&cid=10134666. Case T-612/17. [62] U.S. Food and Drug Administration. Considerations for the use of artificial intelligence to support regulatory decision-making for drug and biological products. Draft guidance for industry and other interested parties, U.S. Food and Drug Administration, January 2025. URL https://www.fda. gov/media/184830/download. Draft guidance; distributed for comment purposes only. 23

[63] U.S. Food and Drug Administration. Using artificial intelligence & machine learning in the development of drug & biological products. Discussion paper and request for feedback, U.S. Food and Drug Administration, February 2025. URL https://www.fda.gov/media/167973/download. Originally published May 2023; revised February 2025. [64] European

of

artifi-

Reflection

Paper

EMA/CHMP/CVMP/83833/2023, European Medicines Agency, September 2024.

URL

cial

Medicines

intelligence

Agency.

(AI)

in

the

Reflection medicinal

paper

product

on

the

lifecycle.

use

https://www.ema.europa.eu/en/documents/scientific-guideline/ reflection-paper-use-artificial-intelligence-ai-medicinal-product-lifecycle_ en.pdf. [65] Ankur saved

Taly one

and of

the

Kaz

Sato.

largest

ml

Monitoring services

in

feature

trouble,

attributions:

September

29

How

google

2021.

URL

https://cloud.google.com/blog/topics/developers-practitioners/

monitoring-feature-attributions-how-google-saved-one-largest-ml-services-troubl Google Cloud Blog. [66] Jianlong Zhou and Fang Chen. Human and Machine Learning: Visible, Explainable, Trustworthy and Transparent. Springer Publishing Company, Incorporated, 1st edition, 2018. ISBN 3319904027. [67] Luca Deck, Jakob Schoeffer, Maria De-Arteaga, and Niklas Kühl. A critical survey on fairness benefits of explainable ai. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’24. ACM, June 2024. doi: 10.1145/3630106.3658990. URL http://dx.doi.org/10.1145/ 3630106.3658990. [68] Andrea Saltelli, Marco Ratto, Stefano Tarantola, and Francesca Campolongo. Sensitivity analysis for chemical models. Chemical Reviews, 105(7):2811–2828, 2005. doi: 10.1021/cr040659d. URL https://doi.org/10.1021/cr040659d. PMID: 16011325. [69] H Rabitz, M Kramer, and D Dacol. Sensitivity analysis in chemical kinetics. Annual Review of Physical Chemistry, 34(Volume 34, 1983):419–461, 1983. ISSN 1545-1593. doi: https://doi.org/ 10.1146/annurev.pc.34.100183.002223. URL https://www.annualreviews.org/content/ journals/10.1146/annurev.pc.34.100183.002223. [70] Richard H. McCuen. The role of sensitivity analysis in hydrologic modeling. Journal of Hydrology, 18(1):37–53, 1973. ISSN 0022-1694. doi: https://doi.org/10.1016/0022-1694(73)90024-3. URL https://www.sciencedirect.com/science/article/pii/0022169473900243. [71] Linda Lilburne and Stefano Tarantola. Sensitivity analysis of spatial models. International Journal of Geographical Information Science, 23(2):151–168, 2009. doi: 10.1080/13658810802094995. URL https://doi.org/10.1080/13658810802094995.

24

[72] Hans B. Nielsen. Sensitivity analysis in linear systems (assem deif). SIAM Review, 31:706–707, 1989. doi: 10.1137/1031159. URL https://doi.org/10.1137/1031159. [73] Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen. Ablation studies in artificial neural networks. CoRR, abs/1901.08644, 2019. URL http://arxiv.org/abs/1901. 08644. [74] B. Pegourie, J.-M. Picchiottino, H.-W. Drawin, A. Geraud, and M. Chatelier. Pellet ablation studies on tore supra. Nuclear Fusion, 33(4):591, apr 1993. doi: 10.1088/0029-5515/33/4/I06. URL https: //dx.doi.org/10.1088/0029-5515/33/4/I06. [75] Michael Keidar, Iain D. Boyd, Anthony Williams, and Richard Beyer. Ablation study in a capillary sustained discharge. IEEE Transactions on Magnetics, 43(1):308–312, 2007. doi: 10.1109/TMAG. 2006.887672. [76] Arnon Rosenthal. The gdnf protein familygene ablation studies reveal what they really do and how. Neuron, 22(2):201–203, 1999. ISSN 0896-6273. doi: https://doi.org/10.1016/S0896-6273(00)81077-6. URL https://www.sciencedirect.com/science/article/pii/S0896627300810776. [77] Elisabeth A. Murray. What have ablation studies told us about the neural substrates of stimulus memory?

Seminars in Neuroscience, 8(1):13–22, 1996. ISSN 1044-5765. doi: https://doi.org/

10.1006/smns.1996.0003. URL https://www.sciencedirect.com/science/article/ pii/S1044576596900032. [78] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2668–2677. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/kim18d.html. [79] Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in neural information processing systems, 34:4699–4711, 2021. [80] Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: towards a transparent evaluation of post hoc model explanations. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781713871088. [81] Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. Why a right to explanation of automated decision-making does not exist in the general data protection regulation. International data privacy law, 7(2):76–99, 2017. [82] Andrew D Mitchell, Dominic Let, and Lingxi Tang. Ai regulation and the protection of source code. International Journal of Law and Information Technology, 31(4):283–301, 2023. 25

[83] David Hartmann, José Renato Laranjeira de Pereira, Chiara Streitbörger, and Bettina Berendt. Addressing the regulatory gap: moving towards an eu ai audit ecosystem beyond the ai act by including civil society. AI and Ethics, pages 1–22, 2024. [84] Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, et al. Black-box access is insufficient for rigorous ai audits. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2254–2272, 2024. [85] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Why fairness cannot be automated: Bridging the gap between eu non-discrimination law and ai. Computer Law & Security Review, 41:105567, 2021. [86] Sandra Wachter. The theory of artificial immutability: Protecting algorithmic groups under antidiscrimination law. Tul. L. Rev., 97:149, 2022. [87] European Union. Regulation (eu) 2022/2065 of the european parliament and of the council of 19 october 2022 on a single market for digital services and amending directive 2000/31/ec (digital services act). Official Journal of the European Union, 10 2022. URL https://eur-lex.europa.eu/eli/ reg/2022/2065/oj. OJ L 277, 27.10.2022, p. 1–102. [88] United States District Court for the District of Columbia. United States of America et al. v. Google LLC, 2024. Case pending. [89] Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024. URL https://arxiv.org/ abs/2412.13663. [90] Takaya Saito and Marc Rehmsmeier. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One, 10(3):e0118432, March 2015. [91] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423/. [92] Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. pas recidivism algorithm, May 2016.

How we analyzed the com-

URL https://www.propublica.org/article/

how-we-analyzed-the-compas-recidivism-algorithm. [93] Michael Redmond. Communities and Crime. UCI Machine Learning Repository, 2002. DOI: https://doi.org/10.24432/C53W3X. 26

[94] Hans Hofmann. Statlog (German Credit Data). UCI Machine Learning Repository, 1994. DOI: https://doi.org/10.24432/C5NC77. [95] Yuji Ikeda. yuzie007/mpltern: 1.0.4. Zenodo, 2024. URL https://doi.org/10.5281/zenodo. 11068993. Version 1.0.4. [96] J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3):90–95, 2007. doi: 10.1109/MCSE.2007.55.

27

Supplementary Information A

Motivation and Related Work

A.1

Explainability in Machine Learning, Feature Attribution Methods, and SHAP

Machine learning models range from inherently interpretable (eg. decision stumps, linear regressions, rulesets, or single nearest neighbour classifiers) to uninterpretable black boxes (eg. random forest ensembles, deep neural networks, or markov random fields). In order to explain black-box models, methods have been developed to produce post-hoc model explanations (eg. saliency maps, counterfactual explanations, anchors, prototypes, etc). These can be either global, explaining model behaviour in general (eg. PDPs, Functional Decomposition, Mechanistic Interpretability etc), or local, explaining individual predictions made by a model (eg. ICE, LIME, anchors, counterfactuals). Some model explanation methods are specific to certain kinds of models (eg. TCAV, Saliency Maps, gini-importance), whereas others are considered to be model agnostic (counterfactual explanations, LIME, anchors, etc). SHAP is a widely used model-agnostic, post-hoc, local explanation method, that produces scores indicating the relative importance of various input features that led to a model prediction [8]. SHAP is often the first technique used by practitioners interested in explaining the behaviour of black box models.

A.2

Explainability in Experimentation and Production

One use of explainability in machine learning is to continuously explain the predictions being made by artificial intelligence systems that have been deployed in the real world. This can help debug errors [65], comply with regulations, and engender trust [66]. Often, explanations are used to gauge if protected attributes are significantly impacting model outputs and bring transparency to otherwise opaque decision making processes [67]. When models are being deployed in real world settings and affecting large numbers of users, it can be important to monitor their behaviour and have explainability in production, which requires explanations to be computed in a scalable manner. A different use of explainability is in the lab, where researchers may want to understand model behaviour in order to study underlying phenomena. When using explainability during model development or exploratory data analysis, explanations can be used to understand the data generating process itself and building machine learning models and then using explainability techniques such as SHAP has been proposed as a means of scientific discovery [32–36]. This strategy has been employed in diverse fields such as policy-making [40], physics [42], engineering [38, 43], epidemiology [44, 45], chemistry [39, 46], healthcare [37], ecology [47], and geology [41], research where model explanations are used to derive insights about underlying natural phenomena using a machine learning model, rather than explain the behaviour of a model used in decision making. This can be termed explainability in experimentation, and requires explanations to be robust and consistent. These two use cases for model explanations are different, and present themselves with different desiderata and technical challenges. ROT provides a scalable method that is important in production settings, as well

S1

as a robust method that does not suffer from causal vulnerabilities or the Rashomon effect important in experimentation settings.

A.3

Implicit Assumptions about Causality

Machine learning classification is concerned with learning the conditional probability of obtaining a response y from a given set of inputs x. Borrowing from Judea Pearl’s causal hierarchy [51], this involves “level 1” reasoning - the ability to observe associations. There is broad consensus today that machine learning alone is unable to produce systems with higher “level 2” or “level 3” reasoning abilities involving the ability to answer “interventional” and “counterfactual” questions, respectively. [51] The AI explainability literature so far has been surprisingly forgiving of this deficit typical in all modern machine learning systems. Sensitivity based model explanation methods like SHAP implicitly assume that models produce predictions using level 3 reasoning. If a model only possesses “level 1” observational reasoning, but an explanation method probes it with inputs that do not exist in naturally observed training data, then model predictions in these regions cannot be considered indicative of model behaviours. SHAP’s reliance on perturbing model inputs causes explanations to be overly sensitive to out of distribution model behaviour, which can be arbitrary [48]. The solutions proposed to this problem such as manifold-SHAP [50] or causal SHAP [49] necessarily involve solving the harder problems of density estimation [52] or causal modeling [51], as opposed to learning a simpler classifier decision boundary. Adversarial attacks have been proposed to exploit this vulnerability, and it has been shown that explanations can be manipulated this way, leading to fair-washing. [56] While these two approaches are possible to use in theory, the associated problems with each often make them impossible to implement in practical scenarios. These problems are especially pronounced for uses of model explainability in experimentation, where many models are built and explained iteratively, and the focus is more on the explanations themselves than on deploying the model to real world production settings. In such scenarios, both requiring the user to first build a causal model or estimate the entire density of the data is a non-starter. By contrast, ROT makes no causal assumptions about the model, and does not need to solve the “harder” problems of density estimation either. ROT generates explanations entirely through “level 1” observation.

A.4

Ablation Studies and Sensitivity Analysis

Feature importances can be motivated in at least two distinct ways, involving the effect of: 1. removing or ablating input features on model outputs to determine feature importances through their predictiveness (ablation studies); or 2. perturbing input features and measuring the corresponding perturbations on model outputs to determine feature importances through their impact on the model output (sensitivity analysis). Both these notions have been used widely. Examples of senstivity analysis include applications in chemistry [68] [69], geology [70] [71] and engineering [72]. Ablation studies have roots in medicine and are widely S2

Table S1: Situating ROT among existing local and global XAI. Among those model explanation methods that use feature predictiveness to determine importance, ROT is the only local model explanation method. Amongst all local model explanation methods, ROT is the only method that does not use feature sensitivity to determine feature importance.

Feature Sensitivity Determines Feature Importance (eg. Sensitivity Analysis)

1 2

Feature Predictiveness Determines Feature Importance (eg. Ablation Studies)

1 2

3

3

Global Explanations for Overall Model Behaviour

Local Explanations for Individual Predictions

Partial Dependence Plots Accumulated Local Effects .. .

1 2 3

Permutation Feature Importance MDI (for tree ensembles) .. .

LIME & SHAP Counterfactuals .. .

ROT

used in machine learning [73] today, and sometimes in other fields like physics [74] [75] and biology [76] [77]. In existing machine learning explainability literature, there exist global methods motivated by sensitivity analysis, local methods motivated by sensitivty analysis, global methods motivated by ablation studies, but no local methods motivated by ablation studies. ROT is the first model agnostic post-hoc local explanation method that defines feature importance using the predictiveness of features rather than the sensitivity of the output to perturbations, akin to ablation studies rather than sensitivity analysis. Table S1 provides a non-exhaustive list of model explanation methods, classified along different axes: whether the explanations are global or local, and whether feature importance is determined by sensitivity or predictiveness. ROT is a unique explanation method, as can be seen from the table.

A.5

Problems with Model Explanations in Practice

Returning to our previous examples of trying to debug LLM behaviour by explaining it’s classification outputs, and of trying to monitor a large-scale proprietary e-commerce recommendation system, we can illustrate the common practical problems explaining machine learning predictions. When trying to explain the behaviour of an LLM, it is not possible to arbitrarily modify the input features to the model. For an explanation method like SHAP, it then becomes unclear how to perturb the inputs features. Additionally, explanations can only be used to ascribe importances to input features, and nuanced analyses studying the relationship between the output and external concepts or latent features is difficult. ROT is useful because not only does it not rely on input perturbations, but also because it allows the inclusion of external concepts when generating model explanations. TCAV is a global explanation method that explicitly uses latent concepts to provide model explanations [78], but ROT is able to do this on a local, per-prediction level.

S3

When trying to audit the behaviour of proprietary systems, often model artifacts cannot be accessed to obtain output predictions for new input points . This makes explanation methods like SHAP impossible to use without first creating a model to mimic the original. In real world settings, it is often easier to explain such a mimic model, rather than explaining the actual model [23] [21], and it is hard to assert that the mimic explanations generated also apply to the original model due the predictive multiplicity of classifiers [28]. This “Rashomon effect”, where two models have similar predictions but different internal mechanisms [25], makes model explanations obtained using a mimic model inherently untrustworthy when explaining the original model behaviour [29]. ROT sidesteps this issue entirely and does not need access to a model at all, it generates model explanations instead by looking at its predictions.

B

RoT Implementation Details

Recall our formulation from Equation 2. In all experiments, we treat ℓ as the log loss, and F as the sigmoid function, arriving at an objective similar to logistic regression. We consider three variants of the importance function fθj (xj ). The linear form fθj (xj ) = aj (xj + bj )

(S1)

A more general additive form consisting of a linear form plus a mixture of Gaussians fθj (xj ) = aj (xj + bj ) +

X

−

Si,j e

(xj −µi,j )2 σi,j

(S2)

i<I

While we explored the use of other additive models including polynomial kernels and the double exponentiated form proposed in [79], the mixture of Gaussians combined with a linear term, showed better stability, and robustness to extreme values. And finally, for explaining the behaviour of text-based classifiers, such as ChatGPT for zero-shot classification, we use a linear form shared over all tokens fθj (xj ) = a · xj + b

(S3)

Here xj is now a vector representing the text embedding taken from a model such as BERT, and a and b are common parameters shared over all token locations j ∈ J. Based on this, we provide two canonical implementations of ROT. One uses dropout on the features and a simple linear model to learn classifier logits or regression outputs using either cross entropy or mean squared error loss, and the other uses text embeddings, performing dropout on both embedding features and tokens, to explain zero shot classification tasks. Our experiments were performed on a MacBook with 24 GB memory and an Apple M4 Pro Processor. All the code used for our experiments will be made public. We also plan to make ROT available for the community as an open-source software package. We make use of existing implementations of explainers such as SHAP [8] and LIME [1], other open sourced data and models [10, 13, 17, 20], and evaluation benchmarks S4

and code [56, 80] previously released by the community.

B.1

Derivation

We wish to explain the behavior of a model C(·), over an empiric data distribution X. Given a data-point, x ∈ X , we use xj to refer to the j th component of the feature vector of x. We define our explanations as a set of functions, similarly indexed by j, and refer to fj (xj ) as the importance of xj . Given an arbitrary subset of features J ⊆ J over a data-point x, we wish to be able make predictions about the model response P C(x) using only the sum of importance weights j∈J fj (xj ) In this way, we wish to be able to make a prediction about the model response C(x) using incomplete information about the feature vector of x, where only j ∈ J of the feature values are known. The fj (xj ) individually serve as feature importances, and when summed are used to predict the model output. B.1.1

Formulating Explanations from Incomplete Information

We give a general formulation suitable both for modelling discrete decisions using a logistic regression approximation, and for modelling continuous regression tasks using a linear regression approximation. In practice, these can be replaced with any additive model. The functions fj are parameterized by weights θj and the model is approximated by a function F of their sum and a global bias term G. Recall that we want to find the optimal set of additive functions f such that for any data point x ∈ X and arbitrary subset of features J we can approximate C(x).

 C(x) ≈ F 

 X

fθj (xj ) + G ∀J ∈ P(J)

(S4)

j∈J

As mentioned above, we use the linear and logistic approximations of C(·), in which fθj (xj ) = aj xj + bj . In the case of linear regression F is the identity function, whereas for logistic regression F is the logistic function σ. For regression models, this yields:

C(x) ≈

X

(aj xj + bj ) + G ∀J ∈ P(J)

(S5)

j∈J

and for classification this yields:   X C(x) ≈ σ  (aj xj + bj ) + G ∀J ∈ P(J)

(S6)

j∈J

For regression models, we minimise the mean squared error, further simplifying equation S5. The per feature bias term bj can disregarded if the mean of every feature j is 0, and the global bias term becomes S5

equal to the mean model response C(x). For classification models, we maximize the likelihood by marginalising over a distribution of randomly sampled features. We seek the values of θj that minimize the expected loss L[·, ·] (typically cross-entropy error) over all subsets of features J ⊆ J and data-points x ∈ X.  min

X



p|J| (1 − p)(|J|−|J|) ·

X



L C(x), σ 

x∈X

J⊆J

 X

fθj (xj ) + G

(S7)

j∈J

Like the formulation of SHAP, this objective is exponentially large with respect to the feature space. However, a key difference that allows us solve it effectively is that the number of function calls to C is equal to the number of data points |X|, and completely independent of the feature space J. By commuting this is equivalent to  min

XX





p|J| (1 − p)(|J|−|J|) · L C(x), σ 

x∈X J⊆J

  X

fθj (xj ) + G 

(S8)

j∈J

Upon careful inspection, P this is equivalent  to performing dropout with probability p [14] on the initial approximation model σ j∈J fθj (xj ) + G . This allows us to directly find the weights θj (recall we defined fθj = aj xj + bj ) and G using stochastic optimization by treating this as a regression problem regulated by dropout. Implementation of the method is straightforward, with the only subtlety to be aware of is that the weights and biases should be zero-initialized to avoid problems with vanishing gradients in the logistic regression loss. B.1.2

Uniqueness and Stability

Each component of the objective defined in equation (S8) is convex. As such, if for every estimated variable there exists a strictly convex component containing that variable, the problem as a whole is strictly convex. Therefore, the overall objective has a unique minimum provided the individual functions fθj (·) are well-posed and strictly convex with each function having a single minimum associated with it. B.1.3

From Approximation Model Weights to Interpretable Feature Attributions

The optimization procedure returns a set of functions fθj that then need to be applied to each feature. The response fθj (xj ) can be directly interpreted as a signed measure of the importance of feature j on data-point P x. For logistic regression, the sum j∈J fθj (xj ) can be directly interpreted as probability in logit space of the classifier response C(x); while for linear regression the same sum is an estimate of the model prediction C(x).

C

XAI for Regulation and Auditing

Enforcement of existing AI regulation often falters due to a lack of evidence, stemming from weak or missing requirements for transparency and model access for regulators, third-party auditors, and researchers [81–83]. S6

Gathering sufficient evidence of potentially illegal model behaviour is highly difficult without full API access that would allow regulators to feed the model data to evaluate its responses or oversee model outputs holistically [84–86]. By working without model call access and requiring only observations of prior model outputs or predictions, ROT can bypass significant barriers to regulatory enforceability created by trade secrecy and confidentiality. ROT thus democratizes access to explaining and auditing AI systems. This is crucial because strong provisions enabling access or auditing by third parties (e.g., civil society organizations, vetted researchers) are uncommon in AI regulation, with Article 40 of the EU Digital Services Act being a rare exception [58, 87]. While Article 74(13) grants market surveillance authorities access to the source code of a high-risk AI system under a limited set of circumstances 6 , Article 74(12) grants market surveillance authorities access to the training, validation, and testing datasets to carry out investigations into these systems. Importantly, this is limited to dataset access, and normally does not include full model access or enable regulators to feed new data to a model to check its responses 7 . ROT offers a clear advantage over other methods in this regard because it can compute accurate explanations using only these datasets, while others require full API access or training a mimic model. ROT can also be used to identify anti-competitive practices and prevent self-preferencing. By explaining model predictions and recommendations, ROT can be used to infer “how” a digital service makes certain recommendations or “why” it behaves in a certain way. Competition law, however, does not always require establishing causality. A recent case against Google in the United States has shown, for example, that proving the mere existence of anti-competitive conditions is sufficient to show the law has been breached [88]. In this context, even if specific instances of this behaviour in a production environment (e.g., actual transactions with customers) cannot be shown due to a lack of access to model outputs, ROT can demonstrate that a model displays anti-competitive behaviour. ROT can help demonstrate that model behaviour is likely to contravene the competition law prohibitions, and thus assist in the enforcement of competition law. Unlike measuring the sensitivity of mimic models, ROT makes no causal assumptions and thus its ability to predict behaviour of models directly makes it a stronger tool to demonstrate potentially discriminatory behaviour.

D

Experimental Details

D.1

Explaining Zero-Shot Image Classification from the GPT-4-mini API

We use the GPT-4o-mini API from OpenAI to label a balanced subset of 5000 images from a Kaggle dataset of cats and dogs [3]. To do this we pass each image to the language model via the API, along with the prompt ‘Is this a dog or a cat?’. We then extract features from the images using mobilenetv3[15] as a backbone. The image embeddings extracted this way are used as input features for a ROT classifier, trained using 75% of the 6

One such circumstance being that the testing, auditing and verification procedures and the provider’s documentation have been “proved insufficient” 7 It is worth noting that full API access may be granted under the AI Act to general-purpose AI models (e.g., LLMs) if specific conditions are met and a request made by the European Commission. Specifically, Article 92 (1), (3) – (4) grants the EU AI Office the power to conduct evaluations of the general-purpose model if the EU Commission requests API access (or source code). However, even in such cases ROT has clear advantages because it is one of the only computationally viable methods shown to reliably explain zero-shot classification tasks in LLMs at a meaningful scale.

S7

labelled images. Explanations, comprising of positive or negative importances for the mobilenetv3 features, are computed for the 25% test split, and visualised after per-image-normalisation as overlaid saliency maps. The GPT-4o-mini API has perfect classification accuracy, excluding ambiguous or incorrectly labelled images from the dataset, and the ROT explainer has a training accuracy of 98.5% and test accuracy of 97.6%.

D.2 Evaluating Zero-Shot LLM Classifier Models with XAI: Explaining Judicial Case Predictions from a Fine-Tuned RoBERTa Model From the published PredEx [17] data, we start by removing all annotations that contain hallucinations or text that we could not reliably match in the raw case text. We drop datapoints with missing case text or missing annotations, and datapoints where no text segments were found to be important or where the entire case text was found to be important. The data was annotated by law students who tried to identify text responsible for the judgement. However, the finetuned model making predictions does not always match the ground-truth predictions. Since the explanation annotations are produced from the ground-truth, but our explainers try to explain the model, there is a discrepancy. To resolve this, we drop all cases where the model predictions were incorrect, since we are interested in explaining the model but comparing our outputs with annotations which were produced independent of the model. From the remaining subset, we further drop those cases where highlighted segments were too “short” to be reliable: we enforce the condition that the minimum character length of a highlighted or non-highlighted segment in the text should be at least 75 characters. This threshold is large enough to ensure that common phrases do not get highlighted unnecessarily, and small enough to ensure we still have enough datapoints pass the filter and be included in our experiment. This finally leaves us with a set of 502 cases, from which we compute SHAP and ROT explanations on a random 20% of cases (101 cases total). The ROT explainer we build uses the remaining 401 cases as background, but results using all 502 cases or only the 101 explanation cases as background are qualitatively similar. For the 401 case corpus, ROT is only fit on the final RoBERTa prediction per case, and not on the per-segment annotations within the case text. We report the AUROC of the explanations with respect to the ground truth human annotations (Table 1). Computing this metric involves comparing explanation importances (which can be negative) with human annotations (which are a binary 0 or 1). Since the annotations indicate whether text is important without direction (appeal rejection or acceptance), we post-process the feature importance by aligning them with the case prediction manually. That is, if the case appeal was rejected, we consider negative values to denote higher importance by flipping the sign of the importance for all segments, and if the case appeal was accepted, we consider the importance values as is. This way, positive values always indicate higher importance as defined by the annotators. This post-processing is not specific to ROT and is common to all explainers because it is a peculiarity arising from the PredEx data. For ROT explanations, we use a straightforward implementation where dropout is used both on input features and on input tokens. For SHAP, we use the canonical partition explainer implementation (which defaults to 500 background samples) available for use with language models8 . For LIME we use the text 8 this has been made part of the official SHAP package: https://shap.readthedocs.io/en/latest /text_examples.html , but not documented through any publications

S8

Table S2: Area under the Precision-Recall curve: Evaluating explanations by comparing them with ground annotations. We compare ground truth labels with ROT importances to measure the performance in terms of identifying important text segments. While ROT is better than random, PR-AUC is sensitive to annotation density, and flipping the labels affects the metric significantly. For this reason we choose to use the AUROC metric instead.

Average PR-AUC (flipped) Average PR-AUC

R OT

Random

0.58 0.95

0.26 0.92

explainer with its default bag-of-words tokenizer, with its default 5000 background samples. In addition to abridged example in Figure 4, program runtimes are documented in Figure 9. The ROT and SHAP explanations for the unabridged case can be seen in Figures S2 and S3. Additionally, LIME and Integrated Gradients explanations for this particular case can be seen in Figures S5, S6, and S4 respectively. Results from other cases can also be analysed via https://kairawal.github.io/Rule-of-Thumb -Explaining-Artificial-Intelligence-Systems-using-Partial-Information /JudicialCaseOutcomePrediction/Code/results.html .

D.3

Explaining Movie Review Classifications from an LLM API

The ERASER[19] benchmark provides multiple text datasets with annotations indicating importance of constituent tokens. Most of the tasks however consist of retrieval (from an LLM context window), which can be evaluated using these annotations. The movie review dataset[18], however, presents an exception. The task here is simple zero-shot classification and does not involve any explicit retrieval. We use the GPT-4.1-nano API to obtain predictions for movie reviews (positive or negative), which we then seek to explain using ROT. Our dataset consists of of 1600 movie reviews from the ERASER benchmark dataset [19]. To explain these LLM review sentiment classifications we repeat our process from explaining fine-tuned RoBERTa model predictions for judicial cases exactly, producing ROT explanations for each text segment from the movie reviews and measuring how well they align with human annotations. The notable difference between the two setups is that when explaining judicial cases, we reused the embeddings from the finetuned RoBERTa model itself, whereas now we use ModernBERT [89]. Choice of Evaluation Metric

For both LLM experiments discussed so far, we evaluate explanation quality

using the area under the receiver-operating-characteristic (a plot of the true positive rate against the false Table S3: Area under the precision-recall curve when comparing ROT explanations with ground-truth annotations. The underlying datapoints for this table are identical to Table 1 where we initially presented results from our experiment explaining the juducial case prediction AI system.

Average PR-AUC

R OT

SHAP

Integrated Gradients

LIME (5000 samples)

Random

0.77

0.76

0.69

0.68

0.53

S9

positive rate). We choose this metric over the area under a precision-recall curve because the precision-recall curve is sensitive to annotation density [90]. That is, the precision recall metric is highly sensitive to what proportion of tokens are deemed important for any given text. Since the annotations are booleans, this is dependent entirely on annotator idiosyncracies – we simply do not have ground-truth importance scores per segment). To illustrate the impact of this, consider the areas under the precision-recall curve reported in the top row of Table S2. If we simply flip all the labels in the ground truth annotation and simultaneously flip the signs of the ROT importances (the equivalent of finding / deleting unimportant segments instead of finding / selecting important segments from a given review), the metrics change as is visible in the bottom row of the table.

D.4

Explaining Zero Shot Resume Filtering using Proprietary LLM APIs

We replicate a study that found GPT-3 to be largely unbiased in resume filtering tasks [10]. Starting with a synthetic corpus of resumes consisting of information-technology workers, accountants, aviation workers, construction workers, chefs, advocates, teachers, and salespeople; we use GPT-4.1-nano to select the IT workers from this corpus. In such a scenario where zero shot classifications are obtained from an LLM API which cannot be queried at will, ROT is still able to explain predictions. Just like explaining case outcome prediction and movies, we are able to produce token level importance scores using ROT. This time, we use BERT[91] embeddings instead of ModernBERT[89], showing how our method is embedding agnostic and can be used with various text embeddings. We additionally augment explanations with non-input features so predictions are explained not just in terms of the resume, but also in terms of the demographic attributes of the candidates. The original study used race, gender, and political orientation, and we explicitly add this information to our explanation text (even though the predictions were made using only the resume summary). This is an additional ability unique to ROT, and is particularly useful for black-box algorithmic auditing. It allows us to generate feature importance explanations using a superset of the model inputs – we include race, gender, and political orientation. Our results match the original study, and find that these features have negligible impact on model predictions. In addition to the wordcloud in Figure 5, we visualise the importance ascribed to a particular token across resume summaries. A cursory examination of these shows that IT related words are positively correlated with IT workers, whereas words belonging to other professions are negatively correlated. Some of these can be seen visualised in Figure S7.

D.5

Auditing Proprietary Black-Box AI Systems without Mimic Models

Journalists at The Markup audited Amazon’s product recommendation system. Since the model used is proprietary, it cannot be queried at will, which renders all previous explainers inapplicable. To circumvent this, product recommendation data was scraped from the Amazon website, consisting of product pairs – the first and second product recommended for any particular search. A model would be built as a “proxy” to mimic Amazon product rankings, formulated as a binary classifier selecting one of the products from the scraped pair. Feature inputs to this model were also scraped, consisting of attributes like product stars (the

S10

difference in number of stars between the pair of models), and whether the picked model brand was Amazon, was sold by Amazon, shipped by Amazon, etc. The Markup analysed this data by building a Random Forest model and then computing SHAP explanations of this model9 . Following this methodology, they discovered Amazon to systematically promote products from its own brand over others, as SHAP considered “brand is Amazon” to be the most important feature, and “product reviews” to be second. The model trained and explained was not arbitrary, but chosen after a typical hyperparameter search. When we replicated this audit, we found several models that were (marginally) more accurate, which we list in Table 2. We found the scikit-learn default Random Forest had better performance than the Random Forest with hyperparameters chosen from The Markup’s hyperparameter search. We also found the default Logistic Regression in scikit learn to have better performance, as did both regularised Logistic Regressions (C=0.01). All our models had similar accuracies and formed a Rashomon set. Their internal mechanisms were distinct, as can be verified from the global explanations summarised in Table S6. This variation is also evident from the SHAP explanations for these models, which led to different conclusions about input feature importances depending on the mimic used. All models found “brand is amazon” to be the most important input feature, but following this, subsequent features had different importances depending on the model selected. For brevity, we present SHAP explanations from just Random Forest models in Figure 6, but include results from all models in Figure S8 and Figure S10. ROT explanations, by contrast, do not require mimic models. Results obtained from ROT are therefore stronger in an auditing context, because they cannot be attributed to peculiarities in any mimic model, which can otherwise be used to absolve the original inaccessible, proprietary model.

D.6

Literature Review to Determine the Prevalence of XAI for Scientific Discovery

We measured how often explainability is used as a means of scientific discovery to understand the natural phenomena that underlie models by performing a literature review. To do this, we analysed papers fetched from the Scopus database related to various scientific fields that used some form of interpretable machine learning. We queried the Scopus API using the TITLE-ABS-KEY field with a search string constructed as follows: ( "explainable artificial intelligence" OR "SHAP" OR "interpretable machine learning" ) AND ( "social science" OR "physics" OR "chemistry" OR "biology" OR 9

open-source code from The Markup’s reporting: https://github.com/the-markup/investigation-amazon -brands , with both the scraped data and hyperparameter tuning steps

S11

"geophysics" OR "astrophysics" OR "biochemistry" OR "medicine" OR "neuroscience" OR "scientific discovery" ) This yielded 1151 articles, which we filtered using LLMs to analyse the abstracts for mentions of XAI for forming new scientific hypotheses. Using the LLM filter, we reduced our corpus to 111 articles, which we checked manually to verify the use of XAI in each paper. We found that 56 of these papers used XAI, and 39 of the 56 did so using the SHAP explainer. These article lists can be found at: • https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial -Intelligence-Systems-using-Partial-Information/LitReview/a .Corpus_1151.csv • https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial -Intelligence-Systems-using-Partial-Information/LitReview/b .LLM_filtered_111.csv • https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial -Intelligence-Systems-using-Partial-Information/LitReview/c .Manual_XAI_56.csv • https://github.com/KaiRawal/Rule-of-Thumb-Explaining-Artificial -Intelligence-Systems-using-Partial-Information/LitReview/d .Manual_SHAP_39.csv respectively. Our reported statistic of 4.9% papers using XAI for scientific discovery is thus a lower bound – there might be papers using XAI that were missed by our LLM filter and never included in set of 111 papers we manually inspected. If the LLM false negative rate is assumed to be the same as the false positive (55/111) rate, our final result could be as high as 49%. Our objective however is not to perform a thorough literature review or identify what proportion of papers use SHAP to generate new scientific discoveries. Instead, we merely wish to demonstrate that a significant amount of modern scientific discovery relies on sensitivity based explainers to generate hypotheses that seek to causally explain real-world phenomena.

D.7

Identifying Uninformative Features as Unimportant for Scientific Discovery

When using XAI for scientific discovery, we want explanations not to consider uninformative input features as important. We deliberately transform[55] the “age” feature of the “Pima Indians”[13] diabetes prediction dataset to be uncorrelated with diabetes, the target feature, and then compute explanations with various explainers to test which ones still find “age” to be important. S12

To make a feature statistically uncorrelated with the target, we adopt the FairPCA[55] method to transform our dataset. For our dataset with 8 input features, this transformation works by learning a low dimensional space of 7 features, such that age information is deliberately lost. By transforming our data into this space, any model trained cannot utilise age information. Finally, we learn a Logistic Regression classifier on the data, and use standard SHAP, LIME, and ROT explainers to obtain feature importances. As seen in Figure 7, we find sensitivity based explainers unable to identify the unimportance of age, but ROT does this perfectly.

D.8

Invulnerability to Adversarial Attacks that Mask Important Features

We consider a scenario where an adversary tries to mislead explainers into selecting spurious “foil” features as important, instead of the “sensitive” features the model actually uses to make decisions. We use the same datasets as those used in the paper proposing the adversarial attack [56], and follow its setup exactly: for the COMPAS [92] and Communities & Crime [93] datasets, the “sensitive” feature is “race” whereas for the German Credit [94] dataset it is “gender”. All models are binary classifiers formulated as simple if statements depending on the sensitive feature, predicting recidivism risk for the former two datasets, and loan repayment probabilities for the latter. The “foil” attributes used are synthetically added features for COMPAS and Communities and Crime, whereas for the German Credit [94] dataset, the foil used is not synthetic but a new feature that is a function of other input features. Further, instead of 3 experiments (one for each dataset), for COMPAS and Communities and Crime we conduct experiments twice – once with a single synthetic foil, and once with 2 synthetic foils. The adversary modifies models to mislead explainers. These modifications work can be tailored to the particular perturbations used by a sensitivity based explainer, and we consider models designed to mislead both SHAP (mS ) and LIME (mL ), comparing both with ROT. Like in the original paper, this setup leads to 5 experiments with each explainer: 1 German Credit, 2 COMPAS, and 2 Communities and Crime. For each datapoint in each experiment, we consider the absolute importance of the sensitive feature, average absolute importance of the foil features, and average absolute importance of all other features, forming a sample from a ternary distribution. We visualise this distribution of importances in Figure 8 using publicly available software [95, 96], combined from across each of the 5 experiments so each experiment has the same weight. We also present exact numerical results for detailed perusal in Table S7.

D.9

Benchmark Evaluations of ROT Explanations with OpenXAI

We analysed ROT in traditional tabular modalities using an existing XAI benchmark, since it can be used as a drop-in replacement for other feature-importance explainers. Evaluating explanation quality is a nuanced problem, and we find standard benchmarks unsuitable to measure quality, but are able to observe that ROT produces similar explanations as SHAP in a faster and more computationally efficient manner. Sensitivity analysis underlies XAI evaluation benchmarks just like it underlies explainers. The OpenXAI benchmark [80] consists of explanation evaluation metrics that either perform sensitivity analysis, or comparisons with “ground-truth” explanations.

S13

OpenXAI considers Logistic Regression coefficients as the “ground-truth” explanation for all datapoints. Regardless of the specific feature values for a given datapoint, such ground-truth comparison metrics consider a single explanation as ideal across all input datapoints, promoting a single unchanging explanation regardless of specific input values. Using Logistic Regression coefficients OpenXAI deliberately promotes gradient methods or LIME, which are Taylor-series like approximations of AI system behaviour and can recover the Logistic Regression coefficients exactly for any input datapoint. While these explainers score better than SHAP in the published OpenXAI benchmark scores, they are undesirable in practice because they cannot explain individual predictions and always produce identical feature importance outputs. Due to these aforementioned issues, instead of reporting the official metrics from OpenXAI, we adopt the benchmark to verify that ROT behaviour mirrors SHAP, and every input datapoint does not yield the same explanation output. We compute explanations of the pretrained OpenXAI models by running publicly available code from the benchmark[80] as provided10 . We present results from the COMPAS dataset because it has the fewest features from among OpenXAI datasets, allowing us to present results succinctly. To select an initial point and a final point for plot (a), we find pairs of datapoints with different model predictions such that l0 distance between them is 2 or less, and l1 distance between them is as large as possible. For plot (b), we consider all pairs where l0 distance is exactly 1 while maximising l1 distances as before, and ensuring the feature changed is always “length of stay”. While we present select results, we are able to repeat this process for all provided pretrained models and datasets: COMPAS, Home Equity Line of Credit (HELOC), Adult Income, and German Credit. These provide the runtime statistics seen in Tables S4 and S5. Figure S1 shows how changing input feature values do not always lead to changes in the corresponding feature importances. Only the SHAP and ROT explainers show desirable behaviour, where the local explanation is truly a function of the datapoint being explained. LIME and gradients show the same feature importance regardless of the feature values, undermining the usefulness of OpenXAI metrics which consider these explanations to be the best. All gradient based methods in OpenXAI generated similar explanations, and here we only show Integrated Gradients for clarity.

E

Extended Results

E.1

Benchmark Evaluations and Computational Efficiency

We found fundamental problems with the OpenXAI benchmark that prevented its adoption as-is. In Figure S1 we show how ROT and SHAP explanations for example datapoints in the benchmark are the only ones that actually vary when the prediction being explained varies. The “ground-truth” metrics in the OpenXAI leaderboard are known not to score SHAP explanations highly. However, we were able to use the benchmark to measure runtimes across explainers. Tables S4 and S5 document ROT runtimes against SHAP and LIME across the OpenXAI benchmark and all our experiments with tabular data.

10

Adopted from https://github.com/AI4LIFE-GROUP/OpenXAI/tree/main

S14

initial RoT SHAP Int. Grad. LIME

Feature Importance

3 2 1

final RoT SHAP Int. Grad. LIME

0 1 2 3 4

(Changed Features) age initial: 64 final: 30

(Unchanged Features)

length of stay priors count 13 1 13 474

sex 0 0

Feature Value

two yr recid. 1 1

race 1 1

charge degree 1 1

(a) Feature importances for an initial datapoint and for a final datapoint with a changed prediction. We expect input feature changes to change the corresponding importance, but this happens only for ROT and SHAP. LIME and Gradients are constant despite changes in “age” and “length of stay”.

0.0

Change in Feature Importance for "length of stay"

4

0.5 1.0 1.5

RoT SHAP Int. Grad. LIME

2.0 250

300

350

400

450

500

550

600

Change in "length of stay" Feature Values

(b) Changes in “length of stay” feature importance for changes in input feature values. Like in (a), ROT and SHAP change but LIME and Gradients are constant

Figure S1: Benchmark comparisons of ROT with other feature importance explainers. Analysing explainers using a pretrained binary classifier from the OpenXAI benchmark (COMPAS data). For any initial datapoint, if changing a feature value changes the class predicted by the AI system, that feature should be important. We show all feature importances for one such datapoint (a), and the changes in one feature’s importance across all such datapoints (b). Table S4: Comparison of SHAP and ROT run-times across experiments. For each experiment we denote the dataset name and model type (Logistic Regression or Artificial Neural Network). The per-explanation run-time of ROT is negligible, making it much faster than SHAP despite slower initialization times. This means that ROT provides a greater speedup for larger datasets with more explanations to compute. Data & Model Details (+ Num. Explanations)

ROT Time init (s)

SHAP Time init (s)

ROT Time total (s)

SHAP Time total (s)

Speedup Factor

AI Auditing

Markup RF

283

0.6358

0.0690

0.6359

295.41

464.54

Uninformative Features

Pima Diabetes FairPCA LR

154

1.0191

0.0007

1.0191

9.63

9.45

German Credit CC (1 foil) CC (2 foils) COMPAS (1 foil) COMPAS (2 foils) German Credit LR German credit ANN COMPAS LR COMPAS ANN HELOC LR HELOC ANN Adult Income LR Adult Income ANN

100 200 200 618 618 800 800 4937 4937 7896 7896 36177 36177

0.4273 0.7037 0.6782 0.6126 0.6091 0.2621 0.2596 0.3344 0.3243 0.6619 0.6473 2.1747 2.0596

0.0017 0.0017 0.0022 0.0021 0.0057 0.0013 0.0009 0.0007 0.0011 0.0019 0.0010 0.0015 0.0006

0.4274 0.7038 0.6783 0.6127 0.6092 0.2768 0.2741 0.4127 0.4049 0.7843 0.7733 2.7339 2.6237

32.12 138.99 136.19 69.86 68.43 233.55 169.50 189.93 217.48 522.34 413.89 1437.98 1668.77

75.17 197.47 200.78 114.03 112.32 843.88 618.49 460.23 537.14 666.00 535.20 525.98 636.04

-

6786.2

0.7606

0.0061

0.8647

373.60

432.08

Adversarial Attack

OpenXAI Benchmark

Final Average

S15

Table S5: Comparison of LIME and ROT run-times across experiments. For each experiment we denote the dataset name and model type (Logistic Regression or Artificial Neural Network). The OpenXAI LIME implementation differs from the canonical by reducing background samples used per explanation from 5000 to 1000. In order to make up for this inconsistency we report extrapolated OpenXAI LIME runtimes here by multiplying with 5. Like with SHAP, ROT provides a greater speedup for larger datasets with more explanations to compute. Data & Model Details (+ Num. Explanations)

ROT Time init (s)

LIME Time init (s)

ROT Time total (s)

LIME Time total (s)

Speedup Factor

AI Auditing

Markup

283

0.6358

0.0025

0.6359

21.95

34.52

Uninformative Features

Pima Diabetes FairPCA LR

154

1.0191

0.0020

1.0191

1.40

1.37

German Credit CC (1 foil) CC (2 foils) COMPAS (1 foil) COMPAS (2 foils) German Credit LR German Credit ANN COMPAS LR COMPAS ANN HELOC LR HELOC ANN Adult Income LR Adult Income ANN

100 200 200 618 618 800 800 4937 4937 7896 7896 36177 36177

0.6934 1.0922 0.9712 0.9836 0.6865 0.2621 0.2596 0.3344 0.3243 0.6619 0.6473 2.1747 2.0596

0.0012 0.0008 0.0008 0.0025 0.0025 0.0003 0.0004 0.0004 0.0017 0.0006 0.0063 0.0012 0.0015

0.6935 1.0923 0.9713 0.9838 0.6866 0.2768 0.2741 0.4127 0.4049 0.7843 0.7733 2.7339 2.6237

2.39 86.02 84.87 19.95 20.87 1071.57 1073.84 246.86 420.53 2309.85 3991.78 2057.12 7396.13

3.45 78.75 87.37 20.28 30.40 3871.97 3918.41 598.18 1038.64 2945.13 5161.80 752.44 2818.98

-

6786.2

0.8537

0.0016

0.9577

1253.67

1308.99

Adversarial Attack

OpenXAI Benchmark

Final Average

E.2

Evaluating Zero-Shot LLM Classifier Models with XAI

Unabridged examples of the case prediction explanation from Figure 4 using ROT (Figure S2), SHAP (Figure S3), Integrated Gradients (Figure S4), LIME reduced to 500 background samples to reduce program runtime (Figure S5), and LIME with 5000 default background samples (Figure S6).

S16

Kapur, J. 1. This appeal by special leave is directed against the judgment and order of the High Court of Judicature at Madras. The appellant is the assessee and the respondent is the Commissioner of Income-tax and the question raised is as to the applicability of S. 10(2)(xv) of the Indian Income-tax Act to a gratuity paid by the appellant to one of its officers on his retirement from service. 2. The appeal relates to the assessment year 1950-51. M/s. Gordon Woodroffee and Co. (Madras) Ltd., was incorporated as a private limited company in 1922 and became the Managing Agent of a public limited company M/s. Gordon Woodroffee Leather Manufacturing Company Ltd., which is the assessee. One J. H. Philips was employed in the Managing Agent Company from 1922 to 1935 and from 1935 he became an employee of the appellant company and became its Director from 1940. On March 22, 1949, he wrote a letter to the appellant company expressing his intention to resign from the Board of the Company as from April 4, 1949 upon his retirement from the employment of the company and requested that his resignation be accepted. On March 24, 1949, the Board of Directors of the appellant Company passed a resolution that his resignation be accepted and in appreciation of his long and valuable services to the Company he be paid a gratuity of Rs. 50,000/- out of which the appellant Company was to pay Rs. 40,000/-. and the Managing Agent Company the balance of Rs. 10,000/-. On April 4, 1949 this resolution of the Board of Directors was confirmed, on the same date a resolution to the same effect was passed at an Extraordinary General Meeting of the Company and before the end of its accounting year i.e., October 31, 1949, this amount of Rs. 40,000/- was paid to Mr. J. H. Philips. 3. This amount was claimed as a deduction under S. 10(2)(xv) of the Income-tax Act which reads :-Section 10 (2) : ""Such profits or gains shall be computed after making the following allowances, namely : ...................................................... ............. ................ ...... ......... ....... (xv) any expenditure (not being an allowance of the nature described in any of the clauses (i) to (xiv) inclusive, and not being in the nature of capital expenditure or personal expenses of the assessee) laid out or expended wholly and exclusively for the purpose of such business, profession or vocation."" The amount was disallowed by the Income-tax Officer as well as by the Appellate Assistant Commissioner on the ground that the appellant Company had no pension scheme; the payment was voluntary and that the entry in the assessees books clearly indicated it to be a capital payment. Against this order the appellant Company took an appeal to the Income-tax Appellate Tribunal which upheld the order of the Appellate Assistant Commissioner. It held that according to the resolution the gratuity was paid ""for long and valuable services to the Company""; that there was nothing to indicate that Mr. J. H. Philips had accepted a lower salary in expectation of getting a gratuity at, the end of is service; at there was no such practice in the appellant Company and that during the course of his service he was being remunerated at a graduated scale of salary and a commission of it 2 1/2 on the profits; that there was no ""expectancy"" that at the end of the service there would be a recompense for faithful and efficient service; that he had been suitably rewarded by being given a commission on the profits ""in order to whip up his enthusiasm"". It was also mentioned that in the books of the appellant Company the amount had not been debited in the profit and loss account but was debited to the appropriation account thereby indicating that it was an extra payment or a payment made in the nature of a capital expense. Taking all these circumstances into consideration the Tribunal came to the conclusion that it was difficult to hold that the expenditure was not in the nature of a capital expenditure or that it was expended wholly exclusively for the purpose of the assessees business. At the instance of the appellant Company the case was stated to the High Court under S. 66(1) of the Income-tax Act and the following question was referred :-""Whether the sum of Rs. 40,000/- paid to Mr. J. H. Philips on his retirement from the service of the Company was, not an admissible deduction under S. 10 (2)(xv) of the Income-tax Act, 1922."" 4. The High Court answered the question against the appellant Company. It held that in order that S. 10(2) (xv) applicable it had to be proved that the amount was laid out or expended wholly and exclusively for purposes of the companys business. In his case the amount was paid on retirement and for valuable services rendered by Mr. J. H. Philips; there was no evidence that he expected to receive this amount or the Company contemplated its payment at any time before; the payment was voluntary and there was no evidence to show that it was in the future interest of the business of the Company that the expenditure was incurred. The High Court observed : ""In the case of a payment of a gratuity to a retiring employee recognition of his past services, with nothing more cannot, in our opinion satisfy the requirements of S. 10(2) (xv), even if those requirements are judged from the view point of commercial expediency, as it always should be when a claim arises under S, 10(2)(xv). Was the expenditure incurred in the future interest of the business of the assessee? Was there any connection between the purpose of the payment and the further conduct of the business of the assessee ? These are the tests to be satisfies before it could be said that in paying the gratuity money was laid out or expended wholly and exclusively for the purpose of the business of the company. These tests the assessee did not satisfy in this case,"" Against this judgment and order the appellant Company has brought this appeal by special leave. 5. It was argued on behalf of the appellant that the amount had been paid as a matter of commercial expediency and in the interest of the Company as an inducement to other employees that if they rendered service in a similar manner with efficiency and honesty they would be similarly rewarded. Decisive test, it was submitted was whether such payments of gratuity were likely in future also and was the payment made as an incentive to the employees to give their best to the employer and if it was so then the payment was a matter of commercial prudence. It was also submitted that the Company had acted not with any oblique motive and its good faith was not in doubt and in support of the contention several cases were relied upon, 6. In our opinion on the findings as given the payment in dispute does not fall within the provisions of S. 10(2)(xv). The amount was paid not in pursuance of any scheme of payment of gratuities nor was it an amount which the recipient expected to be paid for long and faithful service but it was a voluntary payment not with the object of facilitating the carrying on of the business of the appellant Company or as a matter of commercial expediency but in recognition of long and faithful service of Mr. J. H. Philips. There was no practice in the appellant Company to pay such amounts and it did not affect the quantum of salary of the recipient . The two cases strongly relied upon by the appellant Company were J. P. Hancook v. General Reversionary and Investment Company Ltd. (1919) 7 Tax Cas 358 and J. W. Smith v. Incorporated Council of Law Reporting for England and Wales (1914) 6 Tax Cas 477. In the former case the assssee Company sought to charge as a trade expense a lump sum which it had pad for the purchase, for the benefit of a former actuary, of an annuity equal in amount to the pension which the Company had resolved to pay him. This was held to bean expense admissible in computing the Companys profits assessable to income-tax. But in that else it was the practice of the assessee company to grant pensions to its servants after a considerable period of service and this practice was known to the employees and affected the rate of salary paid by the Company in that the employees were willing to serve the Company at lower rates than they otherwise would have by reason of the expectation of the pension at the end of their service. In the latter case there was a practice of granting gratuities and that was the ground for holding the amount to be a proper deduction. 7. In our opinion the proper test to apply in this case is, was the payment made as a matter of practice which affected the quantum of salary or was there an expectation by the employee of getting a gratuity or was the sum of money expended on the ground of commercial expediency and in order indirectly to facilitate the carrying on of the business. But this has not been shown and therefore the amounts claimed is not a deductible item under S. 10(2)(xv).

Figure S2: Unabridged ROT explanation for the case from Figure 4, where the appeal was rejected. The “ground-truth” importance annotation is indicated using a bold font, and the highlights are produced using ROT. ROT aligns well with the annotation. S17

Kapur, J. 1. This appeal by special leave is directed against the judgment and order of the High Court of Judicature at Madras. The appellant is the assessee and the respondent is the Commissioner of Income-tax and the question raised is as to the applicability of S. 10(2)(xv) of the Indian Income-tax Act to a gratuity paid by the appellant to one of its officers on his retirement from service. 2. The appeal relates to the assessment year 1950-51. M/s. Gordon Woodroffee and Co. (Madras) Ltd., was incorporated as a private limited company in 1922 and became the Managing Agent of a public limited company M/s. Gordon Woodroffee Leather Manufacturing Company Ltd., which is the assessee. One J. H. Philips was employed in the Managing Agent Company from 1922 to 1935 and from 1935 he became an employee of the appellant company and became its Director from 1940. On March 22, 1949, he wrote a letter to the appellant company expressing his intention to resign from the Board of the Company as from April 4, 1949 upon his retirement from the employment of the company and requested that his resignation be accepted. On March 24, 1949, the Board of Directors of the appellant Company passed a resolution that his resignation be accepted and in appreciation of his long and valuable services to the Company he be paid a gratuity of Rs. 50,000/- out of which the appellant Company was to pay Rs. 40,000/-. and the Managing Agent Company the balance of Rs. 10,000/-. On April 4, 1949 this resolution of the Board of Directors was confirmed, on the same date a resolution to the same effect was passed at an Extraordinary General Meeting of the Company and before the end of its accounting year i.e., October 31, 1949, this amount of Rs. 40,000/- was paid to Mr. J. H. Philips. 3. This amount was claimed as a deduction under S. 10(2)(xv) of the Income-tax Act which reads :-Section 10 (2) : ""Such profits or gains shall be computed after making the following allowances, namely : ...................................................... ............. ................ ...... ......... ....... (xv) any expenditure (not being an allowance of the nature described in any of the clauses (i) to (xiv) inclusive, and not being in the nature of capital expenditure or personal expenses of the assessee) laid out or expended wholly and exclusively for the purpose of such business, profession or vocation."" The amount was disallowed by the Income-tax Officer as well as by the Appellate Assistant Commissioner on the ground that the appellant Company had no pension scheme; the payment was voluntary and that the entry in the assessees books clearly indicated it to be a capital payment. Against this order the appellant Company took an appeal to the Income-tax Appellate Tribunal which upheld the order of the Appellate Assistant Commissioner. It held that according to the resolution the gratuity was paid ""for long and valuable services to the Company""; that there was nothing to indicate that Mr. J. H. Philips had accepted a lower salary in expectation of getting a gratuity at, the end of is service; at there was no such practice in the appellant Company and that during the course of his service he was being remunerated at a graduated scale of salary and a commission of it 2 1/2 on the profits; that there was no ""expectancy"" that at the end of the service there would be a recompense for faithful and efficient service; that he had been suitably rewarded by being given a commission on the profits ""in order to whip up his enthusiasm"". It was also mentioned that in the books of the appellant Company the amount had not been debited in the profit and loss account but was debited to the appropriation account thereby indicating that it was an extra payment or a payment made in the nature of a capital expense. Taking all these circumstances into consideration the Tribunal came to the conclusion that it was difficult to hold that the expenditure was not in the nature of a capital expenditure or that it was expended wholly exclusively for the purpose of the assessees business. At the instance of the appellant Company the case was stated to the High Court under S. 66(1) of the Income-tax Act and the following question was referred :-""Whether the sum of Rs. 40,000/- paid to Mr. J. H. Philips on his retirement from the service of the Company was, not an admissible deduction under S. 10 (2)(xv) of the Income-tax Act, 1922."" 4. The High Court answered the question against the appellant Company. It held that in order that S. 10(2) (xv) applicable it had to be proved that the amount was laid out or expended wholly and exclusively for purposes of the companys business. In his case the amount was paid on retirement and for valuable services rendered by Mr. J. H. Philips; there was no evidence that he expected to receive this amount or the Company contemplated its payment at any time before; the payment was voluntary and there was no evidence to show that it was in the future interest of the business of the Company that the expenditure was incurred. The High Court observed : ""In the case of a payment of a gratuity to a retiring employee recognition of his past services, with nothing more cannot, in our opinion satisfy the requirements of S. 10(2) (xv), even if those requirements are judged from the view point of commercial expediency, as it always should be when a claim arises under S, 10(2)(xv). Was the expenditure incurred in the future interest of the business of the assessee? Was there any connection between the purpose of the payment and the further conduct of the business of the assessee ? These are the tests to be satisfies before it could be said that in paying the gratuity money was laid out or expended wholly and exclusively for the purpose of the business of the company. These tests the assessee did not satisfy in this case,"" Against this judgment and order the appellant Company has brought this appeal by special leave. 5. It was argued on behalf of the appellant that the amount had been paid as a matter of commercial expediency and in the interest of the Company as an inducement to other employees that if they rendered service in a similar manner with efficiency and honesty they would be similarly rewarded. Decisive test, it was submitted was whether such payments of gratuity were likely in future also and was the payment made as an incentive to the employees to give their best to the employer and if it was so then the payment was a matter of commercial prudence. It was also submitted that the Company had acted not with any oblique motive and its good faith was not in doubt and in support of the contention several cases were relied upon, 6. In our opinion on the findings as given the payment in dispute does not fall within the provisions of S. 10(2)(xv). The amount was paid not in pursuance of any scheme of payment of gratuities nor was it an amount which the recipient expected to be paid for long and faithful service but it was a voluntary payment not with the object of facilitating the carrying on of the business of the appellant Company or as a matter of commercial expediency but in recognition of long and faithful service of Mr. J. H. Philips. There was no practice in the appellant Company to pay such amounts and it did not affect the quantum of salary of the recipient . The two cases strongly relied upon by the appellant Company were J. P. Hancook v. General Reversionary and Investment Company Ltd. (1919) 7 Tax Cas 358 and J. W. Smith v. Incorporated Council of Law Reporting for England and Wales (1914) 6 Tax Cas 477. In the former case the assssee Company sought to charge as a trade expense a lump sum which it had pad for the purchase, for the benefit of a former actuary, of an annuity equal in amount to the pension which the Company had resolved to pay him. This was held to bean expense admissible in computing the Companys profits assessable to income-tax. But in that else it was the practice of the assessee company to grant pensions to its servants after a considerable period of service and this practice was known to the employees and affected the rate of salary paid by the Company in that the employees were willing to serve the Company at lower rates than they otherwise would have by reason of the expectation of the pension at the end of their service. In the latter case there was a practice of granting gratuities and that was the ground for holding the amount to be a proper deduction. 7. In our opinion the proper test to apply in this case is, was the payment made as a matter of practice which affected the quantum of salary or was there an expectation by the employee of getting a gratuity or was the sum of money expended on the ground of commercial expediency and in order indirectly to facilitate the carrying on of the business. But this has not been shown and therefore the amounts claimed is not a deductible item under S. 10(2)(xv).

Figure S3: Unabridged SHAP explanation for the case from Figure 4, where the appeal was rejected. The “ground-truth” importance annotation is indicated using a bold font, and the highlights are produced using SHAP. SHAP does not align with human annotation as well as ROT. S18

Kapur, J. 1. This appeal by special leave is directed against the judgment and order of the High Court of Judicature at Madras. The appellant is the assessee and the respondent is the Commissioner of Income-tax and the question raised is as to the applicability of S. 10(2)(xv) of the Indian Income-tax Act to a gratuity paid by the appellant to one of its officers on his retirement from service. 2. The appeal relates to the assessment year 1950-51. M/s. Gordon Woodroffee and Co. (Madras) Ltd., was incorporated as a private limited company in 1922 and became the Managing Agent of a public limited company M/s. Gordon Woodroffee Leather Manufacturing Company Ltd., which is the assessee. One J. H. Philips was employed in the Managing Agent Company from 1922 to 1935 and from 1935 he became an employee of the appellant company and became its Director from 1940. On March 22, 1949, he wrote a letter to the appellant company expressing his intention to resign from the Board of the Company as from April 4, 1949 upon his retirement from the employment of the company and requested that his resignation be accepted. On March 24, 1949, the Board of Directors of the appellant Company passed a resolution that his resignation be accepted and in appreciation of his long and valuable services to the Company he be paid a gratuity of Rs. 50,000/- out of which the appellant Company was to pay Rs. 40,000/-. and the Managing Agent Company the balance of Rs. 10,000/-. On April 4, 1949 this resolution of the Board of Directors was confirmed, on the same date a resolution to the same effect was passed at an Extraordinary General Meeting of the Company and before the end of its accounting year i.e., October 31, 1949, this amount of Rs. 40,000/- was paid to Mr. J. H. Philips. 3. This amount was claimed as a deduction under S. 10(2)(xv) of the Income-tax Act which reads :-Section 10 (2) : ""Such profits or gains shall be computed after making the following allowances, namely : ...................................................... ............. ................ ...... ......... ....... (xv) any expenditure (not being an allowance of the nature described in any of the clauses (i) to (xiv) inclusive, and not being in the nature of capital expenditure or personal expenses of the assessee) laid out or expended wholly and exclusively for the purpose of such business, profession or vocation."" The amount was disallowed by the Income-tax Officer as well as by the Appellate Assistant Commissioner on the ground that the appellant Company had no pension scheme; the payment was voluntary and that the entry in the assessees books clearly indicated it to be a capital payment. Against this order the appellant Company took an appeal to the Income-tax Appellate Tribunal which upheld the order of the Appellate Assistant Commissioner. It held that according to the resolution the gratuity was paid ""for long and valuable services to the Company""; that there was nothing to indicate that Mr. J. H. Philips had accepted a lower salary in expectation of getting a gratuity at, the end of is service; at there was no such practice in the appellant Company and that during the course of his service he was being remunerated at a graduated scale of salary and a commission of it 2 1/2 on the profits; that there was no ""expectancy"" that at the end of the service there would be a recompense for faithful and efficient service; that he had been suitably rewarded by being given a commission on the profits ""in order to whip up his enthusiasm"". It was also mentioned that in the books of the appellant Company the amount had not been debited in the profit and loss account but was debited to the appropriation account thereby indicating that it was an extra payment or a payment made in the nature of a capital expense. Taking all these circumstances into consideration the Tribunal came to the conclusion that it was difficult to hold that the expenditure was not in the nature of a capital expenditure or that it was expended wholly exclusively for the purpose of the assessees business. At the instance of the appellant Company the case was stated to the High Court under S. 66(1) of the Income-tax Act and the following question was referred :-""Whether the sum of Rs. 40,000/- paid to Mr. J. H. Philips on his retirement from the service of the Company was, not an admissible deduction under S. 10 (2)(xv) of the Income-tax Act, 1922."" 4. The High Court answered the question against the appellant Company. It held that in order that S. 10(2) (xv) applicable it had to be proved that the amount was laid out or expended wholly and exclusively for purposes of the companys business. In his case the amount was paid on retirement and for valuable services rendered by Mr. J. H. Philips; there was no evidence that he expected to receive this amount or the Company contemplated its payment at any time before; the payment was voluntary and there was no evidence to show that it was in the future interest of the business of the Company that the expenditure was incurred. The High Court observed : ""In the case of a payment of a gratuity to a retiring employee recognition of his past services, with nothing more cannot, in our opinion satisfy the requirements of S. 10(2) (xv), even if those requirements are judged from the view point of commercial expediency, as it always should be when a claim arises under S, 10(2)(xv). Was the expenditure incurred in the future interest of the business of the assessee? Was there any connection between the purpose of the payment and the further conduct of the business of the assessee ? These are the tests to be satisfies before it could be said that in paying the gratuity money was laid out or expended wholly and exclusively for the purpose of the business of the company. These tests the assessee did not satisfy in this case,"" Against this judgment and order the appellant Company has brought this appeal by special leave. 5. It was argued on behalf of the appellant that the amount had been paid as a matter of commercial expediency and in the interest of the Company as an inducement to other employees that if they rendered service in a similar manner with efficiency and honesty they would be similarly rewarded. Decisive test, it was submitted was whether such payments of gratuity were likely in future also and was the payment made as an incentive to the employees to give their best to the employer and if it was so then the payment was a matter of commercial prudence. It was also submitted that the Company had acted not with any oblique motive and its good faith was not in doubt and in support of the contention several cases were relied upon, 6. In our opinion on the findings as given the payment in dispute does not fall within the provisions of S. 10(2)(xv). The amount was paid not in pursuance of any scheme of payment of gratuities nor was it an amount which the recipient expected to be paid for long and faithful service but it was a voluntary payment not with the object of facilitating the carrying on of the business of the appellant Company or as a matter of commercial expediency but in recognition of long and faithful service of Mr. J. H. Philips. There was no practice in the appellant Company to pay such amounts and it did not affect the quantum of salary of the recipient . The two cases strongly relied upon by the appellant Company were J. P. Hancook v. General Reversionary and Investment Company Ltd. (1919) 7 Tax Cas 358 and J. W. Smith v. Incorporated Council of Law Reporting for England and Wales (1914) 6 Tax Cas 477. In the former case the assssee Company sought to charge as a trade expense a lump sum which it had pad for the purchase, for the benefit of a former actuary, of an annuity equal in amount to the pension which the Company had resolved to pay him. This was held to bean expense admissible in computing the Companys profits assessable to income-tax. But in that else it was the practice of the assessee company to grant pensions to its servants after a considerable period of service and this practice was known to the employees and affected the rate of salary paid by the Company in that the employees were willing to serve the Company at lower rates than they otherwise would have by reason of the expectation of the pension at the end of their service. In the latter case there was a practice of granting gratuities and that was the ground for holding the amount to be a proper deduction. 7. In our opinion the proper test to apply in this case is, was the payment made as a matter of practice which affected the quantum of salary or was there an expectation by the employee of getting a gratuity or was the sum of money expended on the ground of commercial expediency and in order indirectly to facilitate the carrying on of the business. But this has not been shown and therefore the amounts claimed is not a deductible item under S. 10(2)(xv).

Figure S4: Unabridged Integrated Gradients explanation for the case from Figure 2, where the appeal was rejected. The “ground-truth” importance annotation is indicated using a bold font, and the highlights are produced using Integrated Gradients. S19

Kapur, J. 1. This appeal by special leave is directed against the judgment and order of the High Court of Judicature at Madras. The appellant is the assessee and the respondent is the Commissioner of Income-tax and the question raised is as to the applicability of S. 10(2)(xv) of the Indian Income-tax Act to a gratuity paid by the appellant to one of its officers on his retirement from service. 2. The appeal relates to the assessment year 1950-51. M/s. Gordon Woodroffee and Co. (Madras) Ltd., was incorporated as a private limited company in 1922 and became the Managing Agent of a public limited company M/s. Gordon Woodroffee Leather Manufacturing Company Ltd., which is the assessee. One J. H. Philips was employed in the Managing Agent Company from 1922 to 1935 and from 1935 he became an employee of the appellant company and became its Director from 1940. On March 22, 1949, he wrote a letter to the appellant company expressing his intention to resign from the Board of the Company as from April 4, 1949 upon his retirement from the employment of the company and requested that his resignation be accepted. On March 24, 1949, the Board of Directors of the appellant Company passed a resolution that his resignation be accepted and in appreciation of his long and valuable services to the Company he be paid a gratuity of Rs. 50,000/- out of which the appellant Company was to pay Rs. 40,000/-. and the Managing Agent Company the balance of Rs. 10,000/-. On April 4, 1949 this resolution of the Board of Directors was confirmed, on the same date a resolution to the same effect was passed at an Extraordinary General Meeting of the Company and before the end of its accounting year i.e., October 31, 1949, this amount of Rs. 40,000/- was paid to Mr. J. H. Philips. 3. This amount was claimed as a deduction under S. 10(2)(xv) of the Income-tax Act which reads :-Section 10 (2) : ""Such profits or gains shall be computed after making the following allowances, namely : ...................................................... ............. ................ ...... ......... ....... (xv) any expenditure (not being an allowance of the nature described in any of the clauses (i) to (xiv) inclusive, and not being in the nature of capital expenditure or personal expenses of the assessee) laid out or expended wholly and exclusively for the purpose of such business, profession or vocation."" The amount was disallowed by the Income-tax Officer as well as by the Appellate Assistant Commissioner on the ground that the appellant Company had no pension scheme; the payment was voluntary and that the entry in the assessees books clearly indicated it to be a capital payment. Against this order the appellant Company took an appeal to the Income-tax Appellate Tribunal which upheld the order of the Appellate Assistant Commissioner. It held that according to the resolution the gratuity was paid ""for long and valuable services to the Company""; that there was nothing to indicate that Mr. J. H. Philips had accepted a lower salary in expectation of getting a gratuity at, the end of is service; at there was no such practice in the appellant Company and that during the course of his service he was being remunerated at a graduated scale of salary and a commission of it 2 1/2 on the profits; that there was no ""expectancy"" that at the end of the service there would be a recompense for faithful and efficient service; that he had been suitably rewarded by being given a commission on the profits ""in order to whip up his enthusiasm"". It was also mentioned that in the books of the appellant Company the amount had not been debited in the profit and loss account but was debited to the appropriation account thereby indicating that it was an extra payment or a payment made in the nature of a capital expense. Taking all these circumstances into consideration the Tribunal came to the conclusion that it was difficult to hold that the expenditure was not in the nature of a capital expenditure or that it was expended wholly exclusively for the purpose of the assessees business. At the instance of the appellant Company the case was stated to the High Court under S. 66(1) of the Income-tax Act and the following question was referred :-""Whether the sum of Rs. 40,000/- paid to Mr. J. H. Philips on his retirement from the service of the Company was, not an admissible deduction under S. 10 (2)(xv) of the Income-tax Act, 1922."" 4. The High Court answered the question against the appellant Company. It held that in order that S. 10(2) (xv) applicable it had to be proved that the amount was laid out or expended wholly and exclusively for purposes of the companys business. In his case the amount was paid on retirement and for valuable services rendered by Mr. J. H. Philips; there was no evidence that he expected to receive this amount or the Company contemplated its payment at any time before; the payment was voluntary and there was no evidence to show that it was in the future interest of the business of the Company that the expenditure was incurred. The High Court observed : ""In the case of a payment of a gratuity to a retiring employee recognition of his past services, with nothing more cannot, in our opinion satisfy the requirements of S. 10(2) (xv), even if those requirements are judged from the view point of commercial expediency, as it always should be when a claim arises under S, 10(2)(xv). Was the expenditure incurred in the future interest of the business of the assessee? Was there any connection between the purpose of the payment and the further conduct of the business of the assessee ? These are the tests to be satisfies before it could be said that in paying the gratuity money was laid out or expended wholly and exclusively for the purpose of the business of the company. These tests the assessee did not satisfy in this case,"" Against this judgment and order the appellant Company has brought this appeal by special leave. 5. It was argued on behalf of the appellant that the amount had been paid as a matter of commercial expediency and in the interest of the Company as an inducement to other employees that if they rendered service in a similar manner with efficiency and honesty they would be similarly rewarded. Decisive test, it was submitted was whether such payments of gratuity were likely in future also and was the payment made as an incentive to the employees to give their best to the employer and if it was so then the payment was a matter of commercial prudence. It was also submitted that the Company had acted not with any oblique motive and its good faith was not in doubt and in support of the contention several cases were relied upon, 6. In our opinion on the findings as given the payment in dispute does not fall within the provisions of S. 10(2)(xv). The amount was paid not in pursuance of any scheme of payment of gratuities nor was it an amount which the recipient expected to be paid for long and faithful service but it was a voluntary payment not with the object of facilitating the carrying on of the business of the appellant Company or as a matter of commercial expediency but in recognition of long and faithful service of Mr. J. H. Philips. There was no practice in the appellant Company to pay such amounts and it did not affect the quantum of salary of the recipient . The two cases strongly relied upon by the appellant Company were J. P. Hancook v. General Reversionary and Investment Company Ltd. (1919) 7 Tax Cas 358 and J. W. Smith v. Incorporated Council of Law Reporting for England and Wales (1914) 6 Tax Cas 477. In the former case the assssee Company sought to charge as a trade expense a lump sum which it had pad for the purchase, for the benefit of a former actuary, of an annuity equal in amount to the pension which the Company had resolved to pay him. This was held to bean expense admissible in computing the Companys profits assessable to income-tax. But in that else it was the practice of the assessee company to grant pensions to its servants after a considerable period of service and this practice was known to the employees and affected the rate of salary paid by the Company in that the employees were willing to serve the Company at lower rates than they otherwise would have by reason of the expectation of the pension at the end of their service. In the latter case there was a practice of granting gratuities and that was the ground for holding the amount to be a proper deduction. 7. In our opinion the proper test to apply in this case is, was the payment made as a matter of practice which affected the quantum of salary or was there an expectation by the employee of getting a gratuity or was the sum of money expended on the ground of commercial expediency and in order indirectly to facilitate the carrying on of the business. But this has not been shown and therefore the amounts claimed is not a deductible item under S. 10(2)(xv).

Figure S5: Unabridged LIME explanation for the case from Figure 2, where the appeal was rejected. The “ground-truth” importance annotation is indicated using a bold font, and the highlights are produced using LIME with 500 background samples (reduced from 5000). S20

Kapur, J. 1. This appeal by special leave is directed against the judgment and order of the High Court of Judicature at Madras. The appellant is the assessee and the respondent is the Commissioner of Income-tax and the question raised is as to the applicability of S. 10(2)(xv) of the Indian Income-tax Act to a gratuity paid by the appellant to one of its officers on his retirement from service. 2. The appeal relates to the assessment year 1950-51. M/s. Gordon Woodroffee and Co. (Madras) Ltd., was incorporated as a private limited company in 1922 and became the Managing Agent of a public limited company M/s. Gordon Woodroffee Leather Manufacturing Company Ltd., which is the assessee. One J. H. Philips was employed in the Managing Agent Company from 1922 to 1935 and from 1935 he became an employee of the appellant company and became its Director from 1940. On March 22, 1949, he wrote a letter to the appellant company expressing his intention to resign from the Board of the Company as from April 4, 1949 upon his retirement from the employment of the company and requested that his resignation be accepted. On March 24, 1949, the Board of Directors of the appellant Company passed a resolution that his resignation be accepted and in appreciation of his long and valuable services to the Company he be paid a gratuity of Rs. 50,000/- out of which the appellant Company was to pay Rs. 40,000/-. and the Managing Agent Company the balance of Rs. 10,000/-. On April 4, 1949 this resolution of the Board of Directors was confirmed, on the same date a resolution to the same effect was passed at an Extraordinary General Meeting of the Company and before the end of its accounting year i.e., October 31, 1949, this amount of Rs. 40,000/- was paid to Mr. J. H. Philips. 3. This amount was claimed as a deduction under S. 10(2)(xv) of the Income-tax Act which reads :-Section 10 (2) : ""Such profits or gains shall be computed after making the following allowances, namely : ...................................................... ............. ................ ...... ......... ....... (xv) any expenditure (not being an allowance of the nature described in any of the clauses (i) to (xiv) inclusive, and not being in the nature of capital expenditure or personal expenses of the assessee) laid out or expended wholly and exclusively for the purpose of such business, profession or vocation."" The amount was disallowed by the Income-tax Officer as well as by the Appellate Assistant Commissioner on the ground that the appellant Company had no pension scheme; the payment was voluntary and that the entry in the assessees books clearly indicated it to be a capital payment. Against this order the appellant Company took an appeal to the Income-tax Appellate Tribunal which upheld the order of the Appellate Assistant Commissioner. It held that according to the resolution the gratuity was paid ""for long and valuable services to the Company""; that there was nothing to indicate that Mr. J. H. Philips had accepted a lower salary in expectation of getting a gratuity at, the end of is service; at there was no such practice in the appellant Company and that during the course of his service he was being remunerated at a graduated scale of salary and a commission of it 2 1/2 on the profits; that there was no ""expectancy"" that at the end of the service there would be a recompense for faithful and efficient service; that he had been suitably rewarded by being given a commission on the profits ""in order to whip up his enthusiasm"". It was also mentioned that in the books of the appellant Company the amount had not been debited in the profit and loss account but was debited to the appropriation account thereby indicating that it was an extra payment or a payment made in the nature of a capital expense. Taking all these circumstances into consideration the Tribunal came to the conclusion that it was difficult to hold that the expenditure was not in the nature of a capital expenditure or that it was expended wholly exclusively for the purpose of the assessees business. At the instance of the appellant Company the case was stated to the High Court under S. 66(1) of the Income-tax Act and the following question was referred :-""Whether the sum of Rs. 40,000/- paid to Mr. J. H. Philips on his retirement from the service of the Company was, not an admissible deduction under S. 10 (2)(xv) of the Income-tax Act, 1922."" 4. The High Court answered the question against the appellant Company. It held that in order that S. 10(2) (xv) applicable it had to be proved that the amount was laid out or expended wholly and exclusively for purposes of the companys business. In his case the amount was paid on retirement and for valuable services rendered by Mr. J. H. Philips; there was no evidence that he expected to receive this amount or the Company contemplated its payment at any time before; the payment was voluntary and there was no evidence to show that it was in the future interest of the business of the Company that the expenditure was incurred. The High Court observed : ""In the case of a payment of a gratuity to a retiring employee recognition of his past services, with nothing more cannot, in our opinion satisfy the requirements of S. 10(2) (xv), even if those requirements are judged from the view point of commercial expediency, as it always should be when a claim arises under S, 10(2)(xv). Was the expenditure incurred in the future interest of the business of the assessee? Was there any connection between the purpose of the payment and the further conduct of the business of the assessee ? These are the tests to be satisfies before it could be said that in paying the gratuity money was laid out or expended wholly and exclusively for the purpose of the business of the company. These tests the assessee did not satisfy in this case,"" Against this judgment and order the appellant Company has brought this appeal by special leave. 5. It was argued on behalf of the appellant that the amount had been paid as a matter of commercial expediency and in the interest of the Company as an inducement to other employees that if they rendered service in a similar manner with efficiency and honesty they would be similarly rewarded. Decisive test, it was submitted was whether such payments of gratuity were likely in future also and was the payment made as an incentive to the employees to give their best to the employer and if it was so then the payment was a matter of commercial prudence. It was also submitted that the Company had acted not with any oblique motive and its good faith was not in doubt and in support of the contention several cases were relied upon, 6. In our opinion on the findings as given the payment in dispute does not fall within the provisions of S. 10(2)(xv). The amount was paid not in pursuance of any scheme of payment of gratuities nor was it an amount which the recipient expected to be paid for long and faithful service but it was a voluntary payment not with the object of facilitating the carrying on of the business of the appellant Company or as a matter of commercial expediency but in recognition of long and faithful service of Mr. J. H. Philips. There was no practice in the appellant Company to pay such amounts and it did not affect the quantum of salary of the recipient . The two cases strongly relied upon by the appellant Company were J. P. Hancook v. General Reversionary and Investment Company Ltd. (1919) 7 Tax Cas 358 and J. W. Smith v. Incorporated Council of Law Reporting for England and Wales (1914) 6 Tax Cas 477. In the former case the assssee Company sought to charge as a trade expense a lump sum which it had pad for the purchase, for the benefit of a former actuary, of an annuity equal in amount to the pension which the Company had resolved to pay him. This was held to bean expense admissible in computing the Companys profits assessable to income-tax. But in that else it was the practice of the assessee company to grant pensions to its servants after a considerable period of service and this practice was known to the employees and affected the rate of salary paid by the Company in that the employees were willing to serve the Company at lower rates than they otherwise would have by reason of the expectation of the pension at the end of their service. In the latter case there was a practice of granting gratuities and that was the ground for holding the amount to be a proper deduction. 7. In our opinion the proper test to apply in this case is, was the payment made as a matter of practice which affected the quantum of salary or was there an expectation by the employee of getting a gratuity or was the sum of money expended on the ground of commercial expediency and in order indirectly to facilitate the carrying on of the business. But this has not been shown and therefore the amounts claimed is not a deductible item under S. 10(2)(xv).

Figure S6: Unabridged LIME explanation for the case from Figure 2, where the appeal was rejected. The “ground-truth” importance annotation is indicated using a bold font, and the highlights are produced using LIME with 5000 background samples (default configuration). S21

E.3

Auditing Proprietary Black-Box AI Systems without Mimic Models

Replicating The Markup’s audit of Amazon’s Product Recommendation Model Using SHAP on a mimic model is not guaranteed to recover explanations faithful to the original model being audited. We constructed different “mimic” models that emulated the Amazon product recommendation system, and showed that SHAP explanations contradicted each other for the Markup created Random Forest, and the default hyper-parameter Random Forest. We now extend this analysis to the other logistic regression mimic models we created, producing extended versions of Figure 6 showing SHAP explanation distributions and SHAP explanation feature rankings respectively in Figures S10 and S8 respectively. We also show global importances for all models, verifying that mimic models do indeed have different internal mechanisms despite producing predictions with similar accuracy rates. Table S6: Global feature importances (normalised) for the mimic models from Table 2. The two most important features per model are marked in bold. These global explanations (Mean Decrease in Impurity for RF, model coefficients for LogReg) vary significantly, showing that models in the Rashomon set make the same prediction through different internal mechanisms.

product stars product reviews brand is amazon shipped by amazon sold by amazon top clicked random noise

RF (Markup)

RF (Default)

LogReg

LogReg (L1)

LogReg (L2)

0.021 0.055 0.853 0.006 0.052 0.002 0.010

0.122 0.246 0.255 0.027 0.082 0.034 0.233

0.068 0.042 0.650 0.003 0.177 0.032 0.027

0.0 0.0 1.0 0.0 0.0 0.0 0.0

0.084 0.032 0.587 0.012 0.245 0.022 0.018

S22

Importances for "infrastructure" across Resume Summary RoT Explanations

8

LLM Prediction IT Worker Other

20

6 4 2

Neutral Importance 0

0

Importances for "network" across Resume Summary RoT Explanations

Mean RoT Importance: 14.75

Frequency Count

Frequency Count

10

Mean RoT Importance: 19.86

LLM Prediction IT Worker Other

15 10 5

Neutral Importance

5

10

15

20

Token Importance

0

25

0

5

10

15

Token Importance

20

25

(a) ROT importances for “infrastructure” across our resume corpus

(b) ROT importances for “network” across our resume corpus

Importances for "teaching" across Resume Summary RoT Explanations

Importances for "financial" across Resume Summary RoT Explanations

6 4 2 0

Neutral Importance 10

8

6

4

Token Importance

2

Frequency Count

Importances for "construction" across Resume Summary RoT Explanations 20

LLM Prediction IT Worker Other

15 10 5 0

Neutral Importance 10

8

6

4

Token Importance

2

0

(e) ROT importances for “construction” across our resume corpus

Mean RoT Importance: -12.41

LLM Prediction IT Worker Other

30 20 10 0

0

(c) ROT importances for “teaching” across our resume corpus

Mean RoT Importance: -7.48

40

Frequency Count

Frequency Count

8

LLM Prediction IT Worker Other

Neutral Importance 16

14

12

10

8

6

4

2

Token Importance

0

(d) ROT importances for “financial” across our resume corpus

Importances for "professional" across Resume Summary RoT Explanations

Mean RoT Importance: -7.48

100

Frequency Count

Mean RoT Importance: -6.72

LLM Prediction IT Worker Other

80 60 40 20 0

Neutral Importance 15.0

12.5

10.0

7.5

5.0

2.5

Token Importance

0.0

2.5

5.0

(f) ROT importances for “professional” across our resume corpus

Figure S7: Select token importances from zero shot resume classification. Histograms showing the importances obtained using ROT for “infrastructure”, “network”, “teaching”, “financial”, “construction”, and “professional” across resumes. The stacked bar plots are coloured green for token occurrences where the resume belonged to an IT worker, and red for others. Positive numbers on the X axis indicate ROT importances towards hiring as IT workers, and negative S23 numbers indicate non IT workers.

70

Most Important Second Most Important Third or Lesser Importance

60

Relative Importance per Datapoint

abs( SHAP importance )

abs( SHAP importance )

Relative Importance per Datapoint

80

Most Important Second Most Important Third or Lesser Importance

60

50

40

40

30

20

20

Feature Name

random noise

shipped by amazon

product stars

top clicked

random noise

Relative Importance per Datapoint

160

Most Important Second Most Important Third or Lesser Importance

50

(d) The L1 regularized logistic regression picks only “brand is amazon” as important

abs( RoT importance )

abs( SHAP importance )

Relative Importance per Datapoint

60

sold by amazon

Feature Name

(c) The logistic regression model picks a different set of features as most important 70

20

0

random noise

shipped by amazon

product stars

top clicked

sold by amazon

product reviews

brand is amazon

10

40

shipped by amazon

20

60

product stars

30

80

sold by amazon

40

0

Most Important Second Most Important Third or Lesser Importance

product reviews

50

Relative Importance per Datapoint

100

top clicked

Most Important Second Most Important Third or Lesser Importance

60

(b) The default random forest model picks a more diverse set of features as most important

abs( SHAP importance )

abs( SHAP importance )

Relative Importance per Datapoint

70

product reviews

Feature Name

(a) The Markup model finds the most important feature to usually be either “brand is amazon” or “product reviews” 80

brand is amazon

random noise

0

brand is amazon

Feature Name

shipped by amazon

product stars

top clicked

sold by amazon

product reviews

0

brand is amazon

10

Most Important Second Most Important Third or Lesser Importance

140 120 100

40

Feature Name

(e) The L2 regularized logistic regression model picks a different set of features as important

random noise

product stars

random noise

0

top clicked

20

shipped by amazon

Feature Name

shipped by amazon

product stars

top clicked

sold by amazon

product reviews

0

brand is amazon

10

60

sold by amazon

20

80

product reviews

30

brand is amazon

40

(f) ROT usually finds “brand is amazon” to be the most important, and often finds “sold by amazon” to be the second most important

Figure S8: Auditing a proprietary recommendation system using XAI. SHAP finds a different ordering of importances for the features, depending on the mimic model used, whereas ROT provides a consistent result and uncovers an additional insight highlighting the importance of “sold by amazon” S24

Feature Name

Relative Importance per Datapoint

random noise random noise

product stars

Relative Importance per Datapoint

abs( RoT importance )

Most Important Second Most Important Third or Lesser Importance

140 120 100

40

Feature Name

(e) The L2 regularized logistic regression model picks a different set of features as important

random noise

product stars

top clicked

random noise

0

sold by amazon

20

shipped by amazon

Feature Name

shipped by amazon

product stars

sold by amazon

product reviews

0

brand is amazon

20

60

product reviews

40

80

brand is amazon

60

top clicked

abs( LIME importance )

80

(d) The L1 regularized logistic regression picks only “brand is amazon” as important 160

Most Important Second Most Important Third or Lesser Importance

100

20

Feature Name

(c) The logistic regression model picks a different set of features as most important 120

40

0

random noise

shipped by amazon

product stars

top clicked

sold by amazon

product reviews

brand is amazon

25

60

top clicked

50

80

sold by amazon

75

Feature Name

Most Important Second Most Important Third or Lesser Importance

100

100

0

Relative Importance per Datapoint

120

Most Important Second Most Important Third or Lesser Importance

125

(b) The default random forest model picks a more diverse set of features as most important

abs( LIME importance )

abs( LIME importance )

Relative Importance per Datapoint

150

shipped by amazon

random noise

0

(a) The Markup model finds the most important feature to be “brand is amazon” with “sold by amazon” second 175

20

product reviews

Feature Name

shipped by amazon

product stars

top clicked

sold by amazon

product reviews

brand is amazon

20

40

shipped by amazon

40

60

sold by amazon

60

product reviews

80

80

brand is amazon

100

0

Most Important Second Most Important Third or Lesser Importance

100

brand is amazon

120

Relative Importance per Datapoint

120

product stars

Most Important Second Most Important Third or Lesser Importance

140

top clicked

abs( LIME importance )

abs( LIME importance )

Relative Importance per Datapoint

160

(f) ROT usually finds “brand is amazon” to be the most important, and often finds “sold by amazon” to be the second most important

Figure S9: Auditing a proprietary recommendation system using XAI. LIME finds a different ordering of importances for the features, depending on the mimic model used, whereas ROT provides a consistent result and uncovers an additional insight highlighting the importance of “sold by amazon” S25

0.75

0.75

0.50 0.25

0.25

Importance

Importance

0.50

0.00 0.25

0.25

0.50

0.50

0.75

0.75

product stars

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

random noise

product stars

(a) The Markup model finds “brand is amazon” and “product reviews” to have importances of high magnitude

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

random noise

(b) The default random forest model generates a different profile of importance distributions

0.75

0.4

0.50

0.2

0.25

0.0

Importance

Importance

0.00

0.00 0.25

0.2 0.4 0.6

0.50

0.8 0.75

1.0

product stars

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

product stars

random noise

(c) The logistic regression model finds a different set of importances than both random forest models

Feature Name

top clicked

random noise

(d) The L1 regularized regression model produces considers only “brand is amazon” to be important

0.75

0.75

0.50

0.50

0.25

0.25

Importance

Importance

product brand is shipped sold by reviews amazon by amazon amazon

0.00 0.25 0.50

0.00 0.25 0.50

0.75

0.75

product stars

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

random noise

product stars

(e) The L2 regularized regression model produces a different importance distribution

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

random noise

(f) ROT importances can be computed without any mimic models

Figure S10: Auditing a proprietary recommendation system using XAI. SHAP Importance Distributions for Input Features Can Vary by Choice of Mimic Model, whereas ROT provides a Single and Robust Set of Importance Distributions for all Features, and Cannot be Manipulated by Choice of Mimic Model S26

0.6

0.50

0.4

0.25

0.2

Importance

Importance

0.75

0.00 0.25

0.0 0.2

0.50

0.4

0.75

0.6

product stars

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

product stars

random noise

(a) The Markup model finds “brand is amazon” and “product reviews” to have importances of high magnitude

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

random noise

(b) The default random forest model generates a different profile of importance distributions

1.00

0.6

0.75

0.4 0.2

0.25

Importance

Importance

0.50

0.00 0.25 0.50

0.6

product stars

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

product stars

random noise

(c) The logistic regression model finds a different set of importances than both random forest models

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

random noise

(d) The L1 regularized regression model produces considers only “brand is amazon” to be important

0.6

0.75

0.4

0.50

0.2

0.25

Importance

Importance

0.2 0.4

0.75 1.00

0.0

0.0 0.2

0.00 0.25 0.50

0.4

0.75

0.6

product stars

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

random noise

product stars

(e) The L2 regularized regression model produces a different importance distribution

product brand is shipped sold by reviews amazon by amazon amazon

Feature Name

top clicked

random noise

(f) ROT importances can be computed without a mimic model

Figure S11: Auditing a proprietary recommendation system using XAI. LIME Importance Distributions for Input Features Can Vary by Choice of Mimic Model, whereas ROT provides a Single and Robust Set of Importance Distributions for all Features, and Cannot be Manipulated by Choice of Mimic Model S27

E.4

Scientific Discovery using XAI

Identifying Important Features in the presence of Misleading Features Table S7 effectively summarises the most important feature across datapoints, as picked by various explainers. This is the underlying explanation data that was plotted in the ternary distribution plots in Figure 8. To provide a more granular analysis, we also provide additional confusion matrices comparing the first, second, and third most important features as selected by various explainers across various datasets. in Tables S8, S9, S10, S11, and S12. Table S7: ROT Successfully Determines the Sensitive Feature as Most Important, Despite the Presence of Misleading Foil Features: adversarially crafted models mL and mS are designed to mislead LIME and SHAP respectively into selecting the Foil features as most important. For both adversarial models, we verify that ROT successfully identifies the Sensitive feature as most important. In each case, we present the percentage of predictions that yield a particular input feature as most important. Successes are in bold green, adversarial failures in bold red, and non-adversarial failures in bold blue. Dataset

German Credit

COMPAS

Communities and Crime

Adversarial Model

Explainer

mL (1 foil) mS (1 foil) mL (1 foil) mS (1 foil) mL (2 foils) mS (2 foils) mL (1 foil) mS (1 foil) mL (2 foils) mS (2 foils)

LIME ROT SHAP ROT LIME ROT SHAP ROT LIME ROT SHAP ROT LIME ROT SHAP ROT LIME ROT SHAP ROT

Percentage of Dataset where the Most Important Input Feature is: a) Sensitive

b) Foil #1

c) Foil #2

d) Other

0.0 100.0 0.0 100.0 0.0 100.0 1.9 100.0 0.3 100.0 35.0 100.0 0.0 89.5 4.0 90.5 0.0 91.0 1.0 90.0

84.0 0.0 100.0 0.0 100.0 0.0 90.9 0.0 43.9 0.0 18.9 0.0 99.5 0.0 87.0 0.0 14.5 0.0 49.0 0.0

na na na na na na na na 47.9 0.0 22.3 0.0 na na na na 74.5 0.0 50.0 0.0

16.0 0.0 0.0 0.0 0.0 0.0 7.1 0.0 7.9 0.0 23.8 0.0 0.5 10.5 9.0 9.5 11.0 9.0 0.0 10.0

S28

Table S8: COMPAS dataset, with a synthetic foil feature: Confusion matrices showing what percentage of explanations select particular features as the most, second most, and third most important features in explanations. Ideally, the most important feature recovered by an explainer should be the only one used (highlighted green), whereas the adversary tries to have the explaners always pick a different feature (highlighted red). The most important feature (first column) from each matrix is replicated in Table S7 (a) SHAP explanations for adversarially crafted model

Importance Rank Synthetic Foil #1 race other features

1

2

3

90.9 1.9 7.1

7.3 55 37.7

1.8 19.9 78.3

(b) ROT explanations for adversarial model

Importance Rank Synthetic Foil #1 race other features

(c) LIME explanations for adversarial model

Importance Rank Synthetic Foil #1 race other features

1

2

3

100.0 0.0 0.0

0.0 23.1 76.9

0.0 18.8 81.2

1

2

3

0.0 100.0 0.0

0.0 0.0 100.0

0.0 0.0 100

(d) ROT explanations for adversarial model

Importance Rank Synthetic Foil #1 race other features

1

2

3

0.0 100.0 0.0

0.0 0.0 100.0

0.0 0.0 100.0

Table S9: COMPAS dataset, with two synthetic foil features: Confusion matrices showing what percentage of explanations select particular features as the most, second most, and third most important features in explanations. Ideally, the most important feature recovered by an explainer should be the only one used (highlighted green), whereas the adversary tries to have the explaners always pick a different feature (highlighted red). The most important feature (first column) from each matrix is replicated in Table S7 (a) SHAP explanations for adversarially crafted model

Importance Rank Synthetic Foil #1 Synthetic Foil #2 race other features

1

2

3

18.9 22.3 35.0 23.8

23.6 27.0 23.6 25.7

26.4 18.4 12.6 42.6

(b) ROT explanations for adversarial model

Importance Rank Synthetic Foil #1 Synthetic Foil #2 race other features

(c) LIME explanations for adversarial model

Importance Rank Synthetic Foil #1 Synthetic Foil #2 race other features

1

2

3

43.9 47.9 0.3 7.9

39.0 27.5 3.1 30.4

7.0 10.7 13.3 69.1

1

2

3

0.0 0.0 100.0 0.0

1.6 0.0 0.0 98.4

8.1 0.0 0.0 91.9

(d) ROT explanations for adversarial model

Importance Rank Synthetic Foil #1 Synthetic Foil #2 race other features

S29

1

2

3

0.0 0.0 100.0 0.0

0.0 0.0 0.0 100.0

0.0 0.0 0.0 100.0

Table S10: Communities and Crime dataset, with a synthetic foil feature: Confusion matrices showing what percentage of explanations select particular features as the most, second most, and third most important features in explanations. Ideally, the most important feature recovered by an explainer should be the only one used (highlighted green), whereas the adversary tries to have the explaners always pick a different feature (highlighted red). The most important feature (first column) from each matrix is replicated in Table S7 (a) SHAP explanations for adversarially crafted model

Importance Rank Synthetic Foil #1 race other features

1

2

3

87.0 4.0 9.0

12.5 40.5 42.5

0.5 9.5 90.0

Importance Rank Synthetic Foil #1 race other features

(c) LIME explanations for adversarial model

Importance Rank Synthetic Foil #1 race other features

1

2

3

99.5 0.0 0.5

0.0 0.5 99.5

0.0 0.0 100

(b) ROT explanations for adversarial model

1

2

3

0.0 90.5 9.5

0.0 4.5 95.5

0.0 0.5 99.5

(d) ROT explanations for adversarial model

Importance Rank Synthetic Foil #1 race other features

1

2

3

0.0 89.5 10.5

0.0 5.5 94.5

0.0 1.0 99.0

Table S11: Communities and Crime dataset, with two synthetic foil features: Confusion matrices showing what percentage of explanations select particular features as the most, second most, and third most important features in explanations. Ideally, the most important feature recovered by an explainer should be the only one used (highlighted green), whereas the adversary tries to have the explaners always pick a different feature (highlighted red). The most important feature (first column) from each matrix is replicated in Table S7 (a) SHAP explanations for adversarially crafted model

Importance Rank Synthetic Foil #1 Synthetic Foil #2 race other features

1

2

3

49.0 50.0 1.0 0.0

23.5 21.5 21.5 33.5

7.5 14.5 15.5 62.5

Importance Rank Synthetic Foil #1 Synthetic Foil #2 race other features

(c) LIME explanations for adversarial model

Importance Rank Synthetic Foil #1 Synthetic Foil #2 race other features

1

2

3

14.5 74.5 0.0 11.0

23.5 11.5 0.5 64.5

8.5 3.5 1.0 87.0

(b) ROT explanations for adversarial model

1

2

3

0.0 0.0 90.0 10.0

0.0 0.0 4.5 95.5

0.0 0.0 1.5 98.5

(d) ROT explanations for adversarial model

Importance Rank Synthetic Foil #1 Synthetic Foil #2 race other features

S30

1

2

3

0.0 0.0 91.0 9.0

0.0 0.0 4.0 96.0

0.0 0.0 1.0 99.0

Table S12: German credit dataset, with an engineered foil feature (LoanRate as % of Income): Confusion matrices showing what percentage of explanations select particular features as the most, second most, and third most important features in explanations. Ideally, the most important feature recovered by an explainer should be the only one used (highlighted green), whereas the adversary tries to have the explaners always pick a different feature (highlighted red). The most important feature (first column) from each matrix is replicated in Table S7 (a) SHAP explanations for adversarially crafted model

Importance Rank LoanRate % of Income gender other features

1

2

3

100.0 0.0 0.0

0.0 40.0 60.0

0.0 5.0 95.0

(b) ROT explanations for adversarial model

Importance Rank LoanRate % of Income gender other features

(c) LIME explanations for adversarial model

Importance Rank LoanRate % of Income gender other features

1

2

3

0.0 100.0 0.0

0.0 0.0 100.0

0.0 0.0 100.0

(d) ROT explanations for adversarial model

1

2

3

84.0 0.0 16.0

0.0 1.0 99.0

0.0 1.0 99.0

Importance Rank LoanRate % of Income gender other features

S31

1

2

3

0.0 100.0 0.0

0.0 0.0 100.0

2.0 0.0 98.0

Related documents

• Explaining Attention with Program Synthesis#287112
• A Study on the Characteristics of Artificial Intelligence Design Cases#490904
• Understanding and Mitigating Corner Cases with Contextual Knowledge in Deep Learning for Automated Driving / Jingxing Zhou ; Jürgen Beyerer, Steven Peters#1052388
• Perspective: Purposeful Failure in Artificial Life and Artificial Intelligence#478104
• The Impact of Artificial Intelligence on Branding#686430