VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

Viet Vo

Caslon Chua

School of Science, Computing and Emerging Technologies Swinburne University of Technology Hawthorn, Victoria, Australia vvo@swin.edu.au

School of Science, Computing and Emerging Technologies Swinburne University of Technology Hawthorn, Victoria, Australia cchua@swin.edu.au

20 26

Md Jannatul Rakib Joy School of Science, Computing and Emerging Technologies Swinburne University of Technology Hawthorn, Victoria, Australia 103799644@student.swin.edu.au

Fairness audits in clinical Artificial Intelligence convert continuous fairness metrics into binary pass-or-fail verdicts against operational thresholds, where hospital governance boards, payers, and regulators act on the resulting verdicts. Such audits are repeated over time and across hospital sites, thus the same verdict can flip between pass and fail across audits. Existing uncertainty methods such as Bayesian posteriors, bootstrap confidence intervals, and permutation tests address verdict instability only at the continuous-metric level. Converting metric-level uncertainty into a verdict-stability claim remains a manual step that scales poorly across the (model, metric, attribute) cells an audit covers. Existing uncertainty methods also leave open whether bias-mitigation steps, such as reweighing or per-group threshold shifts, yield a stable passing verdict at the cost of model discrimination measured as AUROC or AUPRC. To address this verdict-stability gap, we propose VFR-Audit, a framework built around the Verdict Flip Rate (VFR), a scalar bounded between 0 and 0.5 that measures the probability of verdict reversal under stratified bootstrap resampling. VFR-Audit reports VFR alongside three reliability axes, namely within-cohort resampling stability, audit-size sensitivity, and cross-hospital verdict agreement via Fleiss’ kappa. The framework was evaluated on 925,128 administrative discharge records from 441 Texas hospitals, against an intersectional adaptation of Kamiran and Calders’ reweighing. On the unintervened baseline, the audit detects verdict reversal in 43.5% of cells in the cross-model audit grid. After the canonical intervention, 11 of 28 cells continue to flip. Intersectional reweighing fails the four-fifths rule on this cohort regardless of regularisation strength, while per-cell threshold shifting is the main effective component of the canonical intervention. VFR translates continuous metric uncertainty into a probability of binary-verdict reversal, and a derived-baseline comparison attributes the verdict outcome to specific intervention components, so audit reliability and intervention effects can be reported together rather than as separate claims.

• Applied computing → Health informatics; • Computing methodologies → Machine learning; • Social and professional topics → Computing / technology policy; • Information systems → Data mining.

pt ed

This work is licensed under a Creative Commons Attribution-NonCommercialNoDerivatives 4.0 International License. CIKM ’26, November 7–11, 2026, Rome, Italy © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2539-5/2026/11 https://doi.org/10.1145/3799682.3840827

KM

CCS Concepts

in

Abstract

Ac ce

Keywords

algorithmic fairness, clinical AI, audit reliability, verdict stability, length-of-stay prediction, multi-site validation, responsible deployment ACM Reference Format: Md Jannatul Rakib Joy, Viet Vo, and Caslon Chua. 2026. VFR-Audit: VerdictLevel Reliability for Fairness Audits in Hospital Length-of-Stay Prediction. In Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM ’26), November 7–11, 2026, Rome, Italy. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3799682.3840827

CI

arXiv:2608.30846v1 [cs.AI] 31 Aug 2026

VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

1

Introduction

Length-of-stay (LOS) prediction supports discharge planning, bed management, and care coordination [11, 27, 32, 51]. As clinical AI systems move closer to deployment, fairness evaluation has become a central requirement for assessing whether model behaviour is acceptable across protected patient subgroups defined by race, sex, ethnicity, and age [2, 28, 30, 38, 43, 69]. Recent surveys document the breadth of healthcare-AI fairness concerns and the persistence of subpopulation disparities even in high-performance models [36, 40, 48, 60]. Many fairness audits summarise model behaviour by comparing a continuous fairness metric against a predefined threshold, producing a binary pass-or-fail verdict that hospital governance boards, payers, and regulators read and act upon [5, 54, 66]. This thresholded verdict is data-dependent. In practical deployment, hospitals acquire new patient records over time. Demographic composition, subgroup size, and site-level case mix of the audit cohort change [7, 23, 25, 52, 55]. A fairness verdict that passes on one cohort may therefore fail on another even when the trained model is unchanged [17]. For example, when Disparate Impact is assessed using the four-fifths rule from the U.S. Equal Employment Opportunity Commission [20, 68], small changes in cohort composition around the 0.80 threshold can change the audit conclusion. Two deployed-system audits illustrate the operational stakes. Obermeyer et al. [47] audited a commercial risk-prediction tool deployed across major U.S. health systems and found that, at the same predicted risk score, Black patients had substantially worse health than White

CIKM ’26, November 7–11, 2026, Rome, Italy

Md Jannatul Rakib Joy, Viet Vo, and Caslon Chua

Ac ce

pt ed

in

1 Throughout this paper, “clinical AI” refers broadly to machine-learning systems used

for patient-risk prediction, hospital operations, and decision support. The LOS task studied here is primarily a hospital-operational prediction task with clinical-decisionsupport implications. The framework itself is task-agnostic.

KM

20 26

We validate the framework on the Texas-100X cohort, which contains 9.25 × 105 administrative discharge records from 441 hospitals [63], audited using twelve standard machine-learning classifiers, seven fairness metrics, and four protected attributes (336 audit cells per pass). On the unintervened baseline, 146 of 336 (43.5 %) audited cells in the cross-model grid reverse their verdict under stratified bootstrap resampling. On the canonical post-intervention baseline, 11 of 28 cells continue to flip even when the point-estimate audit certifies all four protected attributes as fair. A four-baseline comparison shows that intersectional reweighing alone fails to satisfy the four-fifths rule on this dataset regardless of the regularisation strength applied, while per-cell threshold shifting is the component responsible for the all-four-DI pass. The main contributions of this paper are as follows. • We propose the Verdict Flip Rate (VFR), a scalar measure of binary fairness-verdict instability under stratified bootstrap resampling, with a fixed resample-count selection rule motivated by worst-case binomial uncertainty and concentration bounds. • We develop a three-axis reliability framework around VFR covering within-cohort resampling, audit-size guidance, and cross-hospital agreement, combined into a four-tier per-cell reliability classification. • We empirically demonstrate the framework on Texas-100X, showing that 146 of 336 cells flip under bootstrap on the unintervened baseline, 11 of 28 flip after the canonical intervention, and per-cell threshold shifting is the component responsible for the all-four-DI pass.

CI

patients, with the corrective audit shifting the Black share of careprogram enrolment from 17.7 % to 46.5 %. Omar et al. [49] stresstested nine large language models on 1,000 emergency-department cases across 32 sociodemographic variants (1.7 × 106 outputs) and found that cases labelled as Black, unhoused, or LGBTQIA+ were more frequently routed to urgent care, invasive interventions, or mental-health evaluations despite identical clinical details. The deployment-relevant verdict therefore depended on whether and how the audit was performed. Modern healthcare-AI systems, including generative and agentic systems, are updated and redeployed more frequently than traditional static models [3, 57, 62, 64], making one-time certification insufficient [8, 16, 17, 22, 52]. Governance work likewise treats fairness as a lifecycle monitoring requirement [43, 69]. The EU AI Act [21], FDA PCCP guidance [65], and NIST AI RMF [61] reinforce repeatable post-deployment evidence, while deployment and generalisability work highlights validation across settings [7, 25, 55]. Existing uncertainty methods quantify metric uncertainty using Bayesian posteriors [6], bootstrap confidence intervals [12], permutation tests [19], and sample-size formulae [59], while mitigation methods modify training or post-processing [41, 71]. Neither directly reports how often the resulting thresholded pass-or-fail verdict reverses across repeated audits. Accordingly, our problem is to determine whether a thresholded fairness verdict for a fixed trained model remains reproducible under cohort resampling, audit-size variation, and hospital shift. To address this gap, we propose the Verdict Flip Rate (VFR), a scalar measure of binary-verdict instability under stratified bootstrap resampling. VFR is the smaller of the pass-count and the fail-count across 𝐾 resamples of the audit cohort, normalised by 𝐾, and bounded in [0, 0.5]. A VFR of zero indicates that all 𝐾 resamples produce the same verdict. A VFR of 0.5 indicates that the pass-count equals the fail-count, so the audit grid carries no information about which verdict the next resample would return. VFR separates fairness status from verdict reliability. A model can be stably fair, stably unfair, or unstable near the threshold. The choice of bootstrap count 𝐾 is fixed by a worst-case binomial standard-error argument with concentration-bound support and transfers across cohorts without re-derivation. To operationalise VFR for repeated deployment audits, we develop the VFR-Audit framework. Axis 1 evaluates within-cohort resampling stability by repeatedly auditing the same trained model on stratified bootstrap resamples. Axis 2 provides audit-size guidance. The axis estimates the minimum cohort size at which a fairness metric becomes stable, defined as the smallest size at which the coefficient of variation of the metric falls below five per cent across thirty repetitions. Axis 3 evaluates cross-hospital verdict agreement using hospital-grouped validation and Fleiss’ 𝜅. Together, the three axes identify whether unreliable fairness conclusions arise from cohort resampling variation, insufficient audit size, or hospital-level heterogeneity. They feed a deterministic decision rule that classifies each (model, metric, attribute) cell into one of four reliability tiers.1

2

Related Work

This section places the Verdict Flip Rate in the context of three lines of prior work: length-of-stay prediction, fairness audits in clinical AI, and audit-instrument uncertainty quantification.

2.1

Length-of-Stay Prediction

Hospital LOS prediction has been studied as binary classification (LOS > 3, 5, or 7 days) and as continuous regression. Reported predictive performance is high. Rajkomar et al. [51] reached AUROC = 0.85 to 0.86 on LOS > 7 d with a deep-learning ensemble (LSTM, time-aware attention, and boosted time-based decision stumps) over 2.16 × 105 admissions from two academic medical centres. Wu et al. [70] trained on 1.17 × 105 eICU-CRD ICU patients (internal 70/30 split, AUROC 0.742) and externally validated on 4.29×104 MIMIC-III patients (AUROC 0.747) using a gradient-boosted decision tree for prolonged-ICU-LOS prediction. Jaotombo et al. [33] analysed 7.32 × 104 hospitalisations from a French university hospital and reached AUC = 0.810 with gradient boosting, outperforming random forest, neural network, and logistic regression baselines (all 𝑝 < 0.0001). In the regression framing, Mekhaldi et al. [45] reached 𝑅 2 = 0.94 and MAE = 0.44 d with gradient boosting on 1.00×105 observations from an open hospital-LOS benchmark. Jain et al. [32] reported 𝑅 2 = 0.82 for newborns using linear regression and 𝑅 2 = 0.43 for non-newborns using CatBoost regression on 2.30 × 106 SPARCS records. Rocheteau et al. [53] reached MAE = 1.55 d on remainingLOS regression with a temporal pointwise convolutional network

VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

CIKM ’26, November 7–11, 2026, Rome, Italy

Table 1: LOS-focused literature comparison across six audit-relevant axes. “Stab.” is verdict stability, “Site” is cross-site validation, “Size” is audit-size guidance. “n/r” denotes not reported. Dataset

𝑁 (unit)

Task

Model

Performance

Stab.

Site

Size

Rajkomar et al. [51]

UCSF/UCM (private)

2.16 × 105 adm.

LOS > 7 d

DL ensemble (LSTM / TANN / boosted time-based stumps) Temporal pointwise conv. GBDT (best of 4) Linear regr. / CatBoost regr. ProtoEHR (proto. learning) GB + mitigation RF + threshold opt.

AUROC 0.85 to 0.86

n/r

multi (2)

n/r

MAE 1.55 d AUROC 0.742 / 0.747 𝑅 2 ≈ 0.82 (NB) +7.4 F1 (MIMIC-IV LOS) AUROC 0.83 AUC 0.759 (M) / 0.756 (F)

n/r n/r n/r n/r n/r n/r

n/r external n/r n/r n/r n/r

n/r n/r n/r n/r n/r n/r

AUROC 0.953

VFR

𝑲hosp =20 min-N∗

THCIC PUDF (public)

9.25 × 105 records

LOS > 3 d

Fairness Audits in Clinical AI

pt ed

2.2

in

on the eICU benchmark. Recent deep-learning advances include ProtoEHR [11], which evaluates LOS prediction on MIMIC-III (5.45×103 patients, 1.43 × 104 visits) and MIMIC-IV (5.15 × 104 patients, 1.67 × 105 visits) and reports a 7.4 % F1 improvement for LOS on MIMIC-IV. Fairness coverage in the LOS literature is sparse. Li et al. [39] audited a 2.10 × 105 -patient heart-failure cohort on Disparate Impact, Equal Opportunity, Equalised Odds, and Predictive Parity and found that adding social-determinant features reduced subgroupAUC disparities by 0.022 to 0.037 across race strata. Abakasanga et al. [1] analysed 9.62 × 103 patients with learning disabilities and reported random-forest AUCs of 0.759 for males and 0.756 for females. Chesley et al. [13] analysed 102,362 adults with sepsis and/or acute respiratory failure across 27 U.S. hospitals and found that, after matching, Black patients had longer LOS than White patients by 1.26 d for sepsis and 0.97 d for acute respiratory failure. None of the surveyed LOS studies reports verdict stability under cohort resampling, cross-hospital verdict agreement through GroupKFold or Fleiss’ 𝜅, or audit-size guidance for the metrics they evaluate.

Ac ce

Group-fairness criteria including Disparate Impact, Statistical Parity Difference, Equal Opportunity, Equalised Odds, the Theil index, Predictive Parity, and Calibration are formalised in the textbook of Barocas, Hardt and Narayanan [5] and surveyed in detail by Verma and Rubin [66] and Mehrabi et al. [44]. Corbett-Davies et al. [15] synthesise the field through a measure-versus-mismeasure lens, arguing that no single metric is sufficient for governance, in part because the well-known impossibility result prevents simultaneous satisfaction of more than one criterion when group base-rates differ. These criteria have been applied to clinical prediction across domains spanning intensive-care mortality, ED triage, sepsis, readmission, chest-X-ray classification, and dermatology [18, 46, 50, 56, 67, 72]. Scoping reviews by Mbakwe et al. [42] and, more recently, Liu et al. [40] document accelerating uptake but also persistent metric heterogeneity, the narrow focus on bias-relevant attributes, and the dominance of group fairness centered on model-performance equality; Hasanzadeh et al. [30] systematically review bias-recognition and mitigation strategies across the AI model lifecycle, from conception through to deployment and longitudinal surveillance. Intersectional fairness, the requirement that fairness hold jointly across combinations of protected attributes [42, 56], motivates Baseline 2 in our experiment to operate on the joint (race × age × sex) cell structure.

GBDT (canon., 12-model panel)

Two purpose-built clinical-AI fairness-audit frameworks have appeared. Cachel and Rundensteiner [10] developed FINS for groupfairness auditing of subset selections such as top-𝑘, multi-winner voting, and clustering. Barnard et al. [4] developed MACAIF, a clinician-facing dashboard layered on the MLighter adversarial tool. Neither framework defines a scalar measure of binary-verdict instability, and neither has been validated on a state-wide clinical cohort.

2.3

Audit-Uncertainty and Verdict Stability

Statistical-uncertainty quantification for fairness metrics has three established branches. The Bayesian branch [6] computes posteriors over fairness-metric values. The frequentist branch [12] computes bootstrap confidence intervals and simultaneous bounds on subgroup performance disparities. The hypothesis-testing branch [19] tests null hypotheses through permutation 𝑝-values. A complementary line of work derives sample-size formulae for fairness audits. Singh et al. [59] provide closed-form expressions via the central limit theorem and the delta method. A separate line of work directly addresses verdict-level stability. Ganesh et al. [26] and Cooper et al. [14] show that fairness verdicts can change under random training-seed variation; Black et al. [9] document predictive multiplicity, the phenomenon that competing models with similar accuracy disagree on individual predictions. The conclusion across this work is consistent. Fairness verdicts are not stable in deployment, and the instability is not solely caused by sampling noise. None of these works, however, defines a scalar instrument for binary-verdict instability under stratified bootstrap that is suitable for governance reporting. The present work fills this gap. Across the closest LOS studies summarised in Table 1, none reports verdict stability under cohort resampling, cross-hospital verdict agreement through GroupKFold or Fleiss’ 𝜅, or audit-size guidance, which defines the empirical gap addressed here.

CI

This study

remaining-LOS regr. prolonged ICU LOS LOS regression LOS multi-class LOS > 5 d (HF) LOS prediction (LD)

KM

Rocheteau et al. [53] eICU-CRD (public) n/r Wu et al. [70] eICU → MIMIC-III 1.17 × 105 / 4.29 × 104 pts. Jain et al. [32] SPARCS (public) 2.30 × 106 records Cai et al. [11] MIMIC-III & MIMIC-IV 5.15 × 104 pts. / 1.67 × 105 visits Li et al. [39] GWTG-HF registry 2.10 × 105 patients Abakasanga et al. [1] SAIL LD (UK; restricted) 9.62 × 103 patients

20 26

Study

3 Proposed Method 3.1 Notation Table 2 fixes the notation used throughout the paper. To avoid ambiguity, 𝑁 denotes the number of discharge records in an audit cohort, not the number of unique patients. The same record may correspond to one of multiple admissions for a single patient, and the audit instrument operates on the record-level joint distribution.

CIKM ’26, November 7–11, 2026, Rome, Italy

Md Jannatul Rakib Joy, Viet Vo, and Caslon Chua

Symbol

Meaning

D 𝑁 𝑓,M 𝑎, A 𝑚, F

Audit dataset Records in D (not patients) Trained model; model panel Protected attribute; set Fairness metric; metric set

𝜏𝑚 𝑣 ( 𝑓 , D, 𝑚, 𝑎) C 𝐾 VFR(𝑐 ) 𝜎 (𝑐 ) 𝑁 ∗ (𝑐 )

Discharge records 103 to 106 XGBoost, | M | = 12 Race, sex, eth., age DI, SPD, EOPP, EOD, TI, PP, Cal Operational threshold for 𝑚 DI ≥ 0.80 Binary verdict { Pass, Fail } Audit grid M × F × A 336 cells (this study) Bootstrap-resample count (Axis 1) 500 Verdict Flip Rate of cell 𝑐 [0, 0.5] Stability margin (SD from 𝜏 ) ≥0 Minimum reliable audit size (Axis 2) Smallest 𝑁 with CV < 0.05 Field-realistic cohort size 5 × 104 Hospital-fold count (Axis 3) 20 Metric-level Fleiss’ 𝜅 assigned to 𝑐 [ −1, 1] , Landis-Koch bands

KM

𝑁 field 𝐾hosp 𝜅 (𝑐 )

Example / range

Í𝐾 𝑛 pass (𝑐) = 𝑘=1 𝑣𝑘 the pass count across 𝐾 resamples. The Verdict Flip Rate is defined as
min 𝑛 pass (𝑐), 𝐾 − 𝑛 pass (𝑐) VFR(𝑐) = . (1) 𝐾 The statistic is bounded in [0, 0.5] by construction. VFR(𝑐) = 0 holds when all 𝐾 resamples produce the same verdict. VFR(𝑐) = 0.5 holds when the verdicts split evenly between pass and fail. The dominant verdict 𝑣¯ (𝑐) ∈ {Pass, Fail} is the majority outcome across the 𝐾 resamples. We set the operational verdict-stability threshold to 𝜏VFR = 0.10, so a cell is practically stable only when at least 90 % of the resampled audits agree with the dominant verdict. This Axis 1 criterion is distinct from the 𝐶𝑉 < 0.05 criterion used by Axis 2 to determine the minimum reliable audit size 𝑁 ∗ , and it is also independent of the DI four-fifths threshold of 0.80. A reliabilityaware audit therefore reports both the dominant verdict and its stability score. The two quantities together separate fairness status from verdict reliability.

20 26

Table 2: Notation used in the VFR-Audit framework.

CI

3.3.1 Selecting the bootstrap count 𝐾. For a fixed cell 𝑐, the pass count 𝑛 pass (𝑐) is an empirical count over 𝐾 resampled audits. For a pass probability 𝑝, the √︁ Monte Carlo standard error of the empirical pass proportion is 𝑝 (1 − 𝑝)/𝐾, which is maximised at 𝑝 = 0.5. With 𝐾 = 500, the worst-case standard error is 0.0224, giving an approximate 95 % half-width of ±0.044. We therefore use 𝐾 = 500 as a fixed audit budget that keeps Monte Carlo error below 0.05 VFR units while remaining tractable across the full audit grid. Hoeffding’s inequality [31] can additionally bound the probability that the empirical pass proportion deviates from its expectation by more than a chosen tolerance 𝜖. The bound is distribution-free and depends neither on the size of the audit cohort nor on the distribution of the protected attribute, so the same 𝐾 = 500 transfers across cohorts at the same precision target without re-derivation.

in

Figure 1: VFR-Audit framework. Audit inputs feed the Fairness Verdict Engine, which evaluates each audit cell on three reliability axes. The Reliability Report records the per-cell outputs that downstream governance review consumes.

Design Intuition and Deployment Rationale

pt ed

3.2

Ac ce

Hospital governance acts on thresholded fairness verdicts, for example DI ≥ 0.80, rather than metric values alone. In practical deployment, fairness audits are repeated over time as new patient records are acquired and across sites as the model is run at different hospitals [52, 56]. Metric-level uncertainty therefore leaves a deployment gap. It does not directly report how often a pass-or-fail decision reverses, while mitigation can improve a point estimate without ensuring stability across cohorts or hospitals. VFR-Audit treats the verdict as the operational object. A single resampling axis is insufficient for repeated multi-site auditing. A verdict that is stable under bootstrap of a single cohort may still be underpowered for small subgroups or flip across hospital sites. As shown in Figure 1, each (model, metric, attribute) cell is evaluated on three complementary axes: resampling stability (VFR), audit-size sensitivity (𝑁 ∗ ), and cross-hospital agreement (𝜅), producing a four-tier reliability classification.

3.3

Verdict Flip Rate

Let D (𝑘 ) denote the 𝑘-th stratified bootstrap resample of size 𝑁 drawn from the audit cohort with replacement, preserving the joint (label × 𝑎) distribution. For a fixed cell 𝑐 = (𝑓 , 𝑚, 𝑎), let 𝑣𝑘 = 𝑣 (𝑓 , D (𝑘 ) , 𝑚, 𝑎) ∈ {0, 1} denote the per-resample verdict and

3.3.2 Selecting the per-resample size 𝑁 . Audit size controls the trade-off between sampling noise and sensitivity to threshold-edge cells. For a fairness metric 𝑚 with population value 𝑚 pop , threshold 𝜏, and margin 𝛿 = |𝑚 pop − 𝜏 |, a verdict flips when |𝑚ˆ 𝑁 − 𝑚 pop | > 𝛿, where 𝑚ˆ 𝑁 is the metric estimate and 𝜎𝑁 its per-resample standard deviation. Smaller 𝑁 inflates 𝜎𝑁 , since the audit loses discriminative power because cells far from threshold also flip on sampling noise. Larger 𝑁 deflates 𝜎𝑁 , since the audit loses sensitivity to thresholdedge cells, which are the cells most informative for governance. We use 𝑁 = 104 within the prespecified 5 × 103 to 5 × 104 single-site audit-size design range. On this cohort, 𝑁 = 104 is the smallest tested size that separates threshold-edge cells (VFR > 0.10) from farfrom-threshold cells (VFR ≈ 0) without excessive sampling noise. Axis 2 independently sweeps audit size to estimate the minimum reliable 𝑁 ∗ . 3.3.3 Model-development and audit protocol. The model-training stage and the audit-reliability stage are treated as separate steps. Predictive models are trained on the training partition and evaluated on the held-out audit partition. The VFR-Audit procedure is then applied to the fixed trained model and the fixed prediction scores to quantify the reliability of thresholded fairness verdicts under resampling, audit-size variation, and hospital-grouped evaluation. The reported VFR values are reliability estimates for the

VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

3.4

Three-Axis Reliability Framework

pt ed

in

Each axis answers one reliability question and is formalised by one of Algorithms 1 to 3. Axes 1 and 2 operate per (model, metric, attribute) cell, while Axis 3 operates per fairness metric across that metric's protected-attribute cells. Axis 1 (Algorithm 1) asks would the same verdict have been reached on a different audit cohort?; it draws 𝐾 = 500 stratified bootstrap resamples and reports VFR via c Equation 1 together with the stability margin 𝜎 = |𝑚ˆ − 𝜏𝑚 |/SD(𝑚). Axis 2 (Algorithm 2) asks how many records does an audit need before the metric stabilises?; it sweeps an audit-size grid with 𝑅 = 30 random sub-samples per point and reports 𝑁 ∗ = min{𝑁𝑖 : CV(𝑁𝑖 ) < 0.05}, flagging cells whose 𝑁 ∗ exceeds 𝑁 field = 5×104 as not reliably auditable at a single hospital site. Axis 3 (Algorithm 3) asks will the audit verdict reached at one hospital recur at another?; it evaluates verdict agreement across 𝐾hosp = 20 hospital-grouped GroupKFold folds. For each fairness metric, the four protected-attribute cells form the rated items and the folds act as raters, so Fleiss’ 𝜅 [24] summarises agreement over this fold-by-cell verdict matrix, classified using the Landis-Koch [37] bands. The metric-level 𝜅 is assigned to each constituent (metric, attribute) cell for the tier computation, because a single cell supplies only one rated item and would render Fleiss’ 𝜅 degenerate. Algorithm 1 Axis 1: Verdict Flip Rate (Resampling Stability).

Ac ce

Require: Trained model 𝑓 , audit cohort D, metric 𝑚 with threshold 𝜏𝑚 , attribute 𝑎, bootstrap count 𝐾 = 500, resample size 𝑁 . Ensure: VFR, dominant verdict 𝑣¯, stability margin 𝜎, 95 % bootstrap CI. 1: 𝑛 pass ← 0; 𝑀 ← [ ] 2: for 𝑘 = 1 to 𝐾 do 3: D𝑘 ← StratifiedBootstrap(D, 𝑁 , strata=𝑦 × 𝑎) 4: 𝑚ˆ 𝑘 ← 𝑚(𝑓 , D𝑘 , 𝑎); 𝑣𝑘 ← 1[𝑚ˆ 𝑘 ⪰ 𝜏𝑚 ] 5: 𝑛 pass ← 𝑛 pass + 𝑣𝑘 ; 𝑀.append(𝑚ˆ 𝑘 ) 6: end for 7: VFR ← min(𝑛 pass , 𝐾 − 𝑛 pass )/𝐾 8: 𝑣¯ ← Pass if 𝑛 pass > 𝐾/2 else Fail 9: 𝜎 ← |mean(𝑀) − 𝜏𝑚 |/SD(𝑀) 10: return (VFR, 𝑣¯, 𝜎, CI95 (𝑀))

20 26

Require: 𝑓 , D, 𝑚, 𝑎, audit-size grid {𝑁 1, . . . , 𝑁 max }, repetitions 𝑅 = 30, CV cutoff 𝜂 = 0.05. Ensure: CV curve, minimum reliable size 𝑁 ∗ . 1: for each 𝑁𝑖 in audit-size grid do 2: 𝑉𝑖 ← [𝑚(𝑓 , RandomSubsample(D, 𝑁𝑖 ), 𝑎) : 𝑟 = 1 . . . 𝑅] 3: CV𝑖 ← SD(𝑉𝑖 )/|mean(𝑉𝑖 )| 4: end for 5: 𝑁 ∗ ← min{𝑁𝑖 : CV𝑖 < 𝜂} (or ∞ if no grid point qualifies) 6: return ({CV𝑖 }, 𝑁 ∗ ) Algorithm 3 Axis 3: Cross-Hospital Verdict Agreement (𝜅). Require: 𝑓 , D with hospital identifiers 𝑠, metric 𝑚, attribute set A, 𝜏𝑚 , fold count 𝐾hosp = 20. Ensure: Fold-by-cell verdict matrix 𝑉 , metric-level Fleiss’ 𝜅𝑚 . 1: {D1 , . . . , D𝐾hosp } ← GroupKFold(D, groups=𝑠, 𝐾hosp ) 2: for each fold ℎ ∈ {1, . . . , 𝐾hosp } do 3: for each attribute 𝑎 ∈ A do 4: 𝑉𝑎,ℎ ← 1[𝑚(𝑓 , Dℎ , 𝑎) ⪰ 𝜏𝑚 ] 5: end for 6: end for 7: 𝜅𝑚 ← FleissKappa(𝑉 ) ⊲ items: |A| cells; raters: 𝐾hosp folds 8: 𝜅 (𝑐) ← 𝜅𝑚 for every cell 𝑐 = (𝑓 , 𝑚, 𝑎) with 𝑎 ∈ A 9: return (𝑉 , 𝜅𝑚 )

KM

3.3.4 Relation to bootstrap confidence intervals. VFR does not replace bootstrap confidence intervals on continuous fairness metrics. A confidence interval describes uncertainty in the continuous metˆ VFR reports the empirical instability of the governanceric value 𝑚; facing binary verdict 1[𝑚ˆ ⪰ 𝜏𝑚 ] after thresholding. Both are useful and they answer different questions. For ratio metrics such as DI, the bootstrap distribution is right-skewed near zero, so a normalapproximation flip probability tends to underestimate the empirical reversal rate near the threshold. VFR uses the empirical resample proportion directly.

Algorithm 2 Axis 2: Audit-Size Sensitivity (𝑁 ∗ ).

CI

specified audit cohort and protocol rather than prospective guarantees. Robustness checks in Section 4.2.5 use a stricter 70/15/15 train, validation, audit split in which intervention thresholds are selected on the validation partition and final statistics are reported on a previously unseen audit partition.

CIKM ’26, November 7–11, 2026, Rome, Italy

Metric-specific pass rule. For each metric 𝑚, the relation ⪰ used in Algorithm 1 denotes the metric-specific pass condition. For ratiostyle metrics such as Disparate Impact, 𝑚ˆ ⪰ 𝜏𝑚 means 𝑚ˆ ≥ 𝜏𝑚 . For two-sided difference-style metrics such as SPD, EOpp, EOD, and PP, ˆ ≤ 𝜏𝑚 . For one-sided metrics such as Calibration 𝑚ˆ ⪰ 𝜏𝑚 means |𝑚| and Theil Index, 𝑚ˆ ⪰ 𝜏𝑚 means 𝑚ˆ ≤ 𝜏𝑚 . The tie case 𝑛 pass = 𝐾/2 assigns Fail as the dominant verdict, since any audit-grade verdict requires a strict majority of resamples to support a pass. Combined reliability tier. A cell 𝑐 is classified Practical-Stability when all three conditions hold, namely VFR(𝑐) ≤ 0.10, 𝑁 ∗ (𝑐) ≤ 𝑁 field , and 𝜅 (𝑐) ≥ 0.40. A cell is classified Caution-Required if one condition is violated, High-Variance if two, and CatastrophicInstability if all three. The classification is conjunctive across the three axes, so any single-axis failure drops the cell to a lower tier. Operationally, a High-Variance cell should trigger additional data accumulation or cross-site pooling and manual governance review before automated certification; Catastrophic-Instability should defer deployment-level certification until reliability improves.

4 Experiment and Evaluation 4.1 Experimental Setting This subsection specifies the dataset, the trained model panel, the fairness metrics and protected attributes, the baselines used for comparison, and the dataset exploratory analysis. Reproducibility. Code and experiment notebooks are available at https://github.com/MdJoy31/VFR-Framework. 4.1.1 Dataset. The Texas-100X dataset comprises 925,128 inpatient discharge records from 441 hospitals in Texas. Records were drawn from the THCIC Public Use Data File for fiscal year 2006 (Quarters 1 to 4) [63]. The binary prediction target is LOS exceeding three days

Md Jannatul Rakib Joy, Viet Vo, and Caslon Chua

in

Ac ce

pt ed

4.1.3 Fairness metrics and protected attributes. Seven group-fairness metrics were evaluated [5, 66], namely Disparate Impact (DI, ≥ 0.80), Statistical Parity Difference (SPD, | · | ≤ 0.10), Equal Opportunity (EOpp, | · | ≤ 0.10), Equalised Odds (EOD, | · | ≤ 0.10), Theil Index (TI, ≤ 0.10), Predictive Parity (PP, | · | ≤ 0.10), and Calibration (Cal, | · | ≤ 0.05). Four protected attributes were evaluated, namely race (5-category THCIC code comprising White, Black, Asian/Pacific Islander, American Indian, and Other/Unknown), sex (Male/Female), ethnicity (Hispanic/non-Hispanic), and age (4-bucket THCIC PUDF scheme comprising Pediatric < 18, Young Adult 18 to 39, MiddleAged 40 to 64, and Elderly ≥ 65). The audit grid contains 7 × 4 = 28 (metric, attribute) cells per model, or 336 cells across all twelve models per audit pass. 4.1.4 Baselines. Four baselines are compared on the same audit instrument so that cross-baseline differences in VFR, 𝑁 ∗ , and Fleiss’ 𝜅 reflect the intervention rather than the audit setup. The dataset, train-test split, feature pipeline, trained model, bootstrap seed, and hospital-fold partition are held constant across all four baselines. • Baseline 1 (B1) Real-only model. No fairness intervention. Standard reference for the unintervened audit. • Baseline 2 (B2) Reweighing baseline. Intersectional adaptation of the Kamiran-Calders pre-processing reweighing strategy [35], evaluated as a baseline in fairness-aware clinical ML [50], controlled by a 𝜆 regularisation strength.

KM

4.1.2 Models. Twelve standard tabular classifiers were trained on identical pre-processed features: logistic regression, decision tree, random forest, gradient boosting, AdaBoost, XGBoost, LightGBM, CatBoost, histogram gradient boosting, bagging, extra trees, and a stacking ensemble. Each classifier was fitted with a fixed, modelspecific hyperparameter configuration, archived in the experiment records and frozen before fairness auditing, preventing fairness outcomes from influencing model configuration. Deep neural baselines were not included because Texas-100X is structured administrative tabular data and the study targets audit reliability rather than sequence-representation learning. XGBoost was designated in advance as the canonical model; on the held-out test partition it achieved AUROC = 0.9528, accuracy = 0.8776, and 𝐹 1 = 0.8627.

Figure 2: Texas-100X dataset composition by protected attribute. Panel (c) reports the largest within-attribute baserate gap among the four protected attributes. The positive rate triples between Young Adult (20.7 %) and Elderly (60.6 %) strata, fixing an empirical lower bound on Age-DI. • Baseline 3 (B3) Threshold-shifting baseline. Per-cell 𝛼search over SR/TPR/PPV on predicted probabilities, adapted from the post-processing equalised-odds method of Hardt et al. [29] and the per-cell threshold-optimisation strategy of Abakasanga et al. [1]. • VFR-Audit (B4). The proposed design. 𝛼-search followed by greedy refinement under the all-four-DI ≥ 0.80 constraint, then audits the resulting verdicts using VFR, audit-size sensitivity, and cross-hospital agreement.

CI

(positive class rate 45.0 %). The dataset was split 80/20 stratified by the LOS-greater-than-three-days target with random seed 42, producing 740,102 training records and 185,026 test records. Administrative discharge data are appropriate for this study because LOS prediction is directly tied to hospital operations, bed planning, and discharge management [32, 33, 51]. Unlike ICU-specific EHR benchmarks such as MIMIC-III or MIMIC-IV [34, 46], the Texas100X dataset provides a state-wide, multi-hospital administrative setting with 441 hospitals, which is better aligned with the crosshospital deployment question studied here. External replication on richer EHR datasets remains necessary. Intersectional sparsity constrains the audit design: the primary 28-cell VFR grid therefore evaluates protected attributes separately, while the race × age × sex intersection is used only for the reweighing baseline. Given that the smallest race stratum contains 3,474 records in the full cohort (Table 3), we do not claim reliable bootstrap inference for finer intersections; Axis 2 instead identifies cells whose required audit size exceeds available single-site support.

20 26

CIKM ’26, November 7–11, 2026, Rome, Italy

VFR-Audit implementation. VFR-Audit starts from the Baseline 3 threshold-shifting solution and greedily adjusts candidate groupspecific decision thresholds under the hard all-four-DI ≥ 0.80 constraint. At each step, a candidate move is accepted only if it preserves the DI constraint on all four protected attributes and improves the selected VFR objective on the audited cell. Ties are resolved by higher accuracy within the intervention-selection partition. The stricter validation-audit robustness check in Section 4.2.5 separates threshold selection from final audit reporting. The selected configuration is then evaluated, post-hoc, with the same audit instrument used for Baselines 1 to 3, namely Verdict Flip Rate, audit-size sensitivity, and hospital-fold agreement. Only VFR-Audit is original to this study. Baselines 1 to 3 are direct adoptions of earlier work. Cross-study comparison against published LOS work was not undertaken because no prior LOS study reports a verdict-stability measure or audit-instrument reliability statistic [32, 33]. 4.1.5 Dataset exploratory analysis. Three properties of the cohort matter for the audit-reliability claims that follow (Figure 2, Table 3). The Age axis carries the largest base-rate gap (0.399) and the Race axis the second largest (0.189). Disparate Impact is upper-bounded by a function of the base-rate gap, so Age is the binding constraint

VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

N

%

Pos. rate

603,368 65.2 186,670 20.2 115,212 12.5 16,404 1.8 3,474 0.4

0.453 0.404 0.523 0.410 0.334

Sex

Male Female

585,840 339,288

63.3 36.7

0.411 0.518

Hispanic Non-Hispanic

670,586 254,542

72.5 27.5

0.471 0.397

Pediatric (< 18) 38,121 4.1 Young Adult (18 to 39) 208,528 22.5 BRG = 0.399 Middle-Aged (40 to 64) 281,409 30.4 Elderly ( ≥ 65) 397,070 42.9

0.403 0.207 0.418 0.606

BRG = 0.107

Ethnicity BRG = 0.074

Age group

Hospital volume

Top 10 Top 50 Top 100 Top 200 Median / site

124,892 13.5 430,184 46.5 652,215 70.5 862,944 93.3 686 n/r

n/r n/r n/r n/r n/r

4.2

Evaluation Questions and Results

in

for any DI-targeted intervention on this cohort, with Race a secondary constraint. The cohort’s race-by-ethnicity coding is nonstandard. 99.4 % of records coded as Black are also coded as Hispanic [63], so the audit reports parity between THCIC race codes as recorded. The hospital-volume distribution is heavily right-skewed (top 50 hospitals hold 46.5 %, top 100 hold 70.5 %). The GroupKFold cross-hospital evaluation at 𝐾hosp = 20 produces folds of approximately 4.6 × 104 records each, well within the field-realistic single-site audit range.

Ac ce

pt ed

Results are organised around five evaluation questions. EQ1. Does the administrative LOS dataset provide sufficient predictive utility for downstream fairness auditing? EQ2. Which fairness intervention satisfies the four-fifths rule, and what accuracy cost does it introduce relative to the Real-only reference? EQ3. Does VFR-Audit reveal verdict instability that point-estimate audits miss? EQ4. What is the predictive-performance, computational, and labour cost of adopting VFR-Audit? EQ5. Are the conclusions stable under alternative splits, feature ablation, and random seeds? 4.2.1 EQ1. Predictive Utility. On the held-out test partition (𝑁 = 1.85 × 105 ), the XGBoost model (Baseline 1) achieves AUROC = 0.9528, accuracy = 0.8776, and 𝐹 1 = 0.8627 before any fairness intervention. This value should be interpreted as the performance of a retrospective administrative audit model. Leakage-sensitivity and validation-audit robustness checks are reported separately in Section 4.2.5. Table 4 reports the full twelve-classifier baseline-audit panel. Predictive performance is consistent with prior LOS studies on EHR datasets [51, 58, 70], which confirms that administrative discharge data carry sufficient signal at the cohort scale required by the cross-hospital audit.

Model

Acc

AUROC

F1

Race

Sex

Eth

Age

Fair/28

XGBoost (canon.) 0.878 LightGBM 0.865 HistGradientBoosting 0.858 Stacking Ensemble 0.853 Random Forest 0.853 Bagging 0.857 CatBoost 0.847 Gradient Boosting 0.838 Decision Tree 0.840 AdaBoost 0.814 Extra Trees 0.799 Logistic Regression 0.801

0.953 0.943 0.939 0.935 0.934 0.934 0.929 0.922 0.920 0.897 0.887 0.881

0.863 0.847 0.840 0.834 0.834 0.842 0.826 0.817 0.816 0.794 0.776 0.775

0.644 0.595 0.603 0.619 0.587 0.676 0.579 0.617 0.609 0.654 0.587 0.534

0.763 0.752 0.746 0.744 0.742 0.764 0.730 0.719 0.742 0.693 0.726 0.684

0.831 0.821 0.820 0.821 0.810 0.831 0.806 0.796 0.825 0.777 0.768 0.757

0.299 0.264 0.256 0.258 0.240 0.303 0.232 0.218 0.270 0.227 0.162 0.184

20 20 20 19 18 19 18 16 18 10 13 15

20 26

Stratum

White Other / Unknown Race Black BRG = 0.189 Asian / Pacific Islander American Indian

Table 5: Per-attribute Disparate Impact across the four baselines. Bold values satisfy DI ≥ 0.80. Baseline

Race

Sex

Eth

Age

Pass

B1 Real-only B2 Reweighing (𝜆=2) B3 Threshold-shifting B4 VFR-Audit

0.644 0.575 0.834 0.801

0.763 0.749 0.968 0.932

0.831 0.821 0.994 1.000

0.299 0.264 0.812 0.800

1/4 1/4 4/4 4/4

KM

Attribute

Table 4: Per-model audit panel on the Baseline 1 Real-only reference. “Fair/28” is the count of (metric, attribute) cells passing the operational threshold.

4.2.2 EQ2. Fairness-Intervention Comparison. Table 5 reports perattribute Disparate Impact movement across the four baselines. Baseline 1 (Real-only) and Baseline 2 (Reweighing) satisfy the fourfifths rule on only 1 of 4 protected attributes. Baseline 3 (Thresholdshifting) and VFR-Audit (B4) satisfy the four-fifths rule on 4 of 4 protected attributes. The Race-DI value moves from 0.644 in Baseline 1 to 0.575 in Baseline 2, a movement away from the 0.80 threshold rather than toward it. An extreme-𝜆 sweep on twenty further reweighing configurations achieves a best Age-DI of 0.283, well below the 0.80 threshold. The Reweighing baseline therefore does not satisfy the four-fifths rule on this dataset regardless of the regularisation strength applied. The panel in Table 4 exposes wide cross-model variation in the baseline fairness profile. All twelve models fail the four-fifths rule on the Age axis (DIAge ∈ [0.16, 0.30]), and none of the twelve satisfies all four DI constraints simultaneously without intervention. The VFR-Audit intervention applied to the canonical XGBoost model raises Race-axis DI from 0.644 to 0.801, Sex-axis from 0.763 to 0.932, Ethnicity-axis from 0.831 to 1.000, and Age-axis from 0.299 to 0.800. The DI Ethnicity = 1.000 value is an algorithmic artefact discussed in the Q3 analysis. Accuracy cost is reported in percentage points relative to the Baseline 1 Real-only reference. For a method 𝑏, the accuracy cost is

CI

Table 3: Texas-100X cohort descriptive statistics on the full 925,128-record cohort. “Pos. rate” is the within-stratum proportion with LOS > 3 days; “BRG” is the base-rate gap.

CIKM ’26, November 7–11, 2026, Rome, Italy

ΔAcc𝑏 = 100 × (Acc𝐵1 − Acc𝑏 ). Using Table 6, Reweighing loses 1.98 percentage points, Thresholdshifting loses 4.60 percentage points, and VFR-Audit loses 4.24 percentage points relative to Baseline 1. The comparison between Threshold-shifting (B3) and VFR-Audit is the substantive one, because both satisfy the all-four-DI criterion on the point estimate. Threshold-shifting reaches DI pass at accuracy 0.8316 with mean VFR 0.0863 and maximum VFR 0.490. VFR-Audit reaches DI pass at higher accuracy 0.8352 with lower mean VFR 0.0809 and lower maximum VFR 0.476. The gain of VFR-Audit over B3 is therefore not a new point-estimate fairness pass, since B3 already passes DI.

CIKM ’26, November 7–11, 2026, Rome, Italy

Md Jannatul Rakib Joy, Viet Vo, and Caslon Chua

Finding: Point-estimate fairness does not imply verdict reliability. On B1, 146/336 audit cells flip at least once and 77/336 exceed VFR > 0.10. After B4 satisfies all four DI point-estimate thresholds, 11/28 cells still flip under resampling.

pt ed

in

Axis 2 audit-size sensitivity. The CV < 5 % cutoff is reached at field-realistic audit sizes for 11 of 28 (metric, attribute) cells on VFRAudit. It requires the full 185,026-record test partition for 8 cells, comprising Calibration (4 of 4 attributes) and Equal Opportunity / Equalised Odds for race and ethnicity. A single hospital running a quarterly audit of approximately 5 × 103 records cannot reliably audit those eight cells without pooling records across multiple quarters or sites. Of the 28 B4 cells, 17 exceed CV = 50 % at 𝑁 = 103 , so small-cohort audits at this size are dominated by sampling noise rather than model behaviour.

20 26

Ac ce

Axis 3 cross-hospital agreement. Fleiss’ 𝜅 at 𝐾hosp = 20 on VFRAudit is moderate for EOpp (𝜅 = 0.465) and EOD (𝜅 = 0.587). It reaches the degenerate ceiling for the Theil index (𝜅 = 1.000, because the metric satisfies the threshold on every fold). It is slight or below-chance for DI (𝜅 = −0.035), SPD (𝜅 = 0.117), PP (𝜅 = 0.315), and Cal (𝜅 = 0.003). Fleiss’ 𝜅 is used here as a descriptive summary of hospital-fold verdict agreement under the grouped split protocol and should not be interpreted as a formal causal transportability estimate. The pattern is consistent across baselines. Equal-opportunity-style metrics show stronger cross-hospital verdict agreement, whereas calibration does not. A fairness verdict on calibration reached at one hospital, where 𝜅 Cal = 0.003, carries no measurable predictive value for the calibration verdict reached on another hospital’s records. Achieving DI pass costs cross-site verdict stability. Mean 𝜅 drops from approximately 0.44 on Baselines 1 and 2 to approximately 0.35 on Baseline 3 and VFR-Audit once threshold-shifting engages, because per-cell 𝛼-search fits to the audited cohort’s intersectional cell structure, and that structure varies across hospital folds.

Root cause of the B3 to VFR-Audit VFR movement. B3 achieves all-four-DI pass on the full test partition. Under 𝐾 = 500 stratified bootstrap, the Race-DI and Age-DI verdicts in B3 are unstable. Their VFRs are 0.41 and 0.40 respectively, so approximately 41 per cent and 40 per cent of bootstrap audits disagree with the dominant verdict for these cells. The greedy-refinement step in VFR-Audit walks the per-cell thresholds inward from the 𝛼-search starting point under the all-four-DI ≥ 0.80 constraint. On the four bindingconstraint cells, the resulting B3 to VFR-Audit VFR movements are DI Age 0.412 → 0.232 (−0.180), SPD Age 0.490 → 0.292 (−0.198), DI Race 0.410 → 0.476 (+0.066), and SPD Race 0.398 → 0.470 (+0.072). The movement shows a redistribution of instability across protected attributes rather than a uniform improvement across all cells. Age-axis VFR decreases for DI and SPD because the greedy refinement moves the Age thresholds farther from the DI boundary while preserving the all-four-DI constraint. Race-axis VFR increases for DI and SPD because the same refinement shifts part of the threshold burden from Age to Race. This is why the aggregate maximum VFR decreases from 0.490 to 0.476 while Race-DI and Race-SPD become less stable individually. The result demonstrates that VFR-Audit exposes where an intervention transfers instability, rather than reporting only whether the final DI point estimate passes. Reporting only the 6.3 % relative reduction in mean VFR would conceal this redistribution pattern.

KM

4.2.3 EQ3. Verdict-Reliability Analysis. Under 𝐾 = 500 stratified bootstrap on the cross-model audit grid (12 models × 7 metrics × 4 attributes = 336 cells), 259 of 336 cells (77.1 %) satisfy the operational stability cutoff VFR ≤ 0.10 on Baseline 1, while 146 of 336 reverse their verdict at least once. Race-axis cells exhibit the highest instability. EOpprace reaches VFR = 0.480, EODrace reaches VFR = 0.386, and PPrace reaches VFR = 0.458. Age-axis cells concentrate around the PP instability point (VFR = 0.202). After VFR-Audit satisfies the four-fifths rule on every protected attribute on the point estimate, 11 of 28 B4 cells still register VFR > 0 under the resampling protocol. Cohort-wide mean VFR on B4 is 0.081 (versus 0.066 for B1) and the maximum is 0.476 (versus 0.498). Race-axis cells concentrate the residual instability, while Sex, Ethnicity, and the remaining Age-axis cells reach VFR ≤ 0.10. Under the stricter operational cutoff VFR ≤ 0.10, 21 of 28 B4 cells are stable, which matches the single-split row of Table 7. The percell VFR landscape across the three independent robustness reruns is reported in Figure 3 and analysed in EQ5.

satisfy the all-four-DI criterion on the point estimate. VFR-Audit reduces the mean VFR from 0.0863 to 0.0809 (−6.3 % relative) and the maximum VFR from 0.490 to 0.476 (a reduction of 0.014 VFR units, approximately 2.9 % relative), while raising accuracy from 0.8316 to 0.8352 (+0.36 percentage points). Baselines 1 and 2 produce nearly identical mean VFR (0.066 versus 0.062) and mean cross-site 𝜅 (0.437 versus 0.436), so Reweighing does not change the verdict-reliability profile relative to no intervention on this cohort.

CI

The gain is a verdict-reliability improvement delivered together with a +0.36 percentage-point accuracy improvement versus B3.

4.2.4 EQ4. Adoption Cost. Adopting VFR-Audit involves three cost components. Predictive-performance cost. VFR-Audit reduces accuracy by 4.24 pp (0.8776 → 0.8352) and 𝐹 1 by 4.64 pp (0.8627 → 0.8163), while preserving AUROC at 0.9528. AUROC preservation is mechanically expected under threshold shifting because the score ranking is unchanged. Computational cost. Algorithm 1 runs in 𝑂 (𝐾𝑁 𝑀 A ) time and 𝑂 (𝑁 + 𝐾𝑀 A ) memory, where 𝑀 A = |F | · |A|. The audit scales linearly with bootstrap samples, audit records, fairness metrics, and protected attributes. With prediction scores precomputed, the marginal cost over a bootstrap-CI audit is converting each resampled metric value into a pass-or-fail verdict and maintaining pass and fail counts. Labour cost. Each cell receives a dominant verdict, a VFR score, a minimum reliable audit size, a cross-hospital agreement value, and a reliability tier, so manual interpretation is bounded by the per-cell quintuple rather than by reading bootstrap CIs. Prior LOS fairness studies do not report a comparable verdict-reliability cost.

Four-baseline comparison summary. Table 6 reports the fourbaseline audit on the same instrument. The main reliability difference is between Threshold-shifting (B3) and VFR-Audit. Both

Accuracy and fairness trade-off summary. Across the twelveclassifier panel in Table 4, Race-axis threshold shifting moves every classifier above the four-fifths rule at an accuracy cost in the 4 to 6 percentage-point band. On the canonical model, 𝛼-search alone reaches the all-four-DI pass on the point estimate, but the Age-axis

CIKM ’26, November 7–11, 2026, Rome, Italy

20 26

VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

Figure 3: VFR heatmaps across three robustness protocols on the audit partition.

KM

Table 6: Four-method audit comparison under the same audit instrument. Baseline

Acc

AUROC

VFR

VFR max

#flipped

𝜅

𝜅 EOpp

𝜅 EOD

DI pass

B1 Real-only B2 Reweighing (𝜆=2) B3 Threshold-shifting B4 VFR-Audit

0.8776 0.8578 0.8316 0.8352

0.9528 0.9373 0.9532 0.9528

0.066 0.062 0.086 0.081

0.498 0.426 0.490 0.476

12 10 11 11

0.437 0.436 0.346 0.350

0.588 0.618 0.606 0.465

0.595 0.607 0.607 0.587

1/4 1/4 4/4 4/4

in

CI

on the XGBoost VFR-Audit configuration raises the accuracy cost monotonically from 4.24 to 5.97 percentage points. The robustness checks preserve the main conclusion. All four protocols satisfy the all-four-DI criterion, and 21 to 24 of 28 cells remain VFR-stable. The admission-lean protocol produces the lowest AUROC (0.8567) because it removes TOTAL_CHARGES and PAT_STATUS, two end-of-encounter variables with strong predictive signal. Relative to the leakage-controlled full-feature baseline (0.9418), the drop is 0.0851 on the [0, 1] AUROC scale. Despite this predictive-performance loss, the fairness-verdict stability pattern remains similar to the single-split result (24/28 VFR-stable cells under admission-lean versus 21/28 under single-split). This indicates that the VFR pattern is not solely an artefact of leakagesensitive features. The stratified 70/15/15 protocol selects intervention thresholds on the validation partition and reports VFR, 𝜅, and DI on an audit partition never seen during tuning. AUROC moves to 0.9418 at an accuracy cost of 4.98 pp, consistent with the primary single-split result while using a stricter validation-audit protocol. All-four-DI continues to pass, and 21/28 cells remain VFR-stable. The independent-seed check reproduces the leakage-controlled result within stochastic noise (AUROC 0.9426, accuracy cost 5.04 pp, 21/28 VFR-stable). Together, these checks reduce concern that the main VFR-Audit conclusions are artefacts of a single classifier, a single split, threshold tuning, or end-of-encounter variables.

pt ed

Figure 4: Per-hospital-fold metric distribution across the 20 THCIC hospital folds. Equal Opportunity and Equalised Odds show tight inter-fold agreement; Disparate Impact and Calibration show wide inter-fold spread. Table 7: Robustness checks for the VFR-Audit pipeline. Protocol

AUROC

Acc cost

all-4-DI

VFR-stable

0.9528 0.9418 0.8567 0.9426

4.24 pp 4.98 pp 5.02 pp 5.04 pp

✓ ✓ ✓ ✓

21/28 21/28 24/28 21/28

Ac ce

Single-split main (canonical) Stratified 70/15/15 (seed 42) Admission-lean (no TC, no PS) Independent seed (seed 123)

and Race-axis DI verdicts reverse on approximately 41 per cent of bootstrap resamples. The VFR-Audit greedy refinement reduces aggregate verdict instability while preserving the all-four-DI pointestimate pass, although the Race-DI and Age-DI cells remain above the VFR ≤ 0.10 stability cutoff. 4.2.5 EQ5. Robustness Checks. Table 7 reports four protocol-level audit-reliability checks against the single-split main result. As a cross-model check, the VFR-Audit intervention is applied to four model families (XGBoost, LightGBM, Random Forest, Logistic Regression). All four satisfy the four-fifths rule on every protected attribute post-intervention at an accuracy cost in the 4.24 to 5.39 percentage-point band. Sweeping the DI target from 0.80 to 0.90

CIKM ’26, November 7–11, 2026, Rome, Italy

Md Jannatul Rakib Joy, Viet Vo, and Caslon Chua

in

pt ed

Ac ce

20 26

Discussion

KM

5

VFR-Audit separates fairness status from verdict reliability by jointly reporting the dominant verdict, VFR, minimum reliable audit size, and cross-hospital agreement. The results show that a point-estimate pass can remain unstable under resampling or site shift. Calibration is a persistent failure mode on this cohort (VFRCal ≈ 0 with a consistent Fail verdict and 𝜅 Cal ≈ 0) and should therefore be reported by site rather than only in pooled form. The B3-versus-VFR-Audit comparison is the substantive costbenefit decision in this paper. Both methods pass the four-fifths rule on every protected attribute on the point estimate. Accuracy cost is reported relative to Baseline 1 using ΔAcc𝑏 = 100 × (Acc𝐵1 − Acc𝑏 ). From Table 6, with Acc𝐵1 = 0.8776, Reweighing pays 1.98 pp, Threshold-shifting pays 4.60 pp, and VFR-Audit pays 4.24 pp, so VFR-Audit raises the same DI pass at 0.36 pp lower cost than B3. Under 𝐾 = 500 stratified bootstrap, the Race-DI and Age-DI verdicts in B3 carry VFRs of 0.41 and 0.40, so 40 to 41 per cent of bootstrap audits disagree with the dominant verdict on those cells. The greedy-refinement step walks the per-cell thresholds inward from the 𝛼-search start under the all-four-DI ≥ 0.80 constraint, producing per-cell VFR movements DIAge 0.412 → 0.232 (−0.180), SPDAge 0.490 → 0.292 (−0.198), DIRace 0.410 → 0.476 (+0.066), and SPDRace 0.398 → 0.470 (+0.072). The Age-axis VFR drops because the refinement opens headroom between Age decision thresholds and the DI boundary. The Race-axis VFR rises because the refinement transfers part of the threshold slack from Age to Race to maintain the all-four-DI constraint. The aggregate is a maximumVFR reduction from 0.490 to 0.476 and a mean-VFR reduction from 0.0863 to 0.0809 (6.3 % relative). Reporting only the aggregate would conceal this redistribution pattern. Reweighing fails on this dataset because the Age base-rate gap acts as a feature-based floor on the achievable selection-rate ratio. With Elderly LOS > 3 d rate 0.606 against 0.207 for young adults (a 2.93× ratio), Reweighing modifies the empirical risk the optimiser minimises but leaves the input features unchanged [35]. The selection-rate ratio defining DI is then floored by the feature-based predictability gap, which explains why Race-DI moves from 0.644 to 0.575 between Baselines 1 and 2 (away from the threshold) and why no 𝜆 setting in the sweep satisfies the four-fifths rule. Per-cell threshold shifting bypasses this constraint by operating on the operating point per group rather than on the empirical risk, and satisfies all four DI constraints at the costs reported above. The audit-reliability conclusions persist under both the stricter validation-audit protocol and the admission-lean sensitivity check.

CI

Figure 5: Validation-protocol comparison on the Texas-100X audit cohort. (a) AUROC. (b) Accuracy cost of the VFR-Audit intervention. (c) Minimum Disparate Impact across the four protected attributes. All four protocols satisfy the four-fifths rule on every protected attribute.

Under the leakage-controlled 70/15/15 split, all-four-DI continues to pass on the audit slice, the VFR-stable cell count remains 21/28, and AUROC moves to 0.9418 at an accuracy cost of 4.98 pp. Removing the two end-of-encounter features TOTAL_CHARGES and PAT_STATUS drops AUROC from the leakage-controlled fullfeature baseline of 0.9418 to 0.8567 (a leakage-diagnostic score of 0.0851 on the [0, 1] scale) but preserves all-four-DI and yields 24/28 VFR-stable cells, so the fairness-audit reliability story is preserved under the admission-lean feature set on this cohort. Texas100X is, however, a retrospective administrative discharge dataset, and this leakage-sensitivity analysis does not establish prospective admission-time deployability. The protected-attribute results should also be read with two caveats. The THCIC race-by-ethnicity coding is non-standard (Table 3), so race and ethnicity results reflect administrative parity rather than self-identified parity, and only Disparate Impact admits a regulatory threshold (the four-fifths rule). Counterfactual and individual-fairness instruments are not evaluated. The primary single-split experiment should therefore be interpreted as an intervention-analysis setting, while the 70/15/15 protocol provides the cleaner validation-audit evidence. Future work. VFR-Audit should be validated prospectively on admission-time features and temporally held-out hospitals, and extended to sparse intersectional groups using hierarchical pooling or adaptive minimum-support rules. High-Variance cells should trigger additional data accumulation or cross-site pooling and governance review before automated certification. For high-speed clinical deployment, a streaming implementation could recompute Axis 1 over rolling audit windows from cached prediction scores, while the more expensive audit-size and cross-hospital axes run asynchronously at lower cadence, enabling near real-time equity monitoring without retraining the predictive model after every batch. Sensitivity to alternative 𝜏VFR values should also be evaluated rather than treating 0.10 as universal. Future work will extend VFR-Audit to additional datasets, protected attributes, deep neural networks, and LLM-derived clinical systems, while evaluating distributed cross-hospital deployment, including federated learning, and AI-agent-assisted continuous fairness auditing.

6

Conclusion

VFR-Audit provides a model-agnostic audit layer for assessing whether thresholded fairness verdicts remain reliable under cohort resampling, audit-size variation, and hospital shift. On Texas-100X, point-estimate fairness can remain unstable even after intervention satisfies the four-fifths rule, demonstrating that deployment decisions should report audit reliability alongside fairness status. Robustness checks under a stricter validation-audit split, admissionlean features, and an independent seed support this conclusion. GenAI Usage Disclosure. Generative AI tools were used solely for grammar checking. No content in this manuscript was written or generated by AI. All text, technical content, experiments, analysis, and interpretations were authored and verified by the authors.

VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

Ac ce

pt ed

in

20 26

in Machine Learning Models for Hospitalization and Emergency Department Visit Risk Prediction in Home Healthcare Patients with Heart Failure. International Journal of Medical Informatics 191 (2024), 105534. doi:10.1016/j.ijmedinf.2024. 105534 [19] Cyrus DiCiccio, Sriram Vasudevan, Kinjal Basu, Krishnaram Kenthapadi, and Deepak Agarwal. 2020. Evaluating fairness using permutation tests. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’20). Association for Computing Machinery, New York, NY, USA, 1467–1477. doi:10.1145/3394486.3403199 [20] Equal Employment Opportunity Commission. 1978. Uniform Guidelines on Employee Selection Procedures (1978). 29 CFR Part 1607, 43 FR 38295 (Aug. 25, 1978). https://www.ecfr.gov/current/title-29/part-1607 [21] European Parliament and Council of the European Union. 2024. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, OJ L, 2024/1689, 12.7.2024. https: //eur-lex.europa.eu/eli/reg/2024/1689/oj/eng [22] Jean Feng, Rachael V. Phillips, Ivana Malenica, et al. 2022. Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. npj Digital Medicine 5, 1 (2022), 66. doi:10.1038/ s41746-022-00611-y [23] Amelia Fiske, Sarah Blacker, Lester Darryl Geneviève, et al. 2025. Weighing the benefits and risks of collecting race and ethnicity data in clinical settings for medical artificial intelligence. The Lancet Digital Health 7, 4 (2025), e286–e294. doi:10.1016/j.landig.2025.01.003 [24] Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 5 (1971), 378–382. doi:10.1037/h0031619 . [25] Joseph Futoma, Morgan Simons, Trishan Panch, Finale Doshi-Velez, and Leo Anthony Celi. 2020. The Myth of Generalisability in Clinical Research and Machine Learning in Health Care. The Lancet Digital Health 2, 9 (2020), e489–e492. doi:10.1016/S2589-7500(20)30186-2 [26] Prakhar Ganesh, Hongyan Chang, Martin Strobel, and Reza Shokri. 2023. On the Impact of Machine Learning Randomness on Group Fairness. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’23). Association for Computing Machinery, New York, NY, USA, 1789–1800. doi:10.1145/3593013.3594116 [27] Konstantin Georgiev, Dimitrios Doudesis, Joanne McPeake, Nicholas L. Mills, Susan D. Shenkin, Jacques D. Fleuriot, and Atul Anand. 2025. Machine learningbased predictions of healthcare contacts following emergency hospitalisation using electronic health records. npj Digital Medicine 8, 1 (2025), 764. doi:10.1038/ s41746-025-02138-4 [28] Judy Wawira Gichoya, Imon Banerjee, Ananth Reddy Bhimireddy, et al. 2022. AI Recognition of Patient Race in Medical Imaging: A Modelling Study. The Lancet Digital Health 4, 6 (2022), e406–e414. doi:10.1016/S2589-7500(22)00063-2 [29] Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of Opportunity in Supervised Learning. In Advances in Neural Information Processing Systems 29 (NIPS 2016). 3315–3323. [30] Fereshteh Hasanzadeh, Colin B. Josephson, Gabriella Waters, et al. 2025. Bias recognition and mitigation strategies in artificial intelligence healthcare applications. npj Digital Medicine 8 (2025), 154. doi:10.1038/s41746-025-01503-7 [31] Wassily Hoeffding. 1963. Probability Inequalities for Sums of Bounded Random Variables. J. Amer. Statist. Assoc. 58, 301 (1963), 13–30. doi:10.1080/01621459.1963. 10500830 . [32] Raunak Jain, Mrityunjai Singh, A. Ravishankar Rao, and Rahul Garg. 2024. Predicting Hospital Length of Stay Using Machine Learning on a Large Open Health Dataset. BMC Health Services Research 24 (2024), 860. doi:10.1186/s12913-02411238-y [33] Franck Jaotombo, Vanessa Pauly, Guillaume Fond, Veronica Orleans, Pascal Auquier, Badih Ghattas, and Laurent Boyer. 2023. Machine-learning prediction for hospital length of stay using a French medico-administrative database. Journal of Market Access & Health Policy 11, 1 (2023), 2149318. doi:10.1080/20016689.2022. 2149318 [34] Alistair E. W. Johnson et al. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data 3 (2016), 160035. doi:10.1038/sdata.2016.35 [35] Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems 33, 1 (2012), 1–33. doi:10.1007/s10115-011-0463-8 [36] Ira Ktena, Olivia Wiles, Isabela Albuquerque, Sylvestre-Alvise Rebuffi, Ryutaro Tanno, Abhijit Guha Roy, Shekoofeh Azizi, Danielle Belgrave, Pushmeet Kohli, Taylan Cemgil, Alan Karthikesalingam, and Sven Gowal. 2024. Generative models improve fairness of medical classifiers under distribution shifts. Nature Medicine 30 (2024), 1166–1173. doi:10.1038/s41591-024-02838-6 [37] J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159–174. doi:10.2307/2529310

KM

[1] Emeka Abakasanga, Rania Kousovista, Georgina Cosma, Ashley Akbari, Francesco Zaccardi, Navjot Kaur, Danielle Fitt, Gyuchan Thomas Jun, Reza Kiani, and Satheesh Gangadharan. 2025. Equitable hospital length of stay prediction for patients with learning disabilities and multiple long-term conditions using machine learning. Frontiers in Digital Health 7 (2025), 1538793. doi:10.3389/fdgth.2025.1538793 [2] Joseph E. Alderman, Joanne Palmer, Elinor Laws, et al. 2025. Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations. The Lancet Digital Health 7, 1 (2025), e64–e88. doi:10.1016/S2589-7500(24)00224-3 [3] Ghazal Azarfar, Sara Naimimohasses, Sirisha Rambhatla, et al. 2025. Responsible adoption of multimodal artificial intelligence in health care: promises and challenges. The Lancet Digital Health 7, 12 (2025), 100917. doi:10.1016/j.landig.2025. 100917 [4] Pepita Barnard, John Robert Bautista, Joshua Krook, Anqi Liu, Héctor D. Menéndez, Aurora Schmidt, and Tamim Sookoor. 2023. MACAIF: Machine Learning Auditing for Clinical AI Fairness. In Proceedings of the First International Symposium on Trustworthy Autonomous Systems (TAS ’23). Association for Computing Machinery, New York, NY, USA, Article 40, 4 pages. doi:10.1145/3597512.3597522 [5] Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023. Fairness and Machine Learning: Limitations and Opportunities. MIT Press, Cambridge, MA, USA. https: //fairmlbook.org/ [6] Ainhize Barrainkua, Paula Gordaliza, Jose A. Lozano, and Novi Quadrianto. 2024. Uncertainty Matters: Stable Conclusions under Unstable Assessment of Fairness Results. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS) (Proceedings of Machine Learning Research, Vol. 238). PMLR, 1198–1206. [7] Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, and Laura M. Vardoulakis. 2020. A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/3313831.3376718 . [8] Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione, and Inioluwa Deborah Raji. 2024. AI Auditing: The Broken Bus on the Road to AI Accountability. In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 612–643. doi:10.1109/SaTML59370.2024.00037 [9] Emily Black, Manish Raghavan, and Solon Barocas. 2022. Model Multiplicity: Opportunities, Concerns, and Solutions. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 850–863. doi:10.1145/3531146.3533149 [10] Kathleen Cachel and Elke Rundensteiner. 2022. FINS Auditing Framework: Group Fairness for Subset Selections. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (AIES ’22). Association for Computing Machinery, New York, NY, USA, 144–155. doi:10.1145/3514094.3534160 [11] Zi Cai, Yu Liu, Zhiyao Luo, and Tingting Zhu. 2025. ProtoEHR: Hierarchical Prototype Learning for EHR-based Healthcare Predictions. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM ’25). Association for Computing Machinery, New York, NY, USA, 169–178. doi:10.1145/3746252.3761381 [12] John J. Cherian and Emmanuel J. Candès. 2024. Statistical Inference for Fairness Auditing. Journal of Machine Learning Research 25, 149 (2024), 1–49. [13] Christopher F. Chesley, Marzana Chowdhury, Dylan S. Small, Douglas E. Schaubel, Vincent X. Liu, Meghan B. Lane-Fall, Scott D. Halpern, and George L. Anesi. 2023. Racial Disparities in Length of Stay Among Severely Ill Patients Presenting With Sepsis and Acute Respiratory Failure. JAMA Network Open 6, 5 (2023), e239739. doi:10.1001/jamanetworkopen.2023.9739 [14] A. Feder Cooper, Katherine Lee, Madiha Zahrah Choksi, Solon Barocas, Christopher De Sa, James Grimmelmann, Jon Kleinberg, Siddhartha Sen, and Baobao Zhang. 2024. Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification. Proceedings of the AAAI Conference on Artificial Intelligence 38, 20 (2024), 22004–22012. doi:10.1609/aaai.v38i20.30203 [15] Sam Corbett-Davies, Johann D. Gaebler, Hamed Nilforoshan, Ravi Shroff, and Sharad Goel. 2023. The Measure and Mismeasure of Fairness. Journal of Machine Learning Research 24, 312 (2023), 1–117. [16] Sharon E. Davis, Chad Dorn, Daniel J. Park, and Michael E. Matheny. 2025. Emerging algorithmic bias: fairness drift as the next dimension of model maintenance and sustainability. Journal of the American Medical Informatics Association 32, 5 (2025), 845–854. doi:10.1093/jamia/ocaf039 [17] Sharon E. Davis, Peter J. Embí, and Michael E. Matheny. 2024. Sustainable deployment of clinical prediction tools: a 360 degree approach to model maintenance. Journal of the American Medical Informatics Association 31, 5 (2024), 1195–1198. doi:10.1093/jamia/ocae036 [18] Anahita Davoudi, Sena Chae, Lauren Evans, Sridevi Sridharan, Jiyoun Song, Kathryn H. Bowles, Margaret V. McDonald, and Maxim Topaz. 2024. Fairness Gaps

CI

References

CIKM ’26, November 7–11, 2026, Rome, Italy

CIKM ’26, November 7–11, 2026, Rome, Italy

Md Jannatul Rakib Joy, Viet Vo, and Caslon Chua

Ac ce

pt ed

in

KM

20 26

doi:10.1146/annurev-biodatasci-103123-095332 [58] Seyedmostafa Sheikhalishahi, Vevake Balaraman, and Venet Osmani. 2020. Benchmarking machine learning models on multi-centre eICU critical care dataset. PLOS ONE 15, 7 (2020), e0235424. doi:10.1371/journal.pone.0235424 [59] Harvineet Singh, Fan Xia, Mi-Ok Kim, Romain Pirracchio, Rumi Chunara, and Jean Feng. 2023. A Brief Tutorial on Sample Size Calculations for Fairness Audits. Workshop on Regulatable Machine Learning (RegML), NeurIPS 2023. doi:10.48550/arXiv.2312.04745 . [60] Adarsh Subbaswamy, Berkman Sahiner, Nicholas Petrick, et al. 2024. A datadriven framework for identifying patient subgroups on which an AI/machine learning model may underperform. npj Digital Medicine 7, 1 (2024), 334. doi:10. 1038/s41746-024-01275-6 [61] Elham Tabassi. 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). Technical Report NIST AI 100-1. National Institute of Standards and Technology, Gaithersburg, MD. doi:10.6028/NIST.AI.100-1 [62] Tara Templin, Sophia Fort, Prasanna Padmanabham, et al. 2025. Framework for bias evaluation in large language models in healthcare settings. npj Digital Medicine 8 (2025), 414. doi:10.1038/s41746-025-01786-w [63] Texas Department of State Health Services. 2006. Texas Hospital Inpatient Discharge Public Use Data File, Quarters 1 to 4, 2006. https://www.dshs.texas.gov/center-health-statistics/texas-health-careinformation-collection/download-and-purchase-data/texas-inpatient-publicuse-data-file-pudf/public-use-data-File-pudf-inpatient-free-download . [64] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature Medicine 29, 8 (2023), 1930–1940. doi:10.1038/s41591-02302448-8 [65] U.S. Food and Drug Administration. 2025. Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions. Guidance for Industry and Food and Drug Administration Staff. https://www.fda.gov/regulatory-information/search-fda-guidancedocuments/marketing-submission-recommendations-predetermined-changecontrol-plan-artificial-intelligence [66] Sahil Verma and Julia Rubin. 2018. Fairness Definitions Explained. In Proceedings of the International Workshop on Software Fairness (FairWare ’18). Association for Computing Machinery, New York, NY, USA, 1–7. doi:10.1145/3194770.3194776 . [67] H. Echo Wang, Jonathan P. Weiner, Suchi Saria, and Hadi Kharrazi. 2024. Evaluating algorithmic bias in 30-day hospital readmission models: retrospective analysis. Journal of Medical Internet Research 26 (2024), e47125. doi:10.2196/47125 [68] Elizabeth Anne Watkins and Jiahao Chen. 2024. The four-fifths rule is not disparate impact: a woeful tale of epistemic trespassing in algorithmic fairness. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’24). Association for Computing Machinery, New York, NY, USA, 764–775. doi:10.1145/3630106.3658938 [69] Brian J. Wells, Hieu M. Nguyen, Andrew McWilliams, et al. 2025. A practical framework for appropriate implementation and review of artificial intelligence (FAIR-AI) in healthcare. npj Digital Medicine 8, 1 (2025), 514. doi:10.1038/s41746025-01900-y [70] Jingyi Wu, Yu Lin, Pengfei Li, et al. 2021. Predicting Prolonged Length of ICU Stay through Machine Learning. Diagnostics 11, 12 (2021), 2242. doi:10.3390/ diagnostics11122242 . [71] Jenny Yang, Andrew A. S. Soltan, David W. Eyre, and David A. Clifton. 2023. Algorithmic fairness and bias mitigation for clinical machine learning with deep reinforcement learning. Nature Machine Intelligence 5 (2023), 884–894. doi:10.1038/s42256-023-00697-3 [72] Yuzhe Yang, Haoran Zhang, Judy W. Gichoya, Dina Katabi, and Marzyeh Ghassemi. 2024. The limits of fair medical imaging AI in real-world generalization. Nature Medicine 30, 10 (2024), 2838–2848. doi:10.1038/s41591-024-03113-4

CI

[38] Karim Lekadir, Alejandro F. Frangi, Antonio R. Porras, et al. 2025. FUTUREAI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ 388 (2025), e081554. doi:10.1136/bmj-2024-081554 [39] Yikuan Li, Hanyin Wang, and Yuan Luo. 2022. Improving Fairness in the Prediction of Heart Failure Length of Stay and Mortality by Integrating Social Determinants of Health. Circulation: Heart Failure 15, 11 (2022), e009473. doi:10.1161/CIRCHEARTFAILURE.122.009473 [40] Mingxuan Liu, Yilin Ning, Salinelat Teixayavong, et al. 2025. A scoping review and evidence gap analysis of clinical AI fairness. npj Digital Medicine 8, 1 (2025), 360. doi:10.1038/s41746-025-01667-2 [41] Shaina Mackin, Vincent J. Major, Rumi Chunara, and Remle Newton-Dame. 2025. Identifying and mitigating algorithmic bias in the safety net. npj Digital Medicine 8 (2025), 335. doi:10.1038/s41746-025-01732-w [42] Agatha B. Mbakwe, Ismini Lourentzou, Leo A. Celi, and Joy T. Wu. 2023. Fairness Metrics for Health AI: We Have a Long Way to Go. eBioMedicine 90 (2023), 104525. doi:10.1016/j.ebiom.2023.104525 [43] Melissa D. McCradden, Shalmali Joshi, Mjaye Mazwi, and James A. Anderson. 2020. Ethical limitations of algorithmic fairness solutions in health care machine learning. The Lancet Digital Health 2, 5 (2020), e221–e223. doi:10.1016/S25897500(20)30065-0 [44] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A Survey on Bias and Fairness in Machine Learning. Comput. Surveys 54, 6 (2021), 1–35. doi:10.1145/3457607 [45] Rachda Naila Mekhaldi, Patrice Caulier, Sondes Chaabane, Abdelahad Chraibi, and Sylvain Piechowiak. 2021. A Comparative Study of Machine Learning Models for Predicting Length of Stay in Hospitals. Journal of Information Science and Engineering 37, 5 (2021), 1025–1038. [46] Chuizheng Meng, Loc Trinh, Nan Xu, James Enouen, and Yan Liu. 2022. Interpretability and fairness evaluation of deep learning models on MIMIC-IV dataset. Scientific Reports 12 (2022), 7166. doi:10.1038/s41598-022-11012-2 [47] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations. Science 366, 6464 (2019), 447–453. doi:10.1126/science.aax2342 [48] Jed Keenan Obra, Chandan Singh, Kenshata Watkins, et al. 2025. Potential for algorithmic bias in clinical decision instrument development. npj Digital Medicine 8, 1 (2025), 762. doi:10.1038/s41746-025-02119-7 [49] Mahmud Omar, Shelly Soffer, Reem Agbareia, Nicola Luigi Bragazzi, Donald U. Apakama, Carol R. Horowitz, Alexander W. Charney, Robert Freeman, Benjamin Kummer, Benjamin S. Glicksberg, Girish N. Nadkarni, and Eyal Klang. 2025. Sociodemographic biases in medical decision making by large language models. Nature Medicine 31, 6 (2025), 1873–1881. doi:10.1038/s41591-025-03626-6 [50] Stephen R. Pfohl, Agata Foryciarz, and Nigam H. Shah. 2021. An Empirical Characterization of Fair Machine Learning for Clinical Risk Prediction. Journal of Biomedical Informatics 113 (2021), 103621. doi:10.1016/j.jbi.2020.103621 [51] Alvin Rajkomar, Eyal Oren, Kai Chen, Andrew M. Dai, Nissan Hajaj, Michaela Hardt, Peter J. Liu, Xiaobing Liu, Jake Marcus, Mimi Sun, et al. 2018. Scalable and Accurate Deep Learning with Electronic Health Records. npj Digital Medicine 1, 1 (2018), 18. doi:10.1038/s41746-018-0029-1 [52] Michael Roberts, Derek Driggs, Matthew Thorpe, Julian Gilbey, Michael Yeung, Stephan Ursprung, Angelica I. Aviles-Rivero, Christian Etmann, Cathal McCague, Lucian Beer, Jonathan R. Weir-McCall, Zhongzhao Teng, Effrossyni Gkrania-Klotsas, James H. F. Rudd, Evis Sala, and Carola-Bibiane Schönlieb. 2021. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nature Machine Intelligence 3, 3 (2021), 199–217. doi:10.1038/s42256-021-00307-0 [53] Emma Rocheteau, Pietro Liò, and Stephanie Hyland. 2021. Temporal Pointwise Convolutional Networks for Length of Stay Prediction in the Intensive Care Unit. In Proceedings of the Conference on Health, Inference, and Learning (CHIL ’21). Association for Computing Machinery, New York, NY, USA, 58–68. doi:10.1145/ 3450439.3451860 [54] Reva Schwartz, Apostol Vassilev, Kristen K. Greene, Lori Perine, Andrew Burt, and Patrick Hall. 2022. Towards a Standard for Identifying and Managing Bias in Artificial Intelligence. Technical Report NIST Special Publication 1270. National Institute of Standards and Technology, Gaithersburg, MD. doi:10.6028/NIST.SP. 1270 [55] Mark P. Sendak, Joshua D’Arcy, Sehj Kashyap, Michael Gao, Marshall Nichols, Kristin Corey, William Ratliff, and Suresh Balu. 2020. A Path for Translation of Machine Learning Products into Healthcare Delivery. EMJ Innovations (2020). doi:10.33590/emjinnov/19-00172 [56] Laleh Seyyed-Kalantari, Haoran Zhang, Matthew B. A. McDermott, Irene Y. Chen, and Marzyeh Ghassemi. 2021. Underdiagnosis Bias of Artificial Intelligence Algorithms Applied to Chest Radiographs in Under-Served Patient Populations. Nature Medicine 27, 12 (2021), 2176–2182. doi:10.1038/s41591-021-01595-0 [57] Divya Shanmugam, Monica Agrawal, Rajiv Movva, Irene Y. Chen, Marzyeh Ghassemi, Maia Jacobs, and Emma Pierson. 2025. Generative Artificial Intelligence in Medicine. Annual Review of Biomedical Data Science 8 (2025), 199–226.

Related documents

• KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models#414085
• Integrating Group and Individual Fairness Auditing in Clinical AI: A Post-Hoc, Model-Agnostic Approach#238035
• Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters#363257
• Estimating True Patient Cost-Sharing Burden: Multi-Payer Claim Reconciliation in an All-Payer Claims Database.#389062
• A Removal Based Approach to Improve LLM Faithfulness at Test-Time#927639