Demographically-Conditioned Synthetic Medical Images for Bias Mitigation and Bias Detection in Disease Classifiers Mahmoud Ibrahim1,2,3⋆ , Bart Elen3 , Chang Sun1,2 , Gökhan Ertaylan3 , and Michel Dumontier1,2 Institute of Data Science, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands 2 Department of Advanced Computing Sciences, Faculty of Science and Engineering, Maastricht University, Maastricht, The Netherlands 3 VITO, Belgium [email protected]
arXiv:2607.14984v1 [cs.AI] 16 Jul 2026
1
Abstract. Per-subgroup fairness audits of medical image classifiers face a sample-size problem: minority subgroups in held-out test sets have so few samples that the resulting confidence intervals on per-subgroup performance are wider than the bias the audit is meant to detect. We argue that a demographically-conditioned synthetic generator can do both: mitigate bias on the training side and detect bias on the evaluation side. Working on COVID-19 chest CT classification with an end-to-end finetuned Stable Diffusion 2.1 generator, we make two findings. For bias mitigation (training), a demographically-balanced synthetic cohort is most useful as a pretraining prior, not as joint augmentation: with the same fixed data, sequential pretraining followed by fine-tuning substantially outperforms joint augmentation, and the resulting classifier surpasses the full-real baseline at ∼100× real-data efficiency. For bias detection (evaluation), across five synthetic minority cohorts and five classifier seeds, the synthetic estimator reproduces the subgroup ranking of a well-powered real oracle (Spearman ρ = 1.00 on MCC and Recall) and gives the more reliable per-cell estimate where the small real test set runs out of samples. The synthetic cohort is therefore most useful in exactly the cells that fairness audits care about, as both a fix for and a measure of subgroup bias. Keywords: Synthetic data · Medical imaging · Diffusion models · Fairness audit · Subpopulation evaluation · COVID-19. Code
1
Model weights
Introduction
Per-subgroup fairness audits of medical image classifiers face a sample-size problem. Held-out real test sets are dominated by majority demographics, leaving ⋆
Corresponding author: [email protected]
2
M. Ibrahim et al.
minority subgroups with only a few dozen slices per (sex, age) cell. At that sample size, per-subgroup confidence intervals are wider than the bias the audit is meant to detect [1,2,3], and an observed disparity cannot be distinguished from sampling noise. The same imbalance drives the bias being audited: training data also under-represents those cells, pushing the decision boundary away from minorities and producing real per-subgroup performance gaps that aggregate accuracy hides. For COVID-19 chest CT classification, where real-only models routinely reach > 0.99 aggregate ROC-AUC, both problems show up in the same place: the cells with the fewest patients. The minority-subgroup problem in medical imaging is therefore two-sided. On one hand, there is real bias to mitigate: the classifier underperforms on minorities because the training data under-represents them. On the other, there is bias that is hard to detect: the test set under-represents the same cells, so the audit cannot tell a real gap apart from sampling noise. A demographicallyconditioned generative model is a natural lever on both sides of this problem. It can fill the per-cell training gaps that bias the classifier, and it can produce a synthetic minority test set large enough to give statistically stable subgroup estimates. Prior work has explored both directions separately, for chest radiograph synthesis [9,10,11] and for synthetic-test-set evaluation more broadly [15,16,14]. What is missing is a practical account of how synthetic data is most useful in each role: as joint augmentation alongside real data, or as a pretraining prior that the real data later refines? And does the resulting synthetic cohort actually give a more reliable subgroup-fairness estimate than the small real test set a practitioner would typically have? We address these two questions for COVID-19 chest CT classification. We fine-tune Stable Diffusion 2.1 [5] end-to-end on the COVIDx-CT-3A dataset [20] with natural-language prompts conditioned on sex, age, and COVID-19 status, and evaluate the generator on both sides of the problem: a training sweep over augmentation and two-stage pretraining regimes, and a five-classifier × fivecohort subpopulation audit against both a small held-out real test set and a well-powered real minority oracle. 1.1
Main contributions
1. A demographically-conditioned chest CT generator. End-to-end finetuning of a general-image latent-diffusion model on COVIDx-CT-3A produces a usable demographically-conditioned generator, beating a CXR-domainpretrained baseline on FID while matching conditioning accuracy. The generator is the common substrate of the two contributions below. 2. Bias mitigation: balanced-synthetic data, used as a prior rather than as augmentation, turns a tiny real fine-tune into a fair classifier. Using balanced-synthetic data as a pretraining prior, then fine-tuning on 1 % of the real training set, produces a classifier that surpasses the fullreal baseline on both mean and worst-cell MCC at 100× real-data efficiency. A paired comparison at fixed real + synthetic data isolates the schedule (sequential pretraining followed by fine-tuning), not the data composition, as
Demographically-Conditioned Synthetic Chest CT
3
the source of the bias-mitigation lift: the same data combined jointly underperforms the same data applied sequentially by a large margin on worst-cell fairness. 3. Bias detection: a synthetic minority cohort detects subgroup bias more reliably than the small real test, in exactly the cells where bias matters most. Across five hold-out classifiers and five synthetic cohorts, the synthetic estimator reproduces the subgroup ranking of a wellpowered real oracle perfectly on MCC and Recall (Spearman ρ = 1.00). In the minority cells where the small real test has the fewest samples, the synthetic estimator’s per-cell bias is several times smaller than the real test’s. In well-sampled cells the small real test is the better calibrator, so the two estimators are complementary along a sample-size axis.
2
Related Work
Conditional medical image synthesis. Latent diffusion models [5,6] have become the dominant approach for high-resolution medical image generation. In chest radiography, RoentGen [9] fine-tunes Stable Diffusion on chest X-rays with radiology-report conditioning; [10] demonstrates anonymized chest radiograph generation for downstream classification; MediSyn [11] extends to multimodality medical generation. For chest CT specifically, COVID-19 image synthesis has used both GANs [12] and diffusion models. We adopt latent diffusion with classifier-free guidance [7] and CLIP-based text conditioning [8]. Synthetic data for training and evaluation. Train-on-synthetic-test-onreal (TSTR) and augmentation studies show that synthetic data can substitute for or complement real training data when the generator faithfully captures the target distribution [13,14]. For minority subgroups specifically, generator-side techniques such as minority guidance [17,18] and class-balancing diffusion [19] aim to over-sample low-density regions of conditioning space. On the evaluation side, recent work [15,16] has begun to formalize synthetic test sets as a modelevaluation tool when real evaluation data is scarce or privacy-restricted. Our augmentation and sequential pretraining and fine-tuning recipes extend the training line of work by quantifying the data-efficiency gain from balanced-synthetic data; on the evaluation side, we frame audit validity as a combined rank-order and size-gated calibration question on a patient-disjoint subpopulation regime that controls for the size asymmetry between real-test and real-minority partitions. Fairness and subgroup evaluation in medical AI. Subgroup performance gaps have been documented for chest radiograph classifiers [1,2], glaucoma detection [4], and others [3]. Many of these studies are limited by the same problem we target: real minority sample sizes are too small for reliable per-subgroup estimation. Synthetic minorities offer a complement when used carefully, but as we show only as a rank-order identifier, not as an absolute estimator.
4
M. Ibrahim et al.
3
Methods
3.1
Dataset and patient-disjoint partitions
We use the COVIDx-CT-3A release of the COVID-Net family [20], retaining only the Variant-A subset with confirmed labels and excluding pneumonia cases to preserve a binary Normal vs COVID-19 task. After preprocessing this yields 168,694 axial chest CT slices from 1,718 unique patients, with patient-level sex (F/M) and age categorized into five groups (Under 20, 20–40, 40–60, 60–80, Over 80). All splits are patient-disjoint to prevent leakage and stratified on the joint (sex, age, label) distribution. We split the source data into four base partitions (Figure 1, Table 1): train, val, hold-out, and hold-out-minority. train and val have their conventional roles. hold-out (62,829 slices) matches the joint (sex, age, label) distribution of train/val and is the held-back real reference; the name marks its dual role as the generator’s training data (Section 3.2). hold-out-minority (24,969 slices) is a separate partition deliberately enriched for minority (sex, age) cells, set aside so the per-subgroup audit reaches a per-cell sample size hold-out alone cannot deliver. Aggregate experiments (Sections 4.1–4.3) use the standard threeway train/val/hold-out split. The per-subgroup audit (Section 4.4) pools hold-out with hold-out-minority and re-splits the union 80/20 at the patient level under the same stratification, yielding the well-powered minority oracle real-test-large (71,037 slices) and the small comparator real-test-small (16,761 slices). real-test-small emulates the deployment scenario: a small held-out audit set with realistic minority sparsity, the kind of test set a practitioner would actually have. real-test-large is the ground-truth oracle against which both real-test-small and the synthetic proxy are validated. The ∼4× size asymmetry between them is the experimental variable the audit in Section 4.4 is built around. 3.2
Generators
Baseline (CXR-pretrained). Following prior work on chest radiograph synthesis with latent diffusion, we adopt as a baseline the publicly available MONAI implementation [22,23] of a latent diffusion model pre-trained on MIMIC-CXR [21]. The model has the standard three components (U-Net denoiser, VAE, and text encoder), and we fine-tune the U-Net and VAE on COVIDx-CT-3A while keeping the text encoder frozen, matching the recipe of the public release. Ours (SD 2.1 fine-tuned end-to-end). Our generator instead starts from publicly released Stable Diffusion 2.1 [5] weights and fine-tunes the U-Net denoiser and the CLIP text encoder [8] end-to-end on the hold-out partition for 20,000 optimization steps; only the variational autoencoder is held fixed. The VAE compresses 512×512 inputs to 64×64×4 latents. The two design choices are ablated in Table 3.
Demographically-Conditioned Synthetic Chest CT
5
Fig. 1: Data-provenance map. The four COVIDx-CT-3A patientdisjoint partitions, the two derived per-subgroup evaluation sets (real-test-large, real-test-small) obtained by merging and re-splitting hold-out ∪ hold-out-minority, the SD 2.1 fine-tuned generator, and the four synthetic cohorts sampled from it. Exact counts and the downstream use each set feeds are given in Table 1.
Demographic and diagnostic attributes are encoded as a natural-language prompt of the form "{age} year old {SEX}. {COVID-19 findings on / Normal} chest CT scan" which is tokenized and embedded by the CLIP text encoder; the resulting 77-token embeddings condition the U-Net via cross-attention. Classifierfree guidance is implemented with 10 % null-prompt dropout during training [7]. Inference uses the DDIM scheduler with 100 steps and a guidance scale of 7.0. Training is performed in mixed bfloat16 precision with gradient checkpointing across six 40 GB A100 GPUs. 3.3
Synthetic cohorts
From the fine-tuned generator we sample the five synthetic cohorts in Table 1: Synth-Train, Synth-Holdout, and Synth-Test-Small mirror the joint demographic distribution of their real counterparts; Synth-Test-Large mirrors the well-powered real-test-large oracle and is the audit proxy in Section 4.4; Synth-Balanced samples uniformly across the 20 sex×age×label cells (2,000 slices each), providing the demographically-flat pool used by the AUG and FTBalanced regimes (Section 3.4, Table 2). 3.4
Downstream classifier regimes
All downstream classifiers share the same architecture and optimization recipe: a DenseNet-121 binary classifier (Normal vs COVID-19 ) initialized from ImageNet weights, trained with Focal Loss (γ = 2.0), Adam (learning rate 1×10−4 ,
6
M. Ibrahim et al.
Table 1: Real partitions paired with their size-matched synthetic mirrors. All real splits are patient-disjoint; synthetic cohorts are sampled per-slice from the finetuned SD 2.1 generator (DDIM 100-step, CFG 7.0). Synth-Balanced is uniform version of real data over the 20 sex×age×label cells, 2,000 slices each). Real set
Slices Pat. Synthetic mirror
Role / downstream use
train
58,499
588 Synth-Train
val hold-out
22,397 62,829
197 – 644 Synth-Holdout
classifier training; TSTRMatched, FT-Matched early stopping generator fine-tuning; threshold sweep (Fig. 4) merged with hold-out for the audit re-split
hold-out-minority 24,969 real-test-large †
71,037
real-test-small †
16,761
–
40,000
†
289 –
746 Synth-Test-Large per-subgroup oracle / proxy (§4.4) 187 Synth-Test-Small Table 4 aggregate pair – Synth-Balanced
AUG; FT-Balanced (uniform over 20 cells)
Derived by patient-disjoint 80/20 re-split of hold-out ∪ hold-out-minority (87,798 slices / 933 patients). Source totals: 1,718 patients, 168,694 slices.
batch size 64), and ReduceLROnPlateau on 1−val-F1. Early stopping is performed by keeping the epoch with the highest macro F1 on the real val partition. We train 22 regimes, listed individually in Table 2 and grouped into four families: (i) real-only baselines that establish the upper bound a real-data classifier reaches in the demographically-biased and the demographically-balanced settings; (ii) synthetic-only baselines that establish the TSTR lower bound; (iii) a six-point augmentation sweep that mixes a fixed 40,000-slice balanced-synthetic pool with growing fractions of real train; and (iv) two six-point two-stage sweeps that pretrain on a synthetic cohort and then fine-tune on a fraction of real train, contrasting balanced versus distribution-matched pretraining. Fine-tuning uses learning rate 1 × 10−5 (one order of magnitude lower than pretraining) for 15 epochs. 3.5
Evaluation metrics and audit setup
Per-regime metrics. Our primary metric across the regime sweep is Matthews Correlation Coefficient (MCC), robust to per-cell class imbalance and reported as mean and worst-cell across the 10 sex×age subgroups. We additionally report ROC-AUC, Recall, Accuracy, macro-F1, and Precision. For the subpopulation evaluation we use the same 10 subgroups, pooling Normal and COVID labels within each, and report mean and 95 % CI over the 5 classifier seeds (× 5 synthetic cohorts for synthetic estimators).
Demographically-Conditioned Synthetic Chest CT
7
Table 2: The 22 downstream classifier regimes, one row per family. For two-stage families the data spec reads pretrain → fine-tune; for single-stage families it is the direct training data. Sweep families enumerate the six sweep points inline. Acronyms: TRTR = train-on-real-test-on-real, TSTR = train-on-synthetic-teston-real, AUG = augmentation, FT = fine-tuning. Family (n)
Regime
Real-only (2)
TRTR-Biased TRTR-Balanced Synthetic-only (2) TSTR-Matched TSTR-Balanced Augmentation (6) AUG @ N Balanced pretrain (6) FT-Balanced @ p% Matched pretrain (6) FT-Matched @ p%
Data spec real train (full 58,499) real train, class-balanced subset Synth-Train (58,499, matches train) Synth-Balanced (40,000, uniform) realN ∪ Synth-Balanced (40k) Synth-Balanced → realp% Synth-Train → realp%
Sweep grids: N ∈ {500, 2k, 10k, 20k, 40k, full} for AUG; p ∈ {1, 10, 30, 50, 70, 100} for both FT families.
Subpopulation audit. The audit (§4.4) studies how to reliably evaluate persubgroup performance. To do so, we fix a real-data-trained classifier (TRTRBiased) and evaluate it on three test cohorts over the same patient-disjoint subgroups: real-test-large (the well-powered oracle), real-test-small (a small real comparator, the kind of test set a practitioner would actually have at deployment), and synth-test-large (a synthetic proxy sampled from the conditional generator at the oracle’s size and demographic distribution). The synthetic proxy is the candidate solution we propose for the small-real-test-set problem: when real minority slices are scarce, we sample a large, demographically-balanced synthetic cohort to take their place. The audit then asks how closely the proxy reproduces the oracle’s per-subgroup pattern along two axes. Magnitude agreement is the per-subgroup mean F1 and MCC, with absolute bias |mproxy − moracle | on MCC and F1. Rank agreement is Spearman ρ between the proxy’s per-subgroup ranking and the oracle’s on MCC, Recall, and AUC-PR, with a 95 % bootstrap CI.
4
Experiments and Results
4.1
Generator quality
Aggregate FID against the matching real partitions is reported in Table 3, alongside cohort-level conditioning accuracy and COVID-19 classifier AUC. The finetuned SD 2.1 generator achieves FID-Inception 50.14 and FID-RadImageNet 4.68 on the hold-out partition, improving on a CXR-domain-pretrained baseline by 35 % on FID-Inception and 32 % on FID-RadImageNet. Demographic conditioning accuracy on the synthetic cohorts (measured by separately trained sex and age-group classifiers) remains ≥ 0.84 for sex and ≥ 0.33 for the fiveclass age problem (chance 0.20); a separately trained COVID classifier achieves
8
M. Ibrahim et al.
Table 3: Aggregate generator fidelity. Our SD 2.1 (fine-tuned end-to-end) versus the CXR-pretrained latent-diffusion baseline. Sex/Age acc.: conditioning accuracy of separately-trained sex and age-group classifiers on the synthetic cohorts. COVID AUC : ROC-AUC of a separately-trained COVID classifier on the synthetic cohorts. Cohort CXR baseline
n
FID-Inc. ↓ FID-Rad. ↓ Sex ↑ Age ↑ COVID AUC ↑
62.8k
77.04
6.93
Ours / Synth-Holdout 62.8k Ours / Synth-Train 58.5k Ours / Synth-Test- 71.0k Large
50.14 50.37 47.41
4.68 4.54 4.12
–
–
0.854 0.332 0.854 0.348 0.842 0.371
– 0.995 0.994 0.990
AUC > 0.99 on the synthetic cohorts, confirming that diagnosis conditioning is preserved.
4.2
Bias mitigation I: Real+synthetic augmentation improves real-data classifiers
We first evaluate whether augmenting real training data with demographicallybalanced synthetic slices improves the resulting classifier’s per-subgroup performance. The AUG sweep adds 40,000 Synth-Balanced slices to varying real-data quantities ranging from 500 to the full 58,499. We compare these to the two real-only baselines: TRTR-Biased (full real train, mean MCC 0.784, worst-cell 0.172) and TRTR-Balanced (class-balanced subset of train, mean MCC 0.728, worst-cell 0.204). TRTR-Balanced is the fairness-conscious alternative within real-only training. Figure 2 shows the mean and worst-cell MCC across the 10 sex–age subgroups on the held-out real test set. Two observations emerge. (i) Augmentation is most useful in the real-rich regime: AUG @ N= 40k and AUG @ N=full both surpass both real-only baselines on every metric (mean MCC 0.852 and 0.852 respectively, vs 0.78 TRTR-Biased and 0.73 TRTR-Balanced; worst-cell MCC 0.31, 0.33 vs 0.17 and 0.20). (ii) At very small real N (e.g. 500 real slices), adding the same 40,000 synthetic slices harms performance, as the synthetic distribution shift dominates the small real signal. The crossover above TRTR-Biased on mean MCC occurs around Nreal = 20,000. The augmentation regime that best balances accuracy and worst-cell fairness is AUG @ N= 40k: it improves worst-cell MCC by +0.137 over TRTR-Biased and +0.105 over TRTR-Balanced without sacrificing mean MCC against either. This is a Pareto-improving modification of the real training pipeline at modest synthetic cost.
Demographically-Conditioned Synthetic Chest CT
9
Fig. 2: Augmentation lift on Real-Test (5-seed CIs). Mean (left) and worstcell (right) MCC across 10 subgroups, as a function of real training-set size N . AUG curves: mean ± 95 % CI over five classifier seeds. Real-only baselines (TRTR-Biased 0.784 / 0.172, TRTR-Balanced 0.728 / 0.204) shown as horizontals. AUG @ N ≥ 40,000 exceeds both baselines on both metrics.
4.3
Bias mitigation II: Two-stage synthetic pretraining: 100× real-data efficiency
We next test whether synthetic pretraining followed by real fine-tuning can substitute for large real training sets. The two-stage FT sweep pretrains DenseNet121 on one of two synthetic cohorts (Synth-Balanced or Synth-Train) for 20 epochs, then fine-tunes on varying percentages of the real train partition for 15 epochs at 1/10 the pretraining learning rate. Figure 3 shows the resulting mean and worst-cell MCC on the real test set.
Fig. 3: Two-stage synthetic pretraining + real fine-tuning. Mean (left) and worst-cell (right) MCC on Real-Test, as a function of real fine-tuning data size (log scale). FT-Balanced and FT-Matched curves: mean ± 95 % CI over five seeds. The circled point marks FT-Balanced at 1 % real (585 slices), which clears both baselines on both metrics (100× real-data efficiency); FT-Matched stays roughly 0.12 MCC below FT-Balanced across the sweep.
10
M. Ibrahim et al.
The headline finding: FT-Balanced @ 1 % real (585 slices) reaches mean MCC 0.83 and worst-cell MCC 0.52, exceeding both full-real-data baselines on both metrics (TRTR-Biased 0.78 / 0.17; TRTR-Balanced 0.73 / 0.20). Crucially, FT-Balanced beats even the fairness-conscious TRTR-Balanced baseline by +0.10 mean MCC and +0.32 worst-cell MCC. The synthetic demographic balancing in the pretrain provides a much larger fairness lift than rebalancing the real training set itself, while not sacrificing aggregate accuracy. Scaling the same regime from 1 % to 100 % real adds only +0.06 mean MCC, indicating that the balanced-synthetic pretraining provides most of the representational quality. The corresponding fairness gain (worst-cell MCC three times the TRTR-Biased baseline) is the highest of any of the 22 classifier regimes evaluated, including those trained on the full real dataset. Why balanced and not matched-distribution pretraining? FT-Matched (the pretraining cohort matched to the real-train distribution, blue circles in Fig. 3) underperforms FT-Balanced across the entire real-data range: matched-distribution pretraining inherits the real distribution’s demographic bias, while balanced pretraining pre-loads the model with uniform per-cell exposure that small real finetuning preserves. Notably, FT-Matched stays below the real-only TRTR-Biased baseline at every fine-tuning size, while FT-Balanced stays above it throughout. The choice of pretraining distribution is therefore a persistent design decision, not a low-data-regime artifact. Strategy beats data composition. To isolate the effect of the training strategy from the contribution of the data itself, we compare two regimes that use essentially the same data budget but combine it differently: AUG @ N= 500 mixes 500 real slices with 40,000 Synth-Balanced slices in a single optimization pass, while FT-Balanced @ 1% pretrains on the same 40,000 synthetic slices then fine-tunes on 585 real (an 85-slice difference in real budget). Across five paired classifier seeds, AUG reaches mean MCC of 0.696 ± 0.036, while FT-Balanced reaches 0.805 ± 0.023 : a delta of +0.109 ± 0.031. In terms of fairness, AUG reaches worst-cell MCC of 0.157 ± 0.035, while FT-Balanced reaches 0.524 ± 0.021: a 3.3× improvement on the fairness scalar with a delta of +0.367±0.040 worst-cell. The training schedule, not the data, is the source of the lift: the synthetic data must be applied as a representation prior and the small real set as a decisionboundary refinement, not as two streams of training signal pooled together. 4.4
Synthetic cohorts as a test-set proxy
We now ask whether synthetic data can serve as an evaluation proxy for real test data. We answer this at two granularities (aggregate and per-subgroup) and then show that the answer depends systematically on subgroup size, giving a clean operating recipe. Aggregate equivalence. We first compare the same TRTR-Biased classifier on the real test partition and on a size-matched synth-test cohort drawn to match its demographic distribution, averaging across five hold-out classifier seeds. Table 4
Demographically-Conditioned Synthetic Chest CT
11
Table 4: Aggregate-level comparison with multi-seed 95 % CIs across 5 classifier seeds: Real-Test-Small (16,761 slices) vs Synth-Test-Small (16,761 slices) for the real-data-trained TRTR-Biased classifier. Threshold-free metrics (ROC-AUC, Recall) drift by under 1 %; the threshold-sensitive metrics (F1, MCC, Precision) drift by 1.8–3.5 pp; Accuracy drifts by 0.9 pp. All 95 % CIs exclude zero. Metric ROC-AUC Recall Accuracy F1 MCC Precision
Real-Test-Small [95 % CI] Synth-Test-Small [95 % CI] 0.9996 ± 0.0001 0.9935 ± 0.0028 0.9960 ± 0.0005 0.9919 ± 0.0009 0.9838 ± 0.0019 0.9903 ± 0.0020
0.9985 ± 0.0007 0.9847 ± 0.0089 0.9869 ± 0.0041 0.9740 ± 0.0078 0.9486 ± 0.0150 0.9643 ± 0.0140
∆ −0.0011 −0.0088 −0.0091 −0.0179 −0.0352 −0.0261
shows ROC-AUC is essentially identical (∆ = −0.0011) and Recall drifts by under 1 %, while threshold-sensitive metrics (F1, MCC, Precision) drift by 1.8– 3.5 pp. All deltas are statistically significant. Threshold robustness check. A threshold sweep over T ∈ [0.05, 0.95] on p(covid) confirms that the drift in the threshold-sensitive metrics is an operating-point cliff rather than a uniform degradation (Figure 4). At the default T = 0.5, the synth−real gap is −0.025 F1 and −0.047 MCC (synthetic estimate below the real one). Tuning T on real-val (T ∗ = 0.24) and applying the same threshold to both sets, the deployment-realistic choice, shrinks this synth−real gap to −0.013 F1 and −0.026 MCC (∼45 % reduction). Tuning T on synth-test itself (T ∗ = 0.09) shrinks it further to −0.008 F1 and −0.017 MCC. The residual ∼1–2 pp gap is the genuine on-synth degradation; the rest of the drift in Table 4 is an artifact of the default argmax operating point landing on the synth generator’s weakest cell. 4.5
Bias detection: synthetic cohorts as a subgroup auditor.
Recall the audit setup (§3.5): five real-data trained hold-out classifiers are evaluated on three patient-disjoint test cohorts: (i) the well-powered oracle real-test-large (71,037 slices), (ii) real-test-small (16,761, the small real comparator a practitioner would actually have at deployment), and (iii) the synthetic proxy synth-test-large (71,037 slices matching the well-powered oracle) we propose as a stand-in when real minority slices are scarce. The question is whether the synthetic proxy can reproduce the oracle’s per-subgroup pattern well enough to substitute for it in subgroup auditing. Magnitude agreement (Figure 5A). Plotting mean F1 and MCC for the synthetic estimator and the real-test-large oracle along each demographic axis, the two lines track each other on every axis we evaluate: Pearson r = 1.00 across sex, r = 0.84 (F1) / 0.96 (MCC) across the five age bins, and r = 0.83 (F1)
12
M. Ibrahim et al.
Fig. 4: Threshold sweep on aggregate F1 and MCC. Macro-F1 (left) and MCC (right) as a function of the decision threshold T on p(covid), for the TRTR-Biased classifier on real-test (blue) and synth-test (green). Tuning T on real-val (T ≈ 0.24) shrinks the apparent Table 4 drift by ∼45 %.
/ 0.95 (MCC) across the ten joint sex–age cells. The 95 % CIs across the 5 classifier seeds are tight for the synthetic estimator across every cell, while the real-test-small estimator has wide CIs in minority cells (e.g. F-Under-20, F-Over-80) where the small real test has fewer than 200 slices. The synthetic estimator is therefore both unbiased in shape and more precise per cell. Rank agreement (Figure 5B). The Spearman counterpart, computed between each proxy estimator and the oracle’s per-subgroup mean, shows that synth-test-large achieves perfect rank agreement with the oracle on both MCC and Recall: ρ = 1.00, with 95 % bootstrap CIs [0.93, 1.00] and [0.86, 1.00], respectively. This exceeds real-test-small’s rank agreement on the same subgroups (ρ = 0.61, [0.29, 0.68] for MCC; ρ = 0.68, [0.29, 0.82] for Recall). On AUC-PR, rank correlation is weak for both estimators (ρ = 0.43, [0.14, 0.75] for synth-test-large; [0.11, 0.85] for real-test-small): per-cell AUC-PR drifts in the precision–recall trade-off under both proxies, so AUC-PR cannot reliably rank these subgroups regardless of the estimator. Synthetic estimates win where the real test fails. The synthetic estimator’s residual bias is not uniform across subgroups: it is precisely where the real test runs out of samples that the synthetic cohort is most useful. Splitting the subgroups by real-test cell size at 1500 slices, mean per-cell |proxy − oracle| on MCC flips direction between the two proxies (Figure 6): in the under-sampled bin (< 1500 real-test slices) the synthetic cohort’s MCC bias is 0.081 versus 0.183 for the real test (a ∼2.3× reduction in calibration error), while in the adequately-sampled bin (≥ 1500) the real test is ∼2.5× less biased (0.070 vs 0.178). F1 shows the same crossover (0.057 vs 0.107 under-sampled, 0.062 vs 0.170 adequately-sampled). Per-cell, the gap is even sharper on the smallest mi-
Demographically-Conditioned Synthetic Chest CT
13
Fig. 5: Subgroup audit consistency. (A) Per-subgroup F1 (top) and MCC (bottom) for the Real-Test-Large oracle (gold) vs Synth-Test-Large proxy (navy, mean over 5 cohorts ×5 seeds); per-panel Pearson r over the cells shown. (B) Spearman ρ between each proxy’s per-subgroup ranking and the oracle’s, 95 % bootstrap CIs (7 evaluable subgroups, B = 10,000).
nority: on F-Under-20 the synthetic estimator’s F1 bias is 57× smaller than the real test’s (0.002 vs 0.114). The complementarity is sharpest exactly where each proxy has a structural advantage: synthetic estimates draw from a large cohort, so they shine in cells the real test under-samples; the real test, despite being small overall, has zero distribution shift, so it dominates whenever the cell itself contains enough real slices for the estimate to converge. The two findings (perfect rank agreement; size-dependent calibration) combine cleanly: a practitioner can use the synthetic estimator to identify which subgroups warrant attention, then read per-cell magnitudes from whichever proxy has the lower bias at that cell’s real sample size. The crossover at ∼1500 real-test slices is a useful first-cut threshold but is generator-specific (see §5) and should be re-calibrated against a held-out real reference for each generator and metric.
5
Discussion
Why mitigation and detection share a root. The two contributions of this paper are not coincidentally compatible. Both rest on the same property of demographically-conditioned synthetic data: the generator samples cells that the real data distribution under-represents. On the training side, that property fills the per-cell gaps that drive subgroup performance disparities, so a balanced-synthetic pretrain can stand in for the under-represented minorities
14
M. Ibrahim et al.
Fig. 6: Calibration error binned by real-test cell size. Mean per-cell |mean(proxy) − mean(oracle)| on MCC (left) and F1 (right), split at 1500 realtest slices. The order flips between the two proxies: SYN is ∼2.3× less biased than Test on under-sampled cells, while Test is ∼2.5× less biased on adequatelysampled cells. Lower values are better.
in the training distribution. On the evaluation side, the same property gives the synthetic minority cohort more samples per cell than the small real test in the under-represented cells, so the synthetic estimator inherits a precision advantage where it matters most. Mitigation and detection are therefore not two independent uses of the generator; they are two faces of one structural fix. What the 100× data-efficiency claim actually buys. Balanced-synthetic pretraining followed by fine-tuning on 585 real slices (1 % of the real training set) reaches mean MCC 0.83 and worst-cell MCC 0.52, surpassing real-only training on the full 58,499 slices (0.78 / 0.17) on both metrics. In medical imaging, labeled data is scarce overall, especially for minority subgroups. We show how a balanced resource (synthetic slices) can substitute for the expensive and rare one (labeled minority data) at a 100 : 1 ratio without losing accuracy or fairness. A single institution with a few hundred labeled minority slices can therefore train and audit a fair classifier without requiring the large multi-site cohort. Why the bias-mitigation lift lives in the schedule. The same logic explains why sequential pretraining and fine-tuning beats joint augmentation by +0.367± 0.040 worst-cell MCC at fixed data (§4.3). In the joint regime, the 500 real and 40,000 synthetic slices contribute simultaneously to every gradient step; the optimizer sees one pooled distribution dominated 80:1 by synthetic, and the small real signal is unable to correct generator-specific per-cell drift in the decision boundary. In the sequential regime, the synthetic data sets up the representation first, and then the small real fine-tune adjusts only the decision boundary at 1/10 the learning rate. The synthetic cohort and the real cohort therefore play
Demographically-Conditioned Synthetic Chest CT
15
different roles in the pipeline (one builds the representation, the other refines the boundary), and conflating them in one training pass discards the structure that makes the prior useful. Threshold protocol for synthetic evaluation. The aggregate cliff in Table 4 forces a choice of operating point on synthetic test data. Our recommendation: fix T on real-val (or use the deployment default) and report the synth drift at that T . Do not re-tune T on synth: that would overfit the comparison to the synthetic distribution and disguise the realistic cost of substitution (1.3 % F1 / 2.6 % MCC for our classifier at T = 0.24). For per-cell audits, the threshold must match across the real and synth estimators, or the comparison loses meaning. For purely rank-based use cases, threshold-free metrics sidestep the question; the real and synth aggregates agree to ∆AUC = 0.0011. Limitations and future work. The analyses reported here are based on a single fine-tuned generator and on a single dataset (COVIDx-CT-3A). Crossclassifier audit, i.e., whether synthetic minorities can rank different classifiers consistently, is an open question that we leave for future work. Privacy-side analyses (nearest-neighbour in feature space, membership inference) would strengthen claims about synthetic-data deployment.
6
Conclusion
Demographically-conditioned synthetic chest CT can both mitigate subgroup bias (in training) and detect it (in evaluation). On the mitigation side, a balancedsynthetic cohort serves best as a representation prior: applied sequentially before a small real fine-tune, it produces a classifier from 1 % of the real training data that surpasses the full-real baseline on both mean and worst-cell performance. A paired comparison at fixed data isolates the training schedule, not the data composition, as the source of the lift. On the detection side, the synthetic minority cohort is a reliable proxy for evaluation and for detecting subgroup bias, especially in minority cells where labeled real data is scarce. It matches the wellpowered real oracle on both axes the audit cares about: in magnitude (per-cell F1 and MCC bias several times smaller than the small real test in under-sampled cells) and in rank correlation (Spearman ρ = 1.00 on MCC and Recall, against ρ ≈ 0.6 for the small real test). The two estimators are complementary along a sample-size axis: the synthetic proxy wins in under-sampled minority cells, while the small real test wins in well-sampled ones. Synthetic chest CT is most useful where fairness audits are weakest, simultaneously as a fix for and a measure of subgroup bias. Code and data availability. The training and evaluation code is available at github.com/mahmoudibrahim98/synthetic-fairness, and the fine-tuned generator weights are released at huggingface.co/mahmoudibra98/covid-ct-sd21.
16
M. Ibrahim et al.
Acknowledgments. Acknowledgments scrubbed for double-blind review. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.
References 1. Seyyed-Kalantari, L., Liu, G., McDermott, M., Chen, I.Y., Ghassemi, M.: CheXclusion: Fairness gaps in deep chest X-ray classifiers. In: Pacific Symposium on Biocomputing 2021. pp. 232–243 (2020) 2. Seyyed-Kalantari, L., Zhang, H., McDermott, M.B.A., Chen, I.Y., Ghassemi, M.: Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine 27(12), 2176–2182 (2021) 3. Yang, Y., Zhang, H., Gichoya, J.W., Katabi, D., Ghassemi, M.: The limits of fair medical imaging AI in the wild. arXiv preprint arXiv:2312.10083 (2023) 4. Lin, M., Xiao, Y., Hou, B., et al.: Evaluate underdiagnosis and overdiagnosis bias of deep learning model on primary open-angle glaucoma diagnosis in under-served populations. In: AMIA Summits on Translational Science Proceedings. p. 370 (2023) 5. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10684–10695 (2022) 6. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS. vol. 33, pp. 6840–6851 (2020) 7. Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022) 8. Radford, A., Kim, J.W., Hallacy, C., et al.: Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021) 9. Chambon, P., Bluethgen, C., Delbrouck, J.B., et al.: RoentGen: Vision-language foundation model for chest X-ray generation. arXiv preprint arXiv:2211.12737 (2022) 10. Packhäuser, K., Folle, L., Thamm, F., Maier, A.: Generation of anonymous chest radiographs using latent diffusion models for training thoracic abnormality classification systems. In: ISBI. pp. 1–5 (2023) 11. Cho, J., Mathur, R., Zakka, C., et al.: MediSyn: A generalist text-guided latent diffusion model for diverse medical image synthesis. arXiv preprint arXiv:2405.09806 (2024) 12. Jiang, Y., Chen, H., Loew, M., Ko, H.: COVID-19 CT image synthesis with a conditional generative adversarial network. IEEE Journal of Biomedical and Health Informatics 25(2), 441–452 (2020) 13. Esteban, C., Hyland, S.L., Rätsch, G.: Real-valued (medical) time series generation with recurrent conditional GANs. arXiv preprint arXiv:1706.02633 (2017) 14. Koetzier, L.R., Wu, J., Mastrodicasa, D., et al.: Generating synthetic data for medical imaging. Radiology 312(3), e232471 (2024) 15. van Breugel, B., Seedat, N., Imrie, F., van der Schaar, M.: Can you rely on your model evaluation? Improving model evaluation with synthetic test data. In: NeurIPS. vol. 36, pp. 1889–1904 (2023) 16. van Breugel, B., Seedat, N., Imrie, F., van der Schaar, M.: SYNG4ME: Model evaluation using synthetic test data. OpenReview preprint (2023)
Demographically-Conditioned Synthetic Chest CT
17
17. Um, S., Lee, S., Ye, J.C.: Don’t play favorites: Minority guidance for diffusion models. arXiv preprint arXiv:2301.12334 (2023) 18. Sehwag, V., Hazirbas, C., Gordo, A., Ozgenel, F., Canton, C.: Generating high fidelity data from low-density regions using diffusion models. In: CVPR. pp. 11492– 11501 (2022) 19. Qin, Y., Zheng, H., Yao, J., Zhou, M., Zhang, Y.: Class-balancing diffusion models. In: CVPR. pp. 18434–18443 (2023) 20. Gunraj, H., Sabri, A., Koff, D., Wong, A.: COVID-Net CT-2: Enhanced deep neural networks for detection of COVID-19 from chest CT images through bigger, more diverse learning. Frontiers in Medicine 8, 729287 (2022) 21. Johnson, A.E.W., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C., Mark, R.G., Horng, S.: MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data 6(1), 317 (2019) 22. MONAI Generative Models: Chest X-ray with Latent Diffusion Models. GitHub repository, https://github.com/Project-MONAI/GenerativeModels/tree/ 7428fce/model-zoo/models/cxr_image_synthesis_latent_diffusion_model (2023), last accessed 2026-05-30. 23. Pinaya, W.H.L. (Warvito): Latent diffusion models for chest X-ray generation using MONAI Generative Models. GitHub repository, https://github.com/Warvito/ generative_chestxray (2023), last accessed 2026-05-30.