Counterfactual Contrastive Analysis

Counterfactual Contrastive Analysis Yunlong He and Pietro Gori⋆

arXiv:2608.19032v1 [cs.CV] 19 Aug 2026

LTCI, Télécom Paris, Institut Polytechnique de Paris, France pietro.gori@telecom-paris.fr

Abstract. Visual Counterfactual Explanations (VCEs) aim to explain image classifiers by generating minimally edited and realistic versions of an input image that change the classifier’s prediction. Existing VCE methods are inherently classifier-dependent and therefore susceptible to classifier biases and failure modes, such as sensitivity to shortcut features and calibration errors. In this paper, we propose a classifier-free approach for visual counterfactual generation based on Contrastive Analysis (CA). Given two datasets corresponding to different classes (e.g., healthy and patients), we disentangle the generative factors that are common across the two datasets from those that are salient to each dataset, and generate counterfactual images by swapping only the salient factors. By operating directly on data distributions rather than decision boundaries, our method provides model-agnostic VCEs that are less sensitive to classifier biases. Our approach leverages the high-quality synthesis and wellstructured latent space of StyleGAN2. We use the feature space F , instead than the usual W -space, to improve detail preservation. Unlike conventional CA approaches, which typically assume salient factors in only one dataset, we introduce an adapted framework and loss functions for VCE that allow multiple salient factors in each dataset. We evaluate our method on three medical imaging datasets and demonstrate superior counterfactual generation quality compared to existing approaches. Keywords: Visual Counterfactual Explanations · Contrastive Analysis

1

Introduction

With the rapid scaling of modern AI models, the ability to explain black-box decisions has become critical, particularly in medical imaging. Explainability often governs whether models can be trusted and accepted for real-world use. Visual counterfactual explanations (VCEs) [2, 9, 14, 15] are widely studied as a way to probe a classifier’s behavior, seeking the smallest realistic changes to an input image that would alter its prediction. VCEs for image classifiers are typically realized by generating a counterfactual (CF) image as an edited version of the original input. This task is often posed as a constrained generation problem with three objectives. First, the CF must be valid, meaning that it changes the classifier’s prediction to the desired ⋆

Corresponding author.

2

Y. He and P. Gori

outcome. Second, the edited image should remain realistic, staying on the image manifold and avoiding unnecessary changes to irrelevant content. Third, the induced changes should be interpretable and minimal in a semantic sense, so that the explanation clearly indicates the key evidence responsible for the classifier’s decision change. Most VCE methods instantiate these objectives within a generative framework, leveraging VAEs [27], GANs [20, 32], or diffusion models (DMs) [14–16, 35]. Among them, DMs currently achieve state-of-the-art performance for realistic CF image generation, typically guiding the generation process with classifier-driven losses to reach a label-flipping outcome, while penalizing deviations to preserve plausibility and similarity to the input. However, classifier guidance can induce changes aligned with the classifier biases and failure modes, such as sensitivity to shortcut features [35] and calibration errors, rather than realistic changes in the underlying data distribution, potentially reducing edit fidelity. In addition, these methods offer limited insight into which semantic factors drive the prediction flip, thereby limiting their interpretability. In this paper, we introduce a classifier-free approach for visual counterfactual generation grounded in the Contrastive Analysis (CA) framework [1, 7, 10, 22, 23, 34]. Instead of generating CFs based on a classifier’s decision boundary, we operate at the data-distribution level, as in CA. Given two class-specific datasets, the proposed method aims to uncover the common and salient (i.e., class-specific) generative factors. CFs can then be generated by swapping only the salient factors, while preserving the common ones. Most CA methods are based on deep generative models, such as VAEs [3, 6, 19, 30, 34, 38] and GAN [7, 8]. While reconstruction can regularize the salient space by suppressing common information, it is insufficient on its own and typically requires additional constraints for clean latent separation [22, 23, 28, 34]. A recent work [10] extended CA to high-quality image datasets using modern GANs and diffusion models and reported that StyleGAN2 [17] achieves diffusion-level quality while enabling substantially faster inference. In parallel, advances in StyleGAN editing have shown that translating edits from the W-space to the generator’s intermediate feature space (F-space) improves fine-grained details [5]. Inspired by these findings, we propose a StyleGAN-based CA framework that leverages latent disentanglement and F-space refinement for VCEs on medical imaging datasets. Our contributions are: 1. A classifier-free VCE generative framework grounded in CA that enables controllable, classifier-agnostic CF generation via disentangled latent representations, yielding high-fidelity results and high semantic interpretability. 2. A CA-based framework, supporting both background-target and multi-salient assumptions, based on StyleGAN2’s high-quality synthesis and well-structured F-space, instead of standard W-space, for better detail preservation. 3. Experiments on three medical imaging datasets demonstrate improved disentanglement performance over CA baselines and superior generation quality to prior state-of-the-art VCE approaches with faster image editing (0.25s/image).

Counterfactual Contrastive Analysis

3

Fig. 1. Overview of the proposed CA-based framework for CF generations. Left: we first disentangle each input into a common c and a salient factor s, enabling reconstruction and CF editing by swapping the salient factors between images from X and Y . Right: we then refine the synthesized image by adjusting F -space features according to the feature shift ∆fy→x = f (cy + sy ) − f (cy + sx ) common-salient factor manipulation.

2

Method

Problem Statement. In the CA setting, we consider two datasets X = {xi }N i=1 and Y = {yj }M j=1 . Most prior work adopts a background-target (BT) assumption, where only one dataset (typically Y ) contains a dataset-specific (or salient) pattern (e.g., a pathology) absent from the other, while both share common content (e.g., healthy anatomy). We generalize CA to a multiple-salient (MS) setting, where both X and Y contain their own salient patterns that are absent in the other. Our goal is to learn a common generative factor c ∈ Rdc capturing information shared across X and Y , and salient generative factors sx ∈ Rdsx , sy ∈ Rdsy that capture variations unique to X and Y , respectively. Common and Salient (CS)-StyleGAN. In the context of StyleGAN, given samples x ∈ X and y ∈ Y , an encoder E (i.e., pSp [26]) maps them to latent codes wx = E(x) and wy = E(y), where wx , wy ∈ Rdw . We introduce a CS separator Hcs,ϕ , which consists of one common branch Cϕc and two salient branches Sx,ϕsx and Sy,ϕsy . For brevity, we omit the parameters ϕ in the following. We define the latent factors as: cx = C(wx ) and cy = C(wy ). In the MS setting, we set sx = Sx (wx ) and sy = Sy (wy ), while in the BT setting, we remove Sx and use Sy only (all Sx -related terms are omitted). Latent Regularization. We first train the Hcs separator with the following reconstruction loss in the StyleGAN2 latent space: Llat = ∥C(wx ) + Sx (wx ) − wx ∥22 + ∥C(wy ) + Sy (wy ) − wy ∥22 ,

(1)

where each term encourages the sum of the common output and the corresponding salient output to reconstruct the StyleGAN2 latent (wx or wy ). To enforce that the salient factors capture only dataset-specific information, we additionally

4

Y. He and P. Gori

penalize cross-dataset activations, by minimizing ∥Sy (wx )∥22 and ∥Sx (wy )∥22 (we use only ∥Sy (wx )∥22 under the BT assumption, as explained in Fig. 1). The loss in Eq. (1) is necessary but insufficient. In particular, it does not enforce (i) common consistency, i.e., that the common factors capture the same information across X and Y , nor (ii) common-salient independence, i.e., that common and salient factors do not share information. To promote common consistency, we impose distributional alignment of the common factors by introducing a discriminator D that predicts whether C(w) originates from X or Y , while the separator Hcs is trained to make this prediction impossible:   LD = min max − Ewy [log D(C(wy ))] − Ewx [log(1 − D(C(wx )))] . (2) Hcs

D

Here, D is optimized to discriminate X from Y using the common codes, whereas Hcs is optimized adversarially so that C(wx ) and C(wy ) become indistinguishable. To encourage common–salient statistical independence, we introduce two regressors, Rx and Ry , that attempt to predict the salient outputs Sx (wx ) and Sy (wy ) from the corresponding common outputs C(wx ) and C(wy ), respectively. If the regressors can accurately predict salient output from the common one, this indicates that C(w) still contains salient information and the separation is incomplete. We therefore train the separator adversarially against the regressors by minimizing the following objective:      LR = min max − Ewx ∥Rx (C(wx ))−Sx (wx )∥22 − Ewy ∥Rx (C(wy )) Hcs R (3)     − Ewy ∥Ry (C(wy ))−Sy (wy )∥22 − Ewx ∥Ry (C(wx )) . This discourages C(w) from encoding salient patterns. We regress s from c because c is typically bigger (more information) and thus tends to absorb s. Image-Space Losses. We adopt an image-space reconstruction loss combining pixel and perceptual terms. Given a real image i and its reconstruction bi, we use Lrec (i, î) = ∥i − î∥22 + ∥V (i) − V (î)∥22 , where the first term is a pixel-wise ℓ2 loss, and the second term is the LPIPS perceptual loss [37]. The function V (·) extracts perceptual features using a pretrained VGG16 network [31]. We use this Eq. to reconstruct both x and y, minimizing the image-space loss: Limg = Lrec (x, G(cx + sx )) + Lrec (y, G(cy + sy )),

(4)

where G is the pretrained StyleGAN2 generator. The final loss to train Hcs is: LHcs = λlat Llat + λD LD + λR LR + λimg Limg

(5)

F-space Refinement. We first learn common and salient factors in the W space of StyleGAN2. Then, we add a refinement module that edits intermediate StyleGAN feature space (F space), as in [5]. Let f (·) denote an intermediate feature of the generator G, e.g., the output of layer l=9. Given the learned latent factors, we define F-space feature shifts for the two swap directions as: ∆fx→y = f (cx + sx ) − f (cx + sy ); ∆fy→x = f (cy + sy ) − f (cy + sx ).

(6)

Counterfactual Contrastive Analysis

5

Intuitively, ∆fx→y captures the change induced by replacing the salient factor of x with the one of y while keeping cx fixed, and ∆fy→x is defined analogously. To condition the shifts on the source image, we extract features from reconstructions using a pretrained F-space encoder Ef : fx = Ef (G(cx + sx )) and fy = Ef (G(cy +sy )). As in [5], we concatenate f and ∆f , and pass them to an adapter FAdpt , yielding: fbx→y = FAdpt (fx ⊕∆fx→y ) and fby→x = FAdpt (fy ⊕∆fy→x ), where ⊕ denotes concatenation. Decoding the adapted features with G yields the edited images. FAdpt is trained with: Lx→y = Lrec (G(cx + sy ), G(fbx→y )) and Ly→x = Lrec (G(cy + sx ), G(fby→x )). We also include a real-image reconstruction term to preserve performance on real inputs: Lreal = Lrec (x, x b) + Lrec (y, yb), where x b = G(FAdpt (Ef (x))) and yb = G(FAdpt (Ef (y))). Finally, we encourage realism with an adversarial loss Ladv = Ladv-x + Ladv-y , where Ladv-x is applied to G(fby→x ) against x, and Ladv-y to G(fbx→y ) against y. The overall refinement objective is: Lrefine = Lx→y + Ly→x + Lreal + λadv Ladv .

3

Experiments

Datasets. We evaluate our method on three 2D medical imaging datasets: BloodMNIST [36] (Eosinophil Vs Neutrophil; Train=2500/Test=500), OCTMNIST [36] (Normal Vs Choroidal neovascularization; Train=17000/Test=3000 + Diabetic macular edema Vs Drusen; Train=7000/Test=2000), and BraTS2023 [24] (Healthy Vs Tumor; Train=8000/Test=2000). For each dataset, we define two groups, denoted X and Y , based on the provided labels. We construct balanced splits such that X and Y contain the same number of images in both training and test sets. All images are preprocessed to a resolution of 256 × 256. For BloodMNIST and OCTMNIST, we resize the 224 × 224 images from the MedMNIST+ release to 256 × 256. For MRI scans, we use zero-padding to obtain 256 × 256 images while preserving the original aspect ratio. Implementation Details. We train the proposed CS-StyleGAN in two stages. In Stage 1, we optimize the separator Hcs to disentangle common and salient factors in the W + space. We first warm up Hcs for 2,000 steps using only Llat and Limg to obtain informative latent factors. We then alternate between updating D and R and updating Hcs : with Hcs fixed, D and R are updated by maximizing the objectives in Eqs. (2)–(3) using separate Adam optimizers with a learning rate of 10−4 ; with D and R fixed, Hcs is updated by minimizing Eq. (5) using an Adam optimizer with a learning rate of 10−3 . In Stage 2, using the common and salient factors learned in Stage 1 and their corresponding F-space feature shifts, we optimize the refinement module (FAdpt ) with Lrefine . We use the Ranger optimizer employed in SFE [5], with our learning rate set to 10−4 . For each dataset, the pSp encoder E, StyleGAN2 generator G, and F-space encoder Ef are pretrained on the corresponding training set before the two-stage training procedure. Detailed network architectures are provided in the supplementary material. The implementation code is available at https://github.com/BioMedTP/CF_Contrastive_Analysis.

6

Y. He and P. Gori

Fig. 2. Comparisons with CA baselines on BraTS, BloodMNIST, and OCTMNIST (Normal vs. CNV) (top to bottom). Reconstruction: use the common and salient factors from original images. Salient Swap: swap salient factors between x and y while keeping common factors fixed during generation.

Baselines and Evaluation Metrics. We compare our method with CA baselines MM-cVAE [34], SepVAE [23], and Double InfoGAN [7], and with diffusionbased VCE methods: ACE [15], DiME [14], FastDiME [35], and TIME [16]. We also consider off-the-shelf T2I DMs as CF alternatives (not CF-specific methods), including FLUX [4] and SDXL [25] adapted with LoRA [12]. We evaluate the proposed method on CA and CF generation using widely adopted metrics to evaluate synthesis quality: L2, LPIPS [37], MS-SSIM [33], and FID [11]. Furthermore, we assess whether swapped outputs match the post-edit domain distribution using FID, reporting FIDX→Y (FID between X images after swapping to Y and real Y ) and FIDY→X (swapped Y vs. real X). We evaluate editing success using the same U-Net-based classifier architecture as in [35], trained on real images from the corresponding training set. We report the classification accuracy with respect to the desired post-edit label. About latent separation quality, we train a classifier on the learned latent factors to predict the class (X or Y ). Higher accuracy indicates the class is encoded in the corresponding (common or salient) space. Please note that in the BT assumption, the information about X is entirely encoded in c. Eventually, following VCE prior works [15,35], we report FID, sFID, Flip Ratio (FR), Mean Absolute Difference (MAD) and Bidirectional KL Divergence (BKL) for measuring counterfactual quality.

Counterfactual Contrastive Analysis

7

Table 1. Quantitative comparison results on multiple datasets. Best results in bold, second-best underlined (within each dataset block). Ref: F -space refinement; BT/MS Ass.: background-target/multi-salient assumptions. Reconstruction (X) Dataset (X vs. Y )

Swap

20.32 0.008 23.24 0.006 23.96 0.005 24.03 0.004 38.19 0.0002

0.852 0.933 0.925 0.91 0.99

0.163 0.156 0.151 0.054 0.004

89.86 21.06 0.007 115.2 23.75 0.006 125.1 25.44 0.004 44.46 24.75 0.003 2.64 38.20 0.0002

0.951 0.930 0.958 0.90 0.99

0.157 0.152 0.145 0.055 0.005

86.36 109.4 128.3 30.10 2.57

98.63 113.7 137.2 37.10 25.17

92.77 111.9 135.4 45.50 32.56

0.753 0.821 0.901 0.939 0.914

BloodMNIST (Eos. vs. Neutrophil)

MMc-VAE SepVAE Double InfoGAN Ours (pSp-cs) Ours (pSp-cs-Ref )

18.27 0.015 18.04 0.016 19.45 0.015 22.35 0.006 41.99 0.0001

0.556 0.551 0.567 0.725 0.998

0.378 0.421 0.472 0.139 0.004

175.1 182.8 95.97 73.46 2.887

19.42 0.011 18.01 0.016 19.70 0.014 22.43 0.006 42.54 0.0001

0.643 0.579 0.606 0.760 0.998

0.357 0.419 0.491 0.139 0.004

179.6 219.5 104.7 70.42 2.590

181.7 217.5 149.50 84.05 43.37

183.9 221.9 135.7 88.61 45.90

0.819 0.691 0.646 0.979 0.981

OCTMNIST (DME vs. Drusen)

MMc-VAE 23.44 0.005 SepVAE 23.89 0.005 Double InfoGAN 16.84 0.026 Ours (BT Ass.) 23.79 0.005 Ours (BT Ass. Ref ) 38.24 0.0002 Ours (MS Ass.) 26.50 0.005 Ours (MS Ass. Ref ) 38.09 0.0001

0.805 0.822 0.592 0.806 0.992 0.814 0.994

0.404 128.05 26.19 0.003 0.400 107.84 26.26 0.003 0.369 181.38 18.89 0.017 0.111 51.10 24.99 0.004 0.0143 5.832 39.02 0.0001 0.108 48.64 24.70 0.004 0.014 5.356 39.00 0.0001

0.867 0.872 0.698 0.830 0.994 0.837 0.995

0.390 0.395 0.300 0.084 0.012 0.083 0.012

117.71 105.92 179.87 42.62 3.294 39.43 3.911

126.49 131.22 189.06 65.66 44.67 59.21 39.24

128.17 143.67 199.42 79.13 63.62 73.79 55.92

0.665 0.597 0.832 0.627 0.689 0.876 0.863

Table 2. Latent separation results. Mean ± std of 5-fold cross-validated classification accuracy to distinguish between the class of X or Y using logistic regression and using as features either C, Sx or Sy . ∆ = |0.5 − C| + |1.0 − S1 |(+|1.0 − S2 |). BraTS (Healthy vs. Tumored) Model

Reconstruction (Y )

PSNR↑ MSE↓ MS-SSIM↑ LPIPS↓ FID↓ PSNR↑ MSE↓ MS-SSIM↑ LPIPS↓ FID↓ FIDX→Y ↓ FIDY→X ↓ Acc (X+Y)↑

Method

MMc-VAE SepVAE BraTS Double InfoGAN (Healthy vs. Tumored) Ours (pSp-cs) Ours (pSp-cs-Ref )

C

Sy

∆

MM-cVAE 0.68 ± 0.02 0.73 ± 0.02 0.45 SepVAE 0.67 ± 0.02 0.92 ± 0.01 0.25 Double Info. 0.65 ± 0.01 0.86 ± 0.01 0.29 Ours (BT Ass.) 0.58 ± 0.08 0.95 ± 0.01 0.13 Ours (MS Ass.) – – – Expected 0.5 1.0 0

OCTMNIST (DME vs. Drusen) C

Sx

Sy

∆

0.61 ± 0.01 0.69± 0.03 – 0.42 0.64 ± 0.02 0.94 ± 0.01 – 0.20 0.76 ± 0.02 0.61 ± 0.02 – 0.65 0.57 ± 0.08 0.96 ± 0.006 – 0.11 0.54 ± 0.05 0.97 ± 0.008 0.98 ± 0.003 0.09 0.5 1.0 1.0 0

Table 3. Ablation study on the BraTS dataset: Healthy (X) vs. Tumored (Y). Latent Separation Model

C

S

Image Edit Quality ∆ FIDX→Y FIDY→X Acc.

Base (Llat +Limg ) 0.74 ± 0.010 0.73 ± 0.02 0.51 39.78 Base + LD 0.64 ± 0.02 0.92 ± 0.01 0.22 37.03 0.68 ± 0.01 0.86 ± 0.01 0.32 38.78 Base + LD +LDiscMI Base + LD +LR 0.58 ± 0.08 0.95 ± 0.01 0.13 37.10 Base + LD +LR +Lrefine – – – 25.17

55.75 0.535 53.21 0.638 55.04 0.593 45.50 0.939 35.56 0.913

Comparison with CA Baselines. Fig. 2 qualitatively compares our method with CA baselines for reconstruction and salient swapping. Our reconstructions are sharper and better preserve fine anatomical structures. For salient swapping, our approach transfers the salient pattern (tumors in BraTS, eosinophil–neutrophil staining differences in BloodMNIST, and CNV-related abnormalities in OCTMNIST) while preserving shared anatomy (healthy brain tissues, global cell layout/nuclear morphology, and normal retinal layer structure, respectively). Table 1 reports quantitative results, where our pSp-based method, with or without F-space refinement (-Ref ), consistently outperforms all CA baselines on both reconstruction and swapping. Notably, with the MS setting, our method better handles the challenging DME vs. Drusen case, achieving the best reconstruction metrics and the highest swapping quality (FID and classification accuracy of 0.876). Finally, Table 2 evaluates latent separation between the learned common (C) and salient (S) factors. Salient patterns are mainly captured in S rather than in C, and our method attains the lowest separation gap ∆ across all methods, indicating improved semantic disentanglement over prior approaches. In Table 3, we also report an ablation study. Counterfactual Explanations Based on CA. Table 4 reports quantitative comparisons on BraTS Healthy (X) and Tumored (Y ) for two counterfactual directions: adding tumors (X → Y ) and removing tumors (Y → X). Overall, our CA-based swapping achieves the best image fidelity in both directions, as

8

Y. He and P. Gori

Table 4. CF explanation comparison on BraTS. Best in bold, second-best underlined. Metrics for CF edits (X→Y: add tumors; Y→X: remove tumors) and runtime. CF: X→Y (add tumors) Method

FID

ACE 38.553 DiME 38.378 FastDiME 32.431 FastDiME-2+ 32.865 TIME 76.704 SDXL-LoRA 51.258 Ours

sFID

L1

FR

MAD BKL

CF: Y→X (remove tumors) FID

sFID

L1

FR

MAD BKL Time (s/img)

51.428 0.003 0.977 0.942 0.043 39.436 52.841 0.005 0.895 0.855 0.120 21.37 ± 0.34 50.421 0.026 0.947 0.934 0.054 48.752 60.722 0.025 0.741 0.723 0.228 294.12 ± 47.44 44.735 0.017 0.905 0.873 0.108 51.391 64.007 0.016 0.696 0.658 0.285 16.07 ± 0.64 45.283 0.017 0.912 0.876 0.106 52.191 64.753 0.016 0.707 0.668 0.277 30.69 ± 0.91 91.064 0.012 0.389 0.293 0.475 88.706 100.074 0.012 0.341 0.247 0.413 17.59 ± 0.40 60.179 0.027 0.731 0.727 0.230 59.006 69.185 0.028 0.824 0.817 0.164 4.95 ± 0.67

27.533 38.441 0.021 0.953 0.946 0.047 37.492 47.527 0.022 0.914 0.906 0.074 0.25 ± 0.005

Fig. 3. Comparisons of CF generation. Cols. 2–5 show CF outputs from SOTA diffusion-based methods. Cols. 6–8 show CF outputs from T2I diffusion models.

indicated by the lowest FID/sFID scores, and is better or on par with diffusionbased CF baselines (ACE, DiME, and FastDiME) in terms of counterfactual effectiveness (i.e., FR/MAD/BKL). In Fig. 3, we observe that diffusion-based methods struggle to completely remove large tumors from Y images, and the added tumors in X → Y have often limited size and/or are misplaced. In addition, text-conditioned editing (e.g., FLUX) or learning per-image embeddings (e.g., TIME) can drift away from the real image manifold, while fine-tuning large diffusion models with LoRA (e.g., SDXL-LoRA) may alter common content beyond the desired tumor-related changes. By contrast, our approach explicitly swaps salient generative factors between X and Y , improving image fidelity and offering better controllability for tumor-specific counterfactual manipulation. Thanks to the well structured latent space of StyleGAN2, we can also interpolate between salient factors. Given a pair of samples (x, y) from two domains/classes, we can generate a continuous counterfactual trajectory by scaling the salient change with a scalar α ∈ [0, 1], e.g., G(αfbx→y ). This is shown in Fig. 4, where the predicted class probability changes smoothly along the trajectory, suggesting that the generated counterfactuals form a coherent explanation path.

4

Conclusion

We propose a classifier-free, CA-based framework for VCEs that yields strong common–salient disentanglement and high-quality CF generations. Interesting

Counterfactual Contrastive Analysis

9

Fig. 4. Interpolation along salient factors. Gradual interpolation from sx to sy (and viceversa) using an interpolation weight α (0 → 1) during generation. The classifier prediction changes consistently along the trajectory, illustrating gradual insertion/removal of class-specific evidence while preserving common content.

perspectives might be adapting the proposed method to recent VLM, as in [18], and studying its identifiability. Similarly to all recent VCE [14, 15, 35] and CA [7, 22] works, we note that also our method also, without strong assumptions, is not identifiable [13, 21] and does not enable causal discovery. This is generally impossible from purely observational single-modal data. An interesting direction, in line with causal representation learning theory [29], would be using additional prior or longitudinal data, to further constrain the learning process.

References 1. Abid, A., Zhang, M.J., Bagaria, V.K., Zou, J.: Exploring patterns enriched in a dataset with contrastive principal component analysis. Nature Communications (2018) 2. Augustin, M., Boreiko, V., Croce, F., Hein, M.: Diffusion visual counterfactual explanations. In: NeurIPS (2022) 3. Benaim, S., Khaitov, M., Galanti, T., Wolf, L.: Domain intersection and domain difference. In: ICCV (2019) 4. Black Forest Labs, et al.: FLUX.1 Kontext: Flow matching for in-context image generation and editing in latent space (2025) 5. Bobkov, D., Titov, V., Alanov, A., Vetrov, D.: The devil is in the details: StyleFeatureEditor for detail-rich StyleGAN inversion and high quality image editing. In: CVPR (2024) 6. Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., Erhan, D.: Domain separation networks. In: NeurIPS (2016) 7. Carton, F., Louiset, R., Gori, P.: Double InfoGAN for contrastive analysis. In: AISTATS (2024) 8. Gonzalez-Garcia, A., van de Weijer, J., Bengio, Y.: Image-to-image translation for cross-domain disentanglement. In: NeurIPS (2018) 9. Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., Lee, S.: Counterfactual Visual Explanations. In: ICML (2019) 10. He, Y., Lesné, G., Liu, Z., Soumm, M., Gori, P.: Learning common and salient generative factors between two image datasets (2025)

10

Y. He and P. Gori

11. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS (2017) 12. Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: ICLR (2022) 13. Hyvärinen, A., Sasaki, H., Turner, R.E.: Nonlinear ICA using auxiliary variables and generalized contrastive learning. In: AISTATS (2019) 14. Jeanneret, G., Simon, L., Jurie, F.: Diffusion models for counterfactual explanations. In: ACCV (2022) 15. Jeanneret, G., Simon, L., Jurie, F.: Adversarial counterfactual visual explanations. In: CVPR (2023) 16. Jeanneret, G., Simon, L., Jurie, F.: Text-to-image models for counterfactual explanations: a black-box approach. In: WACV. pp. 4757–4767 (2024) 17. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: CVPR (2020) 18. Kazimi, T., Allada, R., Yanardag, P.: Explaining in Diffusion: Explaining a Classifier with Diffusion Semantics. In: CVPR (2025) 19. Kleinman, M., Achille, A., Soatto, S., Kao, J.C.: Gács–körner common information variational autoencoder. In: NeurIPS (2023) 20. Lang, O., Gandelsman, Y., Yarom, M., Wald, Y., Elidan, G., Hassidim, A., Freeman, W.T., Isola, P., Globerson, A., Irani, M., Mosseri, I.: Explaining in style: Training a GAN to explain a classifier in stylespace. In: ICCV (2021) 21. Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., Bachem, O.: Challenging common assumptions in the unsupervised learning of disentangled representations. In: ICML (2019) 22. Louiset, R., Duchesnay, E., Grigis, A., Dufumier, B., Gori, P.: SepVAE: A contrastive VAE to separate pathological patterns from healthy ones. In: MIDL (2024) 23. Louiset, R., Duchesnay, E., Grigis, A., Gori, P.: Separating common from salient patterns with contrastive representation learning. In: ICLR (2024) 24. Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE TMI (2015) 25. Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: SDXL: Improving latent diffusion models for high-resolution image synthesis. In: ICLR (2024) 26. Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., CohenOr, D.: Encoding in style: a StyleGAN encoder for image-to-image translation. In: CVPR (2021) 27. Rodríguez, P., Caccia, M., Lacoste, A., Zamparo, L., Laradji, I., Charlin, L., Vazquez, D.: Beyond trivial counterfactual explanations with diverse valuable explanations. In: ICCV (2021) 28. Sánchez, E.H., Serrurier, M., Ortner, M.: Learning disentangled representations via mutual information estimation. In: ECCV (2020) 29. Schölkopf, B., Kügelgen, J.v.: From Statistical to Causal Learning. In: ICM (2022) 30. Severson, K.A., Ghosh, S., Ng, K.: Unsupervised learning with contrastive latent variable models. In: AAAI (2019) 31. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015) 32. Singla, S., Pollack, B., Chen, J., Batmanghelich, K.: Explanation by progressive exaggeration. In: ICLR (2020)

Counterfactual Contrastive Analysis

11

33. Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: Asilomar Conference on Signals, Systems and Computers (2003) 34. Weinberger, E., Beebe-Wang, N., Lee, S.I.: Moment matching deep contrastive latent variable models. In: AISTATS (2022) 35. Weng, N., Pegios, P., Petersen, E., Feragen, A., Bigdeli, S.: Fast diffusion-based counterfactuals for shortcut removal and generation. In: ECCV (2024) 36. Yang, J., Shi, R., Wei, D., Liu, Z., Zhao, L., Ke, B., Pfister, H., Ni, B.: MedMNIST v2: A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data (2023) 37. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018) 38. Zou, K., Faisan, S., Heitz, F., Valette, S.: Joint disentanglement of labels and their features with VAE. In: ICIP (2022)

12

Y. He and P. Gori

Supplementary Material 5

Architecture Details

5.1

CS Separator

As described in Section 2, the CS separator Hcs,ϕ contains one common branch and one or two salient branches, depending on the contrastive analysis setting. In the background–target (BT) setting, it consists of the common branch Cϕc and the salient branch Sy,ϕsy . In the multiple-salient (MS) setting, an additional salient branch Sx,ϕsx is introduced to capture features specific to dataset X. All branches share the same architecture but have independent parameters. Following He et al. [10], we implement each branch as an MLP adapted from StyleGAN2’s Z-to-W mapping network [17]. Each branch comprises four linear layers with equalized learning-rate scaling, each followed by a LeakyReLU activation with a negative slope of 0.2. Unlike the original StyleGAN2 Z-to-W mapping network, in which each linear layer applies a single 512 × 512 weight matrix to a 512-dimensional latent vector, each linear layer in our separator is parameterized by a weight tensor A(k) ∈ RL×512×512 to process the W + code w = [w1 , . . . , wL ]⊤ ∈ RL×512 . Here, k = 1, . . . , 4 indexes the linear layer, and L denotes the number of style vectors in the W + code. For a generator with a native output resolution of R × R, L is given by L = 2 log2 (R) − 2. In our experiments, all StyleGAN2 generators are pretrained at a native resolution of 256 × 256, yielding L = 14. This design assigns a position-specific weight matrix to each style position. Since each layer preserves the dimensionality of the latent code, it maps from RB×L×512 to RB×L×512 , where B denotes the batch size. 5.2

Regularization Networks D and R

As described in Section 2, we employ a domain discriminator D and a dependency regressor R to regularize the common and salient representations. Given a common representation c ∈ RL×512 , the domain discriminator first flattens the complete representation into a single vector. Since our StyleGAN2 generator operates at a native resolution of 256 × 256, we have L = 14, resulting in a 7168-dimensional input vector. The discriminator is implemented as a binary linear classifier with a single fully connected layer mapping the 7168-dimensional input to one output logit. No hidden layer or output activation is used, and the network directly produces a single domain-classification logit. The dependency regressor R predicts the salient representation from the corresponding common representation. Given a batch of common representations c ∈ RB×L×512 , the batch and style dimensions are merged to form a matrix in R(BL)×512 . The regressor consists of two fully connected layers that map this matrix from (BL) × 512 to (BL) × 512, with a ReLU activation between the two layers and no activation after the output layer. The output is then reshaped back to obtain the predicted salient factor ŝ ∈ RB×L×512 .

Counterfactual Contrastive Analysis

5.3

13

Other Architectures and Implementations

For the remaining network components, we use the pSp architecture [26] for E and the StyleGAN2 architecture [17] for G. The F-space refinement stage builds upon SFE [5], retaining the original architectures of its F-space encoder Ef and trainable adapter FAdpt . For the original StyleGAN2 Z-to-W mapping network on which the separator architecture is based, we refer to the implementation provided at https://nn.labml.ai/gan/stylegan/index.html. For the StyleGAN2 generator G, we use the PyTorch implementation by rosinality, available at https://github.com/rosinality/stylegan2-pytorch. The same pretrained G is used throughout both training stages and evaluation. To compute Ladv , we use the discriminator pretrained jointly with G. For image-level evaluation, we adopt the U-Net-based classifier used in [35]. The classifier is trained on real images from the corresponding training set and subsequently applied to the edited images to assess whether they exhibit the intended target attributes. For latent-space evaluation, we quantify the separation between the learned common and salient factors by training a logistic regression classifier to predict the dataset label from each factor. The classifier is implemented in scikit-learn. Each factor is flattened and normalized before being provided to the classifier.

6

Additional Results

Fig. 5 illustrates the effects of different training components on salient-factor swapping. With only the base reconstruction objectives (W/o Reg.), the swaps produce limited class-specific changes. Adding LD improves tumor removal in the y → x direction but not tumor addition in the x → y direction. Adding LR improves common–salient separation, enabling clearer bidirectional edits while better preserving the source anatomy. Finally, F-space refinement enhances local details and image fidelity without changing the semantic effect of the edit. These results are consistent with the quantitative ablation in Table 3. Fig. 6 compares the BT and MS assumptions on OCTMNIST. While both variants reconstruct the inputs well, the BT variant produces limited swapping effects because it cannot separately model the class-specific patterns of DME and drusen. The proposed MS variant, with two salient branches, enables more meaningful bidirectional swaps while preserving the common retinal structure. Figs. 7–10 show additional reconstruction and salient-factor swapping results. Each pair of columns, separated by dashed lines, corresponds to an input pair (x, y). From top to bottom, the rows show the real images, their reconstructions, and the swapping results. The swapped images are obtained by combining the common representation of each source image with the salient representation of the other image. Figs. 7–9 use the BT assumption, whereas Fig. 10 uses the MS assumption. Under the BT assumption, the salient factors of x is absent, whereas the MS assumption models separate salient factors for both x and y. Figure 11 provides additional interpolation examples on BraTS and OCTMNIST.

14

Y. He and P. Gori

Fig. 5. Qualitative ablation study of the training components on BraTS. The first column shows inputs x ∈ X and y ∈ Y from the healthy and tumor datasets, respectively. Columns 2–5 show the corresponding salient-factor swapping results under different training configurations.

Fig. 6. Comparison of the BT and MS assumptions on OCTMNIST. The input images x and y are sampled from the DME and drusen datasets, respectively. The remaining columns show the reconstruction and salient-factor swapping results under the BT and MS assumptions.

Fig. 7. Additional examples on BloodMNIST under the BT assumption. The input images x and y are sampled from the eosinophil and neutrophil datasets, respectively.

Counterfactual Contrastive Analysis

15

Fig. 8. Additional examples on BraTS under the BT assumption. The input images x and y are sampled from the healthy and tumor MRI scans, respectively.

Fig. 9. Additional examples on OCTMNIST under the BT assumption. The input images x and y are sampled from the normal and choroidal neovascularization (CNV) datasets, respectively.

Fig. 10. Additional examples on OCTMNIST under the MS assumption. The input images labeled x and y are sampled from the DME and drusen datasets, respectively.

16

Y. He and P. Gori

Fig. 11. Additional interpolation results on BraTS and OCTMNIST. The upper two rows correspond to a healthy image x and a tumor image y from BraTS, and the lower two rows to a normal image x and a CNV image y from OCTMNIST. The first row of each pair interpolates from sx to sy while preserving cx , and the second interpolates from sy to sx while preserving cy .

Related documents

• Context-indexed counterfactuals and non-vacuous counterpossibles#761316
• The Narrow Ontic Counterfactual Account of Distinctively Mathematical Explanation#761508
• Counterfactual explanations as interventions in latent space#478134
• Targeted Visual Counterfactual Explanations for Contrastive Vision-Language Model#1143901
• Chaotic and complex dynamics expose the limits of counterfactual reasoning.#258009