Learning Quantifiable Visual Explanations Without Ground-Truth
arXiv:2605.18681v1 [cs.AI] 18 May 2026
Amritpal Singh1,2[0009−0006−1960−2548] , Andrey Barsky1,2[0000−0002−6993−5969] , Mohamed Ali Souibgui1[0000−0003−0100−9392] , Ernest Valveny1,2[0000−0002−0368−9697] , and Dimosthenis Karatzas1,2[0000−0001−8762−4454] 1 2
Computer Vision Center, Barcelona, Spain Autonomous University of Barcelona, Spain
Abstract. Explainable AI (XAI) techniques are increasingly important for the validation and responsible use of modern deep learning models, but are difficult to evaluate due to the lack of good ground-truth to compare against. We propose a framework that serves as a quantifiable metric for the quality of XAI methods, based on continuous input perturbation. Our metric formally considers the sufficiency and necessity of the attributed information to the model’s decision-making, and we illustrate a range of cases where it aligns better with human intuitions of explanation quality than do existing metrics. To exploit the properties of this metric, we also propose a novel XAI method, considering the case where we fine-tune a model using a differentiable approximation of the metric as a supervision signal. The result is an adapter module that can be trained on top of any black-box model to output causal explanations of the model’s decision process, without degrading model performance. We show that the explanations generated by this method outperform those of competing XAI techniques according to a number of quantifiable metrics. Keywords: Computer Vision · Model Interpretability Evaluation · XAI.
1
Introduction
Recent deep learning (DL) models have achieved outstanding success in a range of predictive tasks, especially those that have historically been out of reach of conventional statistical models and pattern recognition systems with handengineered features. However, this success has come at the cost of model interpretability. DL models are often called “black boxes” due to their complex and multivariate decision processes, which cannot be directly explained on a human level. In many applications this is not a critical weakness, but it poses an obstacle to the deployment of such systems where accountability and transparency are key considerations. This challenge only grows as AI systems scale up, both in terms of model architecture and widespread adoption. Historically, a range of explainable AI (XAI) techniques have been used to provide human-interpretable explanations for the output of DL models [18].
2
A. Singh et al.
In the visual domain, these often take the form of pixel-level saliency maps that estimate the contribution of each input pixel to the model’s decision process. Such methods are widely used in debugging visual models during training, to ensure that the patterns they are learning represent real semantic features and are not overfitting to spurious data correlations. Yet despite the availability of sophisticated techniques for explaining model outputs, it remains difficult to adequately quantify the suitability of any given explanation [26]. An open challenge in the XAI research field is how to appropriately rank explanations against each other, given a model and its output on some example of interest. The field lacks consensus on an appropriate metric for explanation quality. This challenge is partly due to the lack of meaningful ground-truth, and partly to how explanations serve diverse, often conflicting purposes in the ML pipeline. Unlike predictions at inference time, which can be objectively validated, explanations exist primarily for human use. Their usefulness depends on context and specific use case - whether debugging, auditing or supporting human-led decisions. As such, what makes a “good" explanation varies across applications, which has contributed to the lack of a standardized evaluation metric. Of the methods that exist to quantify the quality of model explanations, the most prominent rely on perturbation of the input example according to local saliency scores [2]. However, we find that the scores given by such metrics often do not align with intuitive notions of explanation quality - for example, by giving high scores to explanations that cover large, irrelevant regions, or that focus on specific, narrow features. In this paper, we propose the definition of a new perturbation-based metric called Minimality-Sufficiency Integration (MSI), grounded in the information bottleneck framework [25], designed to favour explanations that are simultaneously specific and parsimonious. We also introduce a novel XAI technique that generates explanations according to the requirements specified by the above metric. This method, called Learnable Adapter eXplanation (LAX), consists of training a self-supervised explanation module over an existing pre-trained model, without ground-truth annotations. We discuss the implementation of this approach in later sections.
2
Related Work
2.1
Explainability in Vision Models
XAI methods can be broadly divided into three categories [18]: visualizationbased approaches, distillation approaches, and intrinsic approaches. In the visual domain, visualization approaches are the most commonly used [21, 22]. The generated saliency map explanation represents a matrix of relevance scores associated to the input pixels, intended to give higher scores to relevant pixels and lower scores to irrelevant ones. The most common approaches to generate these saliency maps are gradient-based, for example, Grad-CAM [22], Layer-CAM [13],
Learning Quantifiable Visual Explanations Without Ground-Truth
3
and Finer-CAM [29], alongside various iterations of similar approaches[4, 23]. The shared idea behind these methods is to calculate saliency maps by propagating gradients or relevance scores from the output layer back through the network to estimate the contribution of each input feature. However, they are susceptible to gradient shattering [5], which can make their attributions inconsistent or unstable. More recently, Vision Transformers (ViTs) [10] have motivated attentionbased explainability methods that exploit the self-attention mechanism. In this context, attention rollout [1] aggregates attention matrices across layers to estimate the global flow of information from input patches to the output token. Other approaches are perturbation-based, where the idea is to observe changes in the output by perturbing different regions of the input image, thus inferring the importance of each region based on the impact of its perturbation. Examples include occlusion sensitivity [28], LIME [19] and RISE [17]. While these approaches overcome the gradient shattering problem, they are computationally expensive, as they require evaluating a large number of perturbed input images for each explanation. 2.2
Learning To Explain
An alternative to post-hoc methods is to learn explanations during the model training process [18]. Thus, the vision model is explicitly trained to generate saliency map explanations alongside its predictions, allowing the process to be more efficient and faithful [6, 2]. These models generate the explanation in a single pass, avoiding the need for repeated perturbations at inference time. However, since most datasets lack ground-truth saliency maps, these models often rely on auxiliary supervision or priors to guide the explanation learning process [8, 24]. 2.3
Quantifying Visual Explanations
As discussed, evaluating explanation quality remains challenging due to the lack of ground-truth saliency maps in most datasets [32]. A common evaluation strategy is fidelity-based testing [2, 26]. Under this framework, removing important (high-saliency) pixels should degrade the model’s performance, while removing unimportant (low-saliency) pixels should have minimal impact. This can be done in two directions: MoRF (Most Relevant First) and LeRF (Least Relevant First), which progressively perturb the most and least relevant regions; and by deletion (from full image) or insertion (from a blank image). There exist other methods that utilise the specific performance curves produced by iterative perturbation, such as the "area under AIC/SIC metric" used in XRAI [14]. However, direct pixel perturbation can introduce a domain shift from the original training distribution, potentially confounding the results. To mitigate this, ROAR [12] was proposed, which retrains the model from scratch on data where specific pixels have been removed. While this reduces the impact of domain shift, it is computationally expensive, introduces new biases, and operates only on the dataset level without any ability to quantify individual explanations
4
A. Singh et al.
[20]. To address these limitations, alternative metrics such as R-fidelity and Ffidelity [31, 30] have been proposed. These methods perturb only partial sets of relevant and irrelevant pixels and may include optional fine-tuning to preserve model stability, as does IDSDS [11]. As a result, they offer improved robustness to domain shift and reduced reliance on retraining. However, they nonetheless rely on altering the model that is to be explained, and face problems distinguishing between competing maps (see below).
3
Method
3.1
Proposed Metric
We observe that existing perturbation-based fidelity metrics find difficulty with: (i) Handling large masks. As shown in Table 1, we consider two possible explainability heatmaps over the same input image. In both cases, feeding only the relevant pixels (in red) to the model produces a correct prediction. However, the second mask highlights more information than necessary, including features clearly not used for prediction. Both masks have near-identical scores according to insertion and deletion metrics, failing to reward the more focused explanation. Table 1: Comparison of two valid masks on the same image from the Synthetic-MNIST dataset according to several metrics, including our proposed MSI. MoRF (Pixel Value) Visual
MoRF (% Value)
Fidelity (Pixel Value)
Fidelity (% Value)
Insertion ↑ Deletion ↓ Insertion ↑ Deletion ↓ Fid- (Ins.) ↓ Fid+ (Del.) ↑ Fid- (Ins.) ↓ Fid+ (Del.) ↑
OURS Base Mask MSI Score ↑ Penalty ↓ Score ↑ (αmin = 0.5)
0.981
0.118
0.889
0.164
0.018
0.876
0.109
0.83
0.864
0.247
0.617
0.986
0.118
0.889
0.164
0.011
0.876
0.109
0.83
0.868
0.887
−0.019
(ii) Handling multiple plausible explanations. Figure 1 demonstrates a scenario where both the relevance mask and its complement produce correct predictions when applied to the same input, due to the presence of more than one valid explanation for the prediction. Metrics like MoRF-Deletion implicitly assume that there is a single correct explanation. Here, such metrics give a poor score to this mask, even though the mask correctly captures one of several valid attribution pathways. We propose a single robust metric that addresses both limitations by balancing the minimality and sufficiency of proposed attributions to a model’s output, incentivizing compactness and robustness to multiple informative regions. We name this metric Minimality-Sufficiency Integration (MSI). Minimality-Sufficiency Integration. MSI operates over an attribution heatmap normalized to the range [0, 1], provided as an explanation for the output of a
Learning Quantifiable Visual Explanations Without Ground-Truth
5
Fig. 1: Example illustrating multiple valid solutions. From left to right: original image,
full mask, thresholded mask region (values ≥ 0.5), and its complement (values < 0.5).
model on a given input example. It is computed as the sum of a base score and a mask size penalty. The base score is defined as: 1h (Show(> αmin ) − Show(< αmin )) BaseScoreavg = 2 i avg + (AUCavg (1) show − AUChide ) The threshold parameter αmin ∈ [0, 1] binarises the heatmap into “relevant“ and “non-relevant“ regions. The first term compares predictive performance when masking these regions: – Show(> αmin ): Prediction accuracy when retaining only pixels with heatmap values greater than αmin . – Show(< αmin ): Prediction accuracy when retaining only pixels with values less than or equal to αmin (i.e., the complement region). A greater difference between these two scores indicates stronger separation of important and unimportant regions. This captures the case where multiple plausible explanations exist by pushing toward a correct separation between all the relevant regions and the non-relevant pixels. The second term is based on area under the perturbation curve: Z 1.0 1 AUCavg = Show(> α) dα (2) show ∆α αmin Z 1.0 1 AUCavg = Hide(> α) dα (3) hide ∆α αmin where ∆α = 1.0 − αmin
(4)
– Show(> α) is the prediction accuracy when only the pixels with heatmap values greater than α are retained. This process is analogous to MoRF-Insertion, where pixels are progressively inserted in decreasing order of importance according to the heatmap values—for example, by revealing all pixels above 0.98, then 0.96, and so on. However, instead of continuing all the way to α = 0, the integration is truncated at αmin .
6
A. Singh et al.
– Hide(> α) is the prediction accuracy when pixels with heatmap values below αmin are first removed, and then the remaining pixels are progressively deleted in order of decreasing importance, i.e., from values close to 1 down to αmin . This is analogous to (truncated) MoRF-Deletion. The other term in the MSI metric is the mask size penalty. As discussed earlier, a heatmap can highlight more pixels than necessary while still allowing the model to make correct predictions. In such cases, a smaller, more focused mask that conveys the same predictive information is preferable. To penalise unnecessarily large masks, we define the mask size penalty as: MaskPenalty =
n h i 1X 1 1 M(i) ≥ αmin n i=1 h · w 1
(5)
Here, Mi ∈ Rh×w denotes the heatmap for the i-th input, and 1[·] is the indicator function that returns 1 for entries satisfying the condition inside the brackets, 0 otherwise. Specifically, this term measures the proportion of pixels with values greater than or equal to the threshold αmin , i.e., those considered important. A lower mask penalty indicates a more compact and focused explanation, aligning with the principle of minimality. The final MSI score combines the base score, which reflects sufficiency and discriminative quality, with the mask penalty, which enforces minimality. The final MSI score is given by: MSI = BaseScore − MaskPenalty
(6)
This score is sensitive to the choice of αmin hyperparameter, which depends in turn on the representations learned by the model. In practice, it is simple to optimise this choice for a given model on a given dataset by searching through possible values and selecting the one that maximises final MSI. Calculating the quality of visual attributions in this way produces scores that better distinguish between valid explanations that differ in parsimony. We refer to the quantitative results in Table 2 for a comparison of MSI against existing fidelity metrics. However, our key contribution in this paper is in developing an explanation generation method that aims to directly maximise this metric, detailed in the following section. 3.2
Proposed Self-Explainable Approach
Building on the minimality-sufficiency logic of the MSI metric, we propose Learnable Adapter eXplanation (LAX), a self-supervised method that learns to generate explanations for a given model without requiring ground-truth annotations. While previous work [8, 24] has tackled similar objectives, they typically rely on auxiliary signals that serve as a proxy for ground truth to guide the mask generation process. To define our objective formally, we draw inspiration from the Information Bottleneck (IB) framework [25]. The core idea is to learn a representation that
Learning Quantifiable Visual Explanations Without Ground-Truth
7
is both minimal —containing as little information as possible from the input X as possible—and sufficient—preserving the ability to accurately predict the output y. Our goal is to learn a mask M such that the masked input T = X⊙M retains only the most relevant information needed for prediction. In our case, this translates to two information-theoretic goals: minimizing the mutual information I(X, T) to enforce minimality, and maximizing I(T, y) to ensure sufficiency. These can be formulated as a single minimization objective: L = I(X, T) − β · I(T, y), where β is a hyperparameter that balances the trade-off between compression and predictive power.
Fig. 2: Overview of the proposed LAX framework with respect to a frozen, pre-trained model. (1) The image is processed by the feature extractor to obtain feature representations. (2) These spatial features are sent to both the output MLP for classification and to the explanation module. (3) The MLP produces the original class prediction, while the explanation module generates a corresponding heatmap. (4) The original image is multiplied with the heatmap to produce a masked image. (5) The masked image is passed through the frozen feature extractor and MLP. (6) The model generates a new prediction based on the masked image.
Figure 2 provides a visual overview of our proposed method. More formally, we describe the specific steps in Algorithm 1. The core idea is to augment any pre-trained black-box model with an explanation module that learns to generate visual explanations as pixel-wise importance heatmaps directly. As a supervision signal, we use the model’s accuracy on the input example masked with the proposed heatmap, backpropagated through the explanation module. Our method operates from a pre-trained base model on the target task. We assume that the base model comprises a feature extraction backbone and a classification head, but otherwise the method is architecture-agnostic. The base
8
A. Singh et al.
model’s weights are frozen, and we initialise the new explanation module as a learnable projection from the backbone’s feature representations. The explanation block is modular by design and can adopt various architectures. In our implementation, it is a small Convnet, chosen to efficiently preserve spatial features. This explanation module generates a heatmap in a lower-resolution feature space. Although it is possible to generate a mask at the same resolution as the input image, we empirically found that learning a low-resolution mask and upsampling leads to better and more stable performance. To ensure interpretability, the heatmap values must lie in the range [0, 1], where a value of 0 indicates that a pixel is unimportant and a value of 1 denotes high importance. To achieve this, we apply a sigmoid activation function to the raw heatmap logits (before upsampling). Algorithm 1 LAX Framework Require: Input image X, frozen black-box model f = fMLP ◦ ffeat , learnable explanation module g Ensure: Mask M, masked image T, prediction ŷ 1: Step 1: Extract features: F ← ffeat (X) 2: Step 2: (a) Predict class: ŷorig ← fMLP (F) (b) Generate heatmap: M ← g(F) 3: Step 3: Upsample heatmap M to match input resolution (if needed) 4: Step 4: Compute masked image: T ← X ⊙ M 5: Step 5: Predict from masked image: ŷ ← fMLP (ffeat (T)) 6: return Explanation heatmap M, masked image T, prediction ŷ
As previously introduced, the IB-based loss involves two mutual information terms: (1) I(X, T), which we minimise to enforce minimality, and (2) I(T, y), which we maximise to ensure sufficiency. Optimizing these terms directly is generally intractable; therefore, we propose tractable approximations for each term. Minimizing I(X, T) via entropy regularization. We define T = X ⊙ M, where M is the learnable mask and X is the fixed input. Since X remains constant, the only way to reduce the mutual information I(X, T) is by limiting the information retained in T, effectively encouraging the mask M to be sparse. A common approach is to apply an ℓ1 regularization on M, which promotes sparsity. While this method is effective to some extent, it often produces masks that are less sharp and harder to interpret. In contrast, we opt to use an entropy-based regularization, which yields sharper and more semantically meaningful masks that concentrate more on important regions. Our entropy-based loss is formulated as follows: n 1 X X Lentropy = − Pi,j log(Pi,j + ε) , (7) n i=1 j
Learning Quantifiable Visual Explanations Without Ground-Truth
9
Table 2: Qualitative and quantitative comparison between LAX and Grad-CAM using CNN-based model on randomly chosen examples from different datasets. The threshold αmin is 0.5 for MNIST and CUB-200, and 0.4 for CIFAR-10. Original
Visual LAX
Grad CAM
MoRF (Pixel Value) MoRF (% Value) Fidelity (Pixel Value) Fidelity (% Value) Ours Insertion ↑ Deletion ↓ Insertion ↑ Deletion ↓ Fid- ↓ Fid+ ↑ Fid- ↓ Fid+ ↑ BaseScore MaskPenalty MSI Score ↑ LAX Grad LAX Grad LAX Grad LAX Grad LAX Grad LAX Grad LAX Grad LAX Grad LAX Grad LAX Grad LAX Grad CAM CAM CAM CAM CAM CAM CAM CAM CAM CAM CAM 0.745 0.605 0.125 0.175
0.91
0.53
0.07
0.15
0.275 0.393 0.886 0.818 0.106 0.487 0.938 0.847
0.68 −0.015 0.185 0.553 0.495 −0.568
0.965 0.565
0.615 0.585
0.89
0.71
0.39
0.45
0.303 0.498 0.4799 0.497 0.256 0.375 0.607 0.598
0.46 −0.11 0.351 0.37
0.965 0.875 0.075 0.255
0.93
0.97
0.09
0.07
0.008 0.086 0.802 0.676
0.66
0.955 0.765
0.75
0.77
0.31
0.25
0.115 0.249
0.565 0.255
0.022 0.006 0.772 0.817
0.109 −0.48
0.54 0.204 0.218 0.456 0.322
0.488 0.715 0.304 0.243 0.675 0.722 0.943 0.608 0.318 0.73
0.625 −0.122
where Pi = softmax Mi ∈ Rh×w
max(0, Mi ) t
,
is the mask for sample i,
t is a temperature parameter controlling the sharpness, ε is a small constant for numerical stability, n is the batch size. Minimizing this entropy loss results in heatmaps that assign high confidence primarily to the most critical regions, effectively enforcing minimality. Maximizing I(T, y) via cross-entropy approximation. To ensure that the masked input T = X ⊙ M retains sufficient information for accurate prediction, we maximise the mutual information I(T, y). Using the approximation introduced by [3], we have: max I(T; y) = H(y) −H(y | T) | {z } constant
⇐⇒
min H(y | T) ≈ LCE (y, ŷ)
(8)
where H(y) and H(y | T) represent the marginal and conditional entropies of y, respectively. Here, y denotes the ground-truth label, and ŷ denotes the model’s prediction on the masked input. Note that H(y) is constant because the label distribution is fixed. Final objective. Combining both terms, the LAX module final loss is: min LLAX = LCE (y, ŷ) + λ Lentropy (M),
(9)
where λ is a hyperparameter balancing prediction accuracy and mask sparsity. Minimizing this objective enables the explanation module to learn compact,
10
A. Singh et al.
informative, and interpretable masks without reliance on ground-truth explanations. Table 3: Quantitative results on the Synthetic MNIST dataset using CNN-based mod-
els. LAX outperforms all baseline methods across all standard metrics and the proposed MSI metric, indicating strong minimality and sufficiency under low-ambiguity settings. Method
MoRF (Pixel Value)
MoRF (% Value)
Fidelity (Pixel Value)
Fidelity (% Value)
Insertion ↑ Deletion ↓ Insertion ↑ Deletion ↓ Fid- (Ins.) ↓ Fid+ (Del.) ↑ Fid- (Ins.) ↓ Fid+ (Del.) ↑
Ours BaseScore MaskPenalty MSI Score (αmin = 0.5)
GradCAM
0.744
0.250
0.816
0.198
0.255
0.745
0.183
0.798
0.509
0.418
0.091
GradCAM++
0.712
0.268
0.804
0.204
0.287
0.728
0.194
0.792
0.456
0.371
0.085
LayerCAM
0.716
0.266
0.805
0.203
0.282
0.73
0.194
0.793
0.462
0.378
0.084
KPCACAM
0.612
0.329
0.825
0.201
0.388
0.666
0.175
0.794
0.316
0.187
0.128
FinerCAM
0.616
0.347
0.742
0.244
0.382
0.649
0.256
0.751
0.301
0.318
−0.027
LAX
0.917
0.142
0.906
0.162
0.085
0.851
0.092
0.833
0.787
0.205
0.582
4
Experiments and Results
Experimental Setup. To evaluate our proposed method, we compare it against several existing baseline techniques for explanation generation, using both existing metrics and our proposed MSI metric. For MSI, we report three separate components: the base score, the mask penalty, and the final MSI score. Each heatmap is processed according to the requirements of the respective evaluation metric (e.g., insertion, deletion, Fid, MSI) to compute the corresponding score. Baseline Methods. We compare our LAX method against several state-of-theart techniques for generating explanation heatmaps. For CNN-based models, we consider Grad-CAM [22], Grad-CAM++ [7], Layer-CAM [13], KPCA-CAM [15], and Finer-CAM [29]. For Vision Transformers, we evaluate Grad-CAM [22] and attention rollout [1]. Model. For CNN-based experiments, we use a ResNet18 architecture initialized with pretrained weights and trained for 500 epochs using standard cross-entropy loss. After training the base classifier on the classification task, we initialise and train the LAX module for an additional 500 epochs, while keeping the base model frozen. The optimization objective consists of the classification loss computed on the masked input, combined with an entropy-based regularization term. We also monitor the average value of the generated masks, which assists in hyperparameter tuning, particularly in selecting the weighting coefficient λ for the entropy loss. For Vision Transformer (ViT) experiments, we use the vit_b_16 architecture initialized with pretrained weights. The model is first fine-tuned on the target task for 50 epochs using cross-entropy loss. Subsequently, the LAX module is trained following the same procedure as in the CNN-based setting, except that the number of training epochs is 350.
Learning Quantifiable Visual Explanations Without Ground-Truth
11
Baseline Metrics. To assess the quality of different explanation methods, we employ several widely used evaluation metrics in addition to our proposed MSI score. These include Insertion and Deletion [17], as well as Fid-Insertion and Fid-Deletion [30]. Each metric evaluates the model’s response as information is either progressively added to or removed from the input based on the explanation heatmap. We report two variants for each metric: Pixel-based and percentagebased. Datasets. We evaluate our method across three datasets selected to reflect varying levels of complexity. In the simplest case, we use an augmented variant of MNIST [9] which randomly repositions target digits and adds background noise to make explanation generation non-trivial. We also use CIFAR-10 [16] as a standard evaluation dataset, and CUB-200 [27] for a higher-resolution setting. Metric Evaluation. Table 2 shows the behavior of the various metrics on heatmaps produced by Grad-CAM and by our method (LAX). Notably, MSI is the most consistent and amplifies differences when explanations differ substantially in quality. For example, on the first sample, LAX outperforms Grad-CAM by over 1.0 point in MSI, whereas Insertion and Deletion differ by only 0.15. This indicates that MSI is more sensitive to qualitative differences, rewarding explanations that are both sufficient and compact. Furthermore, MSI values for Grad-CAM remain negative or close to zero across several examples, aligning with visual impressions of noisy or diffuse saliency maps, while LAX consistently yields positive MSI scores that reflect minimal and sufficient regions. Quantitative Method Evaluation. The quantitative results of the different explainability methods are summarized in Table 3, Table 4, and Table 5, corresponding to the Synthetic MNIST, CUB-200, and CIFAR-10 datasets, respectively, for CNN-based models. Results for the Vision Transformer (ViT) on CIFAR-10 are reported in Table 6. Qualitative comparisons are shown in Figures 3,and 4. For CNN-based models, LAX consistently outperforms the baseline methods on Synthetic MNIST across all evaluation metrics, achieving higher MSI scores due to minimal penalties on mask size. On CUB-200, conventional metrics are inconclusive in distinguishing between Grad-CAM, Finer-CAM, and LAX; however, LAX attains a higher MSI score despite producing masks of comparable size, indicating a more discriminative focus. On CIFAR-10, LAX leads in insertion-based metrics, while Grad-CAM achieves better deletion scores. This discrepancy suggests the presence of multiple informative regions, which is more effectively captured by the MSI metric. Across all datasets, MSI proves valuable in differentiating between methods that appear similar under traditional evaluation metrics. For ViT models on CIFAR-10, LAX again achieves superior performance in insertion-based metrics, whereas attention rollout yields better deletion scores. Notably, LAX attains higher MSI values compared to competing methods. Moreover, the MSI scores on CIFAR-10 for ViT models are close to zero, consistent
12
A. Singh et al.
with the CNN-based results, indicating the existence of multiple plausible explanatory solutions. Table 4: Quantitative results on the CUB-200 (Birds) dataset using CNN-based mod-
els. While several methods perform similarly on standard metrics, LAX achieves the highest MSI score, showing success in optimizing for this metric. Method
MoRF (Pixel Value)
MoRF (% Value)
Fidelity (Pixel Value)
Fidelity (% Value)
Insertion ↑ Deletion ↓ Insertion ↑ Deletion ↓ Fid- (Ins.) ↓ Fid+ (Del.) ↑ Fid- (Ins.) ↓ Fid+ (Del.) ↑
Ours Base Score Mask Penalty MSI Score (αmin = 0.5) ↑ ↓ ↑
GradCAM
0.665
0.412
0.768
0.284
0.198
0.421
0.103
0.521
0.244
0.264
−0.019
GradCAM++
0.654
0.431
0.763
0.291
0.210
0.411
0.109
0.517
0.230
0.256
−0.026
LayerCAM
0.662
0.414
0.763
0.291
0.203
0.422
0.112
0.517
0.243
0.275
−0.031
KPCACAM
0.565
0.556
0.757
0.318
0.294
0.315
0.114
0.505
0.058
0.148
−0.091
FinerCAM
0.646
0.455
0.774
0.295
0.218
0.383
0.099
0.504
0.197
0.221
−0.023
LAX
0.767
0.415
0.772
0.317
0.137
0.426
0.101
0.491
0.445
0.265
0.18
Table 5: Quantitative results on the CIFAR-10 dataset using CNN-based models. MSI is able to rate a method highly even when multiple plausible masks exist. Method
MoRF (Pixel Value)
MoRF (% Value)
Fidelity (Pixel Value)
Fidelity (% Value)
Insertion ↑ Deletion ↓ Insertion ↑ Deletion ↓ Fid- (Ins.) ↓ Fid+ (Del.) ↑ Fid- (Ins.) ↓ Fid+ (Del.) ↑
Ours Base Score Mask Penalty MSI Score (αmin = 0.4) ↑ ↓ ↑
GradCAM
0.637
0.384
0.668
0.339
0.301
0.55
0.272
0.593
0.374
0.599
−0.225
GradCAM++
0.620
0.395
0.662
0.342
0.317
0.541
0.278
0.590
0.353
0.576
−0.222
LayerCAM
0.626
0.387
0.659
0.341
0.311
0.547
0.281
0.591
0.364
0.596
−0.232
KPCACAM
0.371
0.674
0.596
0.447
0.566
0.268
0.343
0.489
−0.094
0.272
−0.367
FinerCAM
0.621
0.411
0.674
0.352
0.316
0.525
0.266
0.581
0.347
0.553
−0.206
LAX
0.739
0.581
0.735
0.423
0.207
0.364
0.201
0.516
0.345
0.337
0.007
Fig. 3: Qualitative examples on CUB-200 (left) and CIFAR-10 (right) with CNN-based models.
5
Conclusion
In this paper, we introduced Minimality-Sufficiency Integration (MSI), a novel metric for quantifying the quality of visual explanations without relying on ground-truth saliency annotations. MSI addresses key limitations of existing
Learning Quantifiable Visual Explanations Without Ground-Truth
13
Table 6: Quantitative results on the CIFAR-10 dataset using ViT. MSI is able to rate a method highly even when multiple plausible explanations exist. Method
MoRF (Pixel Value)
MoRF (% Value)
Fidelity (Pixel Value)
Fidelity (% Value)
Insertion ↑ Deletion ↓ Insertion ↑ Deletion ↓ Fid- (Ins.) ↓ Fid+ (Del.) ↑ Fid- (Ins.) ↓ Fid+ (Del.) ↑ GradCAM
0.340
0.774
Attention Rollout
0.587
0.613
0.383
LAX
0.856
0.725
0.611
0.780
0.857
Ours Base Score Mask Penalty MSI Score (αmin = 0.5) ↑ ↓ ↑
0.397
0.375
−0.186
0.297
−0.484
0.256
0.372
0.487
−0.298
0.224
−0.5225
0.217
0.131
0.448
0.380
0.277
0.102
0.645
0.208
0.498
0.599
0.553
0.134
Fig. 4: Qualitative example on CIFAR-10 with ViT.
fidelity-based metrics by jointly evaluating the sufficiency and minimality of explanations, while also being sensitive to multiple valid explanations. Our experiments show that MSI offers more consistent and interpretable scores across varying tasks and datasets. To complement this metric, we proposed LAX (Learnable Adapter eXplanation), an explanation module that learns to generate compact and informative saliency maps by directly optimizing toward the MSI objective. LAX operates as a lightweight adapter that is model-agnostic and requires no ground-truth explanations, making it broadly applicable. Quantitative and qualitative evaluations across three diverse datasets confirm that LAX outperforms established baselines on various metrics, including MSI. Future work will explore extending both MSI and LAX to multi-modal data and large vision and language models. Acknowledgements This work was funded by the European Union - by Next Generation EU, under Grant No. SDC007/25/000075; by ELSA – European Lighthouse on Secure and Safe AI under Grant Agreement No. 101070617; and by the Horizon Europe research and innovation programme under Grant Agreement No. 101214398 (ELLIOT).
Bibliography
[1] Abnar, S., Zuidema, W.: Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00928 (2020) [2] Alvarez Melis, D., Jaakkola, T.: Towards robust interpretability with selfexplaining neural networks. Advances in neural information processing systems 31 (2018) [3] Amjad, R.A., Geiger, B.C.: Learning representations for neural network-based classification using the information bottleneck principle. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(9), 2225–2239 (2019) [4] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one 10(7), e0130140 (2015) [5] Balduzzi, D., Frean, M., Leary, L., Lewis, J., Ma, K.W.D., McWilliams, B.: The shattered gradients problem: If resnets are the answer, then what is the question? In: International conference on machine learning. pp. 342–350. PMLR (2017) [6] Bang, S., Xie, P., Lee, H., Wu, W., Xing, E.: Explaining a black-box by using a deep variational information bottleneck approach. In: Proceedings of the AAAI conference on artificial intelligence. vol. 35, pp. 11396–11404 (2021) [7] Chattopadhay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Gradcam++: Generalized gradient-based visual explanations for deep convolutional networks. In: IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 839–847 (2018) [8] Choi, C., Yu, S., Kampffmeyer, M., Salberg, A.B., Handegard, N.O., Jenssen, R.: Dib-x: Formulating explainability principles for a self-explainable model through information theoretic learning. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 7170–7174 (2024) [9] Deng, L.: The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine 29(6), 141–142 (2012) [10] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021), https: //arxiv.org/abs/2010.11929 [11] Hesse, R., Schaub-Meyer, S., Roth, S.: Benchmarking the attribution quality of vision models. Advances in Neural Information Processing Systems 37, 97928– 97947 (2024) [12] Hooker, S., Erhan, D., Kindermans, P.J., Kim, B.: A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems 32 (2019) [13] Jiang, P.T., Zhang, C.B., Hou, Q., Cheng, M.M., Wei, Y.: Layercam: Exploring hierarchical class activation maps for localization. IEEE transactions on image processing 30, 5875–5888 (2021) [14] Kapishnikov, A., Bolukbasi, T., Viégas, F., Terry, M.: Xrai: Better attributions through regions. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4948–4957 (2019)
Learning Quantifiable Visual Explanations Without Ground-Truth
15
[15] Karmani, S., Sivakaran, T., Prasad, G., Ali, M., Yang, W., Tang, S.: Kpca-cam: Visual explainability of deep computer vision models using kernel pca. arXiv preprint arXiv:2410.00267 (Sep 2024). https://doi.org/10.48550/arXiv.2410.00267 [16] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009) [17] Petsiuk, V.: Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421 (2018) [18] Ras, G., Xie, N., Van Gerven, M., Doran, D.: Explainable deep learning: A field guide for the uninitiated. Journal of Artificial Intelligence Research 73, 329–396 (2022) [19] Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you? explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1135–1144 (2016) [20] Rong, Y., Leemann, T., Borisov, V., Kasneci, G., Kasneci, E.: A consistent and efficient evaluation strategy for attribution methods. In: International Conference on Machine Learning (2023) [21] Schulz, K., Sixt, L., Tombari, F., Landgraf, T.: Restricting the flow: Information bottlenecks for attribution. arXiv preprint arXiv:2001.00396 (2020) [22] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. IEEE international conference on computer vision pp. 618–626 (2017) [23] Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: International conference on machine learning. pp. 3145–3153. PMlR (2017) [24] Souibgui, M.A., Choi, C., Barsky, A., Jung, K., Valveny, E., Karatzas, D.: DocVXQA: Context-aware visual explanations for document question answering. In: Forty-second International Conference on Machine Learning (2025), https: //openreview.net/forum?id=wex0vL4c2Y [25] Tishby, N., Zaslavsky, N.: Deep learning and the information bottleneck principle. In: 2015 IEEE Information Theory Workshop (ITW). pp. 1–5. Ieee (2015) [26] Tomsett, R., Harborne, D., Chakraborty, S., Gurram, P., Preece, A.: Sanity checks for saliency metrics. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 6021–6029 (2020) [27] Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-ucsd birds 200 (2010) [28] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. pp. 818–833. Springer (2014) [29] Zhang, Z., Gu, J., Chowdhury, A., Mai, Z., Carlyn, D., Berger-Wolf, T., Su, Y., Chao, W.L.: Finer-CAM: Spotting the difference reveals finer details for visual explanation. arXiv preprint arXiv:2501.11309 (2025) [30] Zheng, X., Shirani, F., Chen, Z., Lin, C., Cheng, W., Guo, W., Luo, D.: F-fidelity: A robust framework for faithfulness evaluation of explainable ai. In: Proceedings of the International Conference on Learning Representations (ICLR) (2025) [31] Zheng, X., Shirani, F., Wang, T., Cheng, W., Chen, Z., Chen, H., Wei, H., Luo, D.: Towards robust fidelity for evaluating explainability of graph neural networks. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=up6hr4hIQH [32] Zhou, Y., Booth, S., Ribeiro, M.T., Shah, J.: Do feature attribution methods correctly attribute features? In: Proceedings of the AAAI conference on artificial intelligence. vol. 36, pp. 9623–9633 (2022)
Learning Quantifiable Visual Explanations Without Ground-Truth Amritpal Singh1,2[0009−0006−1960−2548] , Andrey Barsky1,2[0000−0002−6993−5969] , Mohamed Ali Souibgui1[0000−0003−0100−9392] , Ernest Valveny1,2[0000−0002−0368−9697] , and Dimosthenis Karatzas1,2[0000−0001−8762−4454] 1 2
Computer Vision Center, Barcelona, Spain Autonomous University of Barcelona, Spain
1
Supplementary Material
1.1
Metrics
MSI Metric Interpreting the MSI Score. To interpret the MSI score meaningfully, it is important to consider both its theoretical and practical range. The MSI score ranges theoretically from −2 to +1. A score of −2 reflects an unrealistic scenario in which the model achieves perfect prediction when the entire image is masked out (i.e., no visual information is shown), while achieving zero accuracy when the full original image is provided. As such, we focus on the practical range of MSI scores, which typically lies between −1 and +1. The score is computed with respect to a chosen threshold αmin , which determines the cutoff for important pixels. Interpretation of the score at a given αmin is as follows: – Score close to +1: Indicates a high-quality explanation. The mask captures minimal yet sufficient information to make correct predictions, and the region above αmin likely includes all relevant evidence (i.e., one or more complete solutions). – Score close to −1: Suggests a poor mask. The retained region fails to provide sufficient information for prediction, indicating that the explanation is not capturing meaningful or discriminative features. – Score near 0: Can indicate one of two cases: • The mask is too large, including many unnecessary pixels. In such cases, qualitative inspection may help determine whether a large mask is truly required (e.g., in tasks with large or diffuse objects), or whether the explanation lacks focus. • The mask is minimal and predictive, but other valid solutions exist in the input that are not captured above the threshold αmin . This situation arises in settings where multiple plausible explanations can lead to correct predictions.
2
A. Singh et al.
Overall, MSI provides a more nuanced evaluation than standard metrics by jointly accounting for sufficiency, minimality, and the possibility of multiple valid solutions. This makes it particularly valuable for tasks involving complex inputs, multi-modal reasoning, or interpretability under uncertainty. While the goal is to achieve an MSI score closer to +1, a score near 0 does not necessarily indicate a poor explanation. In some cases, it may reflect either the presence of alternative explanations or task-specific requirements for larger masks. In addition, a higher value of αmin indicates that the mask is highly effective at separating important from unimportant pixels. This means the model can rely on a smaller, more confidently identified region to make accurate predictions, reflecting a more precise and discriminative explanation. By default, MSI is computed using classification accuracy as the predictive score, but it is easily extendable to other measures such as model confidence scores, allowing flexible application across different interpretability contexts. Other Metrics To assess the quality of different explanation methods, we employ several widely used evaluation metrics in addition to our proposed MSI score. These include Insertion and Deletion [3], as well as Fid-Insertion and Fid-Deletion [5]. Each metric evaluates the model’s response as information is either progressively added to or removed from the input based on the explanation heatmap. We report two variants for each metric: Pixel-based and percentagebased. – (Pixel value-based): In this setting, pixels are inserted or deleted by thresholding the heatmap values directly. For example, we may iteratively remove or restore pixels above a threshold value—e.g., 0.98, 0.96, and so on—based on heatmap intensity. – (Percentage-based): Here, pixels are sorted by heatmap values, and the top k% are inserted or deleted in increasing steps (e.g., top 2%, 4%, etc.). The metrics are interpreted as follows: – Insertion: Measures how quickly the model’s accuracy increases as pixels are progressively inserted. Higher is better. – Deletion: Measures how quickly the model’s performance degrades as pixels are progressively removed. Lower is better. – Fid-Insertion (Fidelity-Insertion): Measures the change in the model’s output when comparing the full image to a progressively inserted version based on the explanation heatmap. Lower is better. – Fid-Deletion (Fidelity-Deletion): Measures the change in the model’s output when comparing the full image to a progressively deleted version based on the heatmap. Higher is better. This comprehensive set of metrics—covering prediction confidence, information contribution, and fidelity—enables us to thoroughly evaluate the effectiveness and faithfulness of different explanation methods.
Learning Quantifiable Visual Explanations Without Ground-Truth
1.2
3
Datasets
1. Synthetic MNIST. The first dataset is a synthetic extension of the MNIST digit dataset [2]. We designed this dataset to increase the visual complexity of MNIST and better test the explanatory power of heatmaps. Specifically, we generate background images using random pixel values, embed randomly placed geometric shapes (e.g., boxes), and insert randomly scaled and positioned MNIST digits. This setup introduces distractors while maintaining the digit as the key predictive region. A key advantage of this dataset is that, in most cases, only the digit itself is required for accurate prediction—making it ideal for validating the minimality objective in our MSI metric. In practice, we observe that MSI scores on this dataset tend to be close to +1, reflecting the presence of a single dominant solution. 2. Caltech-UCSD Birds-200-2011 (CUB-200). The second dataset is the Caltech-UCSD Birds-200-2011 dataset (CUB-200-2011) [4], which contains 200 fine-grained bird species. For our experiments, we randomly select 20 classes. This dataset presents a more challenging scenario: while all birds share certain visual features (e.g., wings, beaks), each class requires the model to attend to specific discriminative cues for accurate classification. Although most examples likely have a single dominant region of interest, a small number of instances may contain multiple disjoint yet sufficient regions. Consequently, on this dataset, we expect the MSI score to be significantly higher than 0. 3. CIFAR-10. The third dataset is CIFAR-10 [1], which contains natural images from 10 object categories. Based on manual inspection and qualitative examples (e.g., as shown in Figures 7, 8, 9, and 10), we observe that many images contain multiple disjoint regions that can independently support accurate predictions. This characteristic makes CIFAR-10 a representative case for evaluating explanation methods under multi-solution scenarios. Accordingly, we expect the MSI scores on this dataset to be lower and closer to zero—which, in this context, is not an indication of poor mask quality, but rather reflects the inherent ambiguity in what constitutes a sufficient explanation. 1.3
Implementation Details
Explanation Generation To generate explanations, we use a lightweight convolutional module that operates on the feature representations extracted from the base model. In our setup, we use a lightweight ResNet as the base model, and the output of its last convolutional layer is used as the input to the explanation generation block. Since the three datasets have different input image sizes, we modify certain internal parameters of the base model—such as stride and padding—to ensure effective training. As a result of these adjustments, the output feature map shapes vary across datasets. Specifically: – For Synthetic MNIST, the feature map shape is 8 × 8 × 512. – For CUB-200, the feature map shape is 7 × 7 × 512.
4
A. Singh et al.
– For CIFAR-10, the feature map shape is 4 × 4 × 512. In both the Synthetic MNIST and CUB-200 cases, we do not apply upsampling, and the explanation heatmap is generated at the native resolution of the feature map. However, in the case of CIFAR-10, we apply upsampling within the explanation module to obtain an 8 × 8 heatmap for better spatial coverage and interpretability. The explanation block progressively reduces the channel dimension from the initial 512 channels to a single-channel output representing the heatmap. This is achieved through a series of convolutional layers: 512 → 256 → 128 → 64 → 1 This final single-channel output corresponds to the explanation heatmap, with spatial resolution either 8 × 8 (for Synthetic MNIST and CIFAR-10) or 7 × 7 (for CUB-200). These resolutions are not fixed by design but were found empirically to perform well in practice. Lower spatial resolutions encourage the network to focus on salient, coarse-level regions and tend to yield more interpretable and stable explanations. After generating the low-resolution heatmap, we upsample it to the original input image size using bilinear interpolation to enable pixel-wise alignment and visualization. Hyperparameters for Explanation Generation The explanation generation module was trained using different hyperparameter settings tailored to each dataset to ensure optimal learning behavior. Table 1 summarizes the learning rate, optimizer, entropy regularization weight (λentropy ), and temperature parameter (T ) used during training. These values were selected based on empirical validation performance and qualitative stability of the generated explanations.
Table 1: Hyperparameters for Explanation Generation across datasets. Hyperparameter Learning Rate Optimizer λentropy Temperature (t)
1.4
Additional Results
1.5
Note on LLM Usage
Synthetic CUB CIFAR MNIST 200 10 0.001 0.0001 0.001 Adam Adam Adam 5 5 2 0.5 0.5 0.5
Large language models were used in a limited capacity during the drafting of this paper, to improve clarity and polish the use of language. All technical contributions, ideas and content are solely attributable to the human authors.
Learning Quantifiable Visual Explanations Without Ground-Truth
Fig. 1: Qualitative example on Synthetic MNIST.
Fig. 2: Qualitative example on Synthetic MNIST.
Fig. 3: Qualitative example on CUB-200.
5
6
A. Singh et al.
Fig. 4: Qualitative example on CUB-200.
Fig. 5: Qualitative example on CIFAR-10.
Fig. 6: Qualitative example on CIFAR-10.
Learning Quantifiable Visual Explanations Without Ground-Truth
7
Fig. 7: Example illustrating multiple valid solutions. From left to right: original image, full mask, thresholded mask region (values ≥ 0.5), and its complement (values < 0.5).
Fig. 8: Example illustrating multiple valid solutions. From left to right: original image, full mask, thresholded mask region (values ≥ 0.5), and its complement (values < 0.5).
Fig. 9: Example illustrating multiple valid solutions. From left to right: original image, full mask, thresholded mask region (values ≥ 0.5), and its complement (values < 0.5).
Fig. 10: Example illustrating multiple valid solutions. From left to right: original image, full mask, thresholded mask region (values ≥ 0.5), and its complement (values < 0.5).
Bibliography
[1] Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech. rep. (2009) [2] LeCun, Y., Cortes, C., Burges, C.: Mnist handwritten digit database. AT&T Labs [Online]. Available: http://yann.lecun.com/exdb/mnist 2 (2010) [3] Petsiuk, V.: Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421 (2018) [4] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds 200-2011 dataset. Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011) [5] Zheng, X., Shirani, F., Chen, Z., Lin, C., Cheng, W., Guo, W., Luo, D.: F-fidelity: A robust framework for faithfulness evaluation of explainable ai. In: Proceedings of the International Conference on Learning Representations (ICLR) (2025)