The Neglected Baseline in Model Interpretation
arXiv:2605.22417v1 [cs.CV] 21 May 2026
Yongjin Cui Zhejiang University [email protected]
Xiaohui Fan Zhejiang University [email protected]
Abstract We observe that existing model interpretation methods generally ignore the baseline, and such neglect often results in imprecise or even incorrect interpretation. In this paper, we reformulate the task of model interpretation and the interpretation principles for model interpretation results to demonstrate the importance of the baseline. We further unify gradient-based methods, Integrated Gradients (IG) methods, and Taylor expansion, clarifying the connections among them and explicitly identifying the baseline for each method. On this basis, we analyze the flaws and errors in related model interpretation methods (IG, LayerCAM, ODAM, Difference Map). We advocate evaluating the quality of model interpretation results precisely through the attribution error between the attribution result and the attribution target, rather than adopting flawed evaluation methods, such as those based on marginal-effect or the assumption of perfect model performance. We revise IG and develope a model interpretation method with a clear and reasonable baseline, achieving better results. Our method supports model interpretation based on features from any layer. Interpretation based on features from different layers are all reasonable, and the differences among these results reflect varying degrees of feature extraction at different feature extraction stages.
1
Introduction
With the widespread application of deep learning models in fields such as computer vision and natural language processing, the "black-box" nature of models has emerged as a significant obstacle to their further deployment in critical domains, including medical diagnostics and financial risk control. Model interpretation refers to interpret or present the rationale behind a model’s decisions. It not only relates to the transparency of the model but also directly influences people’s trust in and acceptance of the model. The interpretation results of models also facilitate accurate evaluation of model capabilities and guide model optimization. In recent years, academic and industrial research on model interpretation has deepened, leading to the development of various interpretation methods and technical approaches, which has been used to identify failure modes Agrawal et al. [2016], Hoiem et al. [2012], debug modelsKoh and Liang [2017] and establish appropriate users’ confidence about models Selvaraju et al. [2017], instruct the knowledge distillation of object detectionZhao et al. [2024a], and distinguish the duplicate detected objectsZhao et al. [2024a]. Current model interpretation methods can be broadly categorized into two types and others as discussed in Section 2. Whether in terms of the computational process or the interpretation of model interpretation results, existing methods generally neglect the baseline. Among these methods, Integrated Gradients (IG) Sundararajan et al. [2017] involves a baseline, but it still has certain drawbacks. Moreover, current model interpretation methods based on IG also also contain serious errors [Another work of ours titled "Integrated Gradients in Transformer Interpretation"]. Neglecting the baseline can render an method theoretically imprecise or even erroneous, while also undermining the rigor of result interpretation. The baseline serves as the foundation for our evaluation of things. For instance, when assessing the speed of a car, the default baseline is a stationary object. Preprint.
Without this baseline, any evaluation of speed becomes meaningless. Similarly, if the baseline setting of an method is inherently vague or uncertain, the interpretation method itself becomes uninterpretable. In interpreting model interpretation results, the absence of a baseline also renders the interpretation meaningless. Gradient is widely applied in model interpretation. However, methods that directly utilize gradient suffer from two significant drawbacks: first, they are prone to large errors, as these approaches essentially represent a linear approximation of complex nonlinear processes; second, they lack a clearly defined baseline. Methods of this type are exemplified by LayerCAM Jiang et al. [2021], ODAM Zhao and Chan [2023], Zhao et al. [2024a] and Difference Map (DM)Xia et al. [2025], with ODAM essentially representing the application of LayerCAM in object detection models and DM being an extension of ODAM. ODAM successively published at ICLR 2023 and TPAMI 2024. DM successively published at ICCVW2025. To a certain extent, this indicates that mainstream authorities in the field have not yet recognized the severe consequences of neglecting the baseline. The introduction of Integrated Gradients (IG) Sundararajan et al. [2017] offers the potential to address the drawbacks of gradient-based methods. However, at present, the IG method itself remains imperfect and even has certain defects. In practical applications, it still encounters numerous issues; for example, process-based methods such as TAM, BT, and DIX, which all utilize IG, each contain errors [Another work of ours titled "Integrated Gradients in Transformer Interpretation"]. Taylor expansion is an extremely important mathematical tool that represents a function, which is differentiable at a certain point (the baseline), as an infinite series (polynomial), thereby approximating the function with a polynomial. Taylor expansion inherently possesses a clear baseline, which aligns perfectly with the theme of our study. In the main text, we will unify gradient-based method, Integrated Gradients (IG) based method, and Taylor expansion, clarifying the connections among them. The evaluation methods currently widely adopted for model interpretation methods primarily rely on marginal-effect or the assumption of perfect model performance. These evaluation methods are very crude, and are even incorrect in some cases. Developing a better evaluation method for model interpretation results is an urgent problem that needs to be addressed. In response to the aforementioned issues, this paper reformulates the task of model interpretation and the interpretation principles for model interpretation results to demonstrate the importance of the baseline. We further unify gradient-based methods, Integrated Gradients (IG) methods, and Taylor expansion, clarifying the connections among them and explicitly identifying the baseline for each method. On this basis, we analyze the flaws and errors in related model interpretation methods (IG, LayerCAM, ODAM, Difference Map). We advocate evaluating the quality of model interpretation results precisely through the attribution error between the attribution result and the attribution target, rather than adopting flawed evaluation methods, such as those based on marginal-effect or the assumption of perfect model performance. We revise IG and develope a model interpretation method with a clear and reasonable baseline, achieving better results. Our method supports model interpretation based on features from any layer. Interpretation based on features from different layers are all reasonable, and the differences among these results reflect varying degrees of feature extraction at different feature extraction stages. Our contributions can be summarized as follows: • We redefine the task of model interpretation and the principles for interpreting model interpretation results, demonstrating the importance of baselines. • We unify gradient-based method, IG based method, and Taylor expansion, clarifying the connections among them, and analyze the flaws and errors in relevant methods. • We elaborate on the drawbacks of existing evaluation methods for model interpretation resluts and proposed using attribution errors between to precisely evaluate the quality of model interpretation results. • We revise IG and develope a model interpretation method with a clear and reasonable baseline, achieving better results. Our method supports model interpretation based on features from any layer. 2
2
Related Works
This paper categorizes model interpretation methods into two main types: process-based and featurebased interpretation methods. This classification is primarily because feature-based methods are more suitable for performance evaluation using attribution errors. 2.1
Process-Based Interpretation Methods
These methods focus on the model’s decision-making process. The core idea of such methods is to "reconstruct" the model’s decision-making process, enabling humans to understand how the model reasons from input to output. A typical representative of this type of model interpretation method is those related to Transformers. Chefer et al. Chefer et al. [2021] introduced Generic Attention-model interpretability (GAE), which combines gradients with multi-head attention maps and then performs attention rollout. Yuan et al. Yuan et al. [2021] interpret the information flow inside Vision Transformers using Markov Chain (TAM). Barkan et al. Barkan et al. [2023] propose Deep Integrated interpretation (DIX), which generates interpretation maps using GAE and Integrated Gradient. Chen et al. Chen et al. [2023] propose the Beyond Intuition Method (BT) based on IG. These methods place greater emphasis on leveraging the model’s attention. The overall idea is to simulate the distribution change process of model attention as it passes through each layer of the model, using the model’s overall attention to different input parts as the interpretation result. 2.2
Feature-Based Interpretation Methods
These methods focus on quantifying the direct contribution of features to the model’s output. The core idea of feature-based interpretation methods is "feature importance," which measures the contribution of each feature to the model’s prediction. Most of these methods are gradient-based methods, such as CAM Zhou et al. [2016], GradCAM Selvaraju et al. [2017], LayerCAM Jiang et al. [2021],GradECLIP Zhao et al. [2024b] and ODAM Zhao and Chan [2023], Zhao et al. [2024a]. CAM utilizes the activations from the convolutional layers of a CNN to obtain saliency maps. The GradCAM method generalizes the CAM method for a broader spectrum of CNN architectures by introducing gradient information to avoid the need for a Global Average Pooling (GAP) layer. LayerCAM refines the weights of feature maps using gradients and integrates multiple layers of saliency maps to achieve more detailed results, as well as the primary baseline method ODAM in our paper. GradECLIP is proposed to interpret Contrastive Language-Image Pre-training (CLIP). ODAM Zhao and Chan [2023], Zhao et al. [2024a] is gradient-weighted Object Detector Activation Maps (ODAM) to interpret the predictions of object detectors. ODAM is the application of LayerCAM in object detection models. In particular, it is worth noting the Integrated Gradient method (IG) Sundararajan et al. [2017]. IG was proposed to address the issue of large errors in gradients within saturation regions. However, in our view, another noteworthy contribution is that IG can explicitly define a baseline. 2.3
Others
There are some relevance propagation methods and perturbation methods. Relevance propagation methods include Layer-Wise Relevance Propagation (LRP) Binder et al. [2016], Contrastive LRP (CLRP) Gu et al. [2018], and Softmax-Gradient-LRP (SGLRP) Iwana et al. [2019], among others. LRP propagates relevance from the predicted class backward to the input image based on Deep Taylor Decomposition (DTD) Montavon et al. [2017]. CLRP improves upon LRP by comparing the predicted category signal with other category signals to interpret the model. SGLRP is a classdiscriminative extension to DTD that uses the gradient of the softmax to backpropagate the relevance of the output probability to the input image. Perturbation methods include ViT-CX Xie et al. [2023] and TIS Englebert et al. [2023]. ViT-CX is based on token embeddings rather than the attentions paid to them and their causal impacts on the model output. TIS is a perturbation-based interpretability method for Vision Transformers that computes a saliency map based on perturbations induced by sampling the input tokens. It is particularly noteworthy here that the three process-based methods—TAM, BT, and DIX—have applied IG, yet all have been proven to contain significant errors [Another work of ours titled "Inte3
grated Gradients in Transformer Interpretation"]. Additionally, Grad-ECLIP, a feature-based method, has also been proven to have significant errors [cui2026debunkinggradeclipcomprehensivestudy].
3
Model Interpretation and Baseline
3.1
Re-clarification of Model Interpretation
Here, we are merely re-clarifying the definition of model interpretation based solely on the model interpretation tasks involved in this study, while an authoritative, rigorous, and comprehensive definition may encompass more aspects. We emphasize here the definition of model interpretability—that is, understanding and explaining the basis on which a model makes specific decisions or predictions. Decisions or predictions represent the output of the model, and the basis for these decisions comes from the input. To conduct model interpretation, one must first measure the model’s input and output. When measuring or evaluating something, there must always be a baseline. For instance, when describing vehicle speed, the default baseline is a stationary object, such as a stationary road surface or building. So, what constitutes the baseline for a model’s input and output? Previous studies have largely overlooked this issue. 3.2
Baseline of Model Interpretation
Here, we take the object detection model DETR Carion et al. [2020] as an example for illustration. The model takes an image as input and produces two outputs: a bounding box and a class score. Without a baseline, it is impossible to accurately describe or evaluate a thing. In many cases where a baseline is not explicitly stated, there is actually an implicit default baseline. For instance, when we describe a vehicle’s speed without specifying a reference object, the default baseline is assumed to be stationary objects. Similarly, when describing the content of an image, the default reference is a completely black image with all pixel values set to zero, rather than another non-zero baseline image. Otherwise, we would first need to subtract the baseline to clearly describe the image content. When describing the coordinates of a detection bounding box, the origin of the coordinate system serves as the baseline, and when describing the class probabilities output by the model, a probability of zero is used as the baseline. It is important to note that, in the vast majority of cases, the baselines for these three elements (input, bounding box, and class score) cannot be aligned. When a completely black image is fed into the model, it is extremely challenging to ensure that the coordinates of the target bounding boxes and the class score output by the model correspond to their respective baselines. To put it plainly, even if we input a completely black image, the model will still generate corresponding detection bounding boxes and class scroes, as shown in Figures 5 to 7. When the baseline image (a completely black image) is input, the class probabilities and coordinates output by the model do not correspond to the expected baselines. We have noticed that object detection models (such as DETR) have one input and two outputs. When interpreting the basis for the model’s inferences, we first need to select an object from the input image and the two outputs to establish a baseline. When determining the baseline based on a certain output, for instance, if we choose the coordinate origin as the baseline, we now need to establish baselines for the input and class scroe. In this case, our approach can be summarized as first determining a baseline input through the output of the origin coordinates, and then feeding this baseline image input into the model to establish the baseline for class score. However, we find this extremely challenging because it is almost impossible to determine an input based on the output of a deep learning network. An alternative approach is to use the input to establish the baseline for the entire interpretation system. When we input an image baseline (a completely black image), the obtained object bounding box coordinates and class probabilities serve as the corresponding baselines for the outputs. More importantly, both humans and models base their understanding of image content on the numerical values of image pixels. The implicit pixel baseline is 0. Therefore, it is reasonable to choose a full-zero input as the baseline for the input when interpreting the basis for the model’s output. 4
Therefore, to provide a more accurate and reasonable interpretation of the model, it is essential to establish a baseline during the model interpretation process. Given both the difficulty and the rationality of baseline determination, the baseline should be established through the input. 3.3
Correct Interpretation of Model Interpretation Results based on the Baseline
Please refer to Section 5.2.
4
Gradients and Integrated Gradients
4.1
Gradients in Model Interpretation
Gradients are often used as an indicator to measure the importance of a variable. When evaluating the contribution of a certain factor to the output, a common approach is to take the gradient as the weight of that factor and compute their product as the contribution value. This approach can be traced back to attribution methodsBaehrens et al. [2010], Simonyan et al. [2014]. Typical methods employing this method in other models include GradCAMSelvaraju et al. [2017], LayerCAMJiang et al. [2021], while in the context of Transformer interpretation, representative methods are GAEChefer et al. [2021], ODAMZhao and Chan [2023], Zhao et al. [2024a] and DMXia et al. [2025]. Although this kind of gradient application is quite widespread, we haven’t found the theoretical basis for this application approach in the aforementioned papers. It is absurd that the principle of a model-interpretation scheme is itself difficult to explain. Researchers have been using this method all along, yet no one has explored its theoretical underpinnings. We analyze this in Section 4.3. This is one of the contributions of this paper. 4.1.1
The Limitation of Gradients in Model Interpretation
Gradient-based methods are limited because they are essentially linear approximations of complex nonlinear processes(Section 4.3), resulting in large errors.There are even cases where it completely fails to correctly reflect the magnitude of the input’s contribution. Figure 1 shows the case of F (x) = 1 − ReLU(1 − x) presented in the IG paper Sundararajan et al. [2017]. Taking x = 0 as the baseline (with a baseline contribution of 0), when x > 1, the gradient is 0. If the contribution is defined as the product of the gradient and the input, the contribution of x in this case is also 0, which clearly violates SensitivitySundararajan et al. [2017]. 4.1.2
LayerCAM
Here, we take an image classification model as an example to introduce LayerCAM. Formally, let f denote the image classifier, θ represents its parameters, and y represents the predicted score. For a given image I and target class c, y c = f c (I, θ)
Figure 1: F (x) = 1 − ReLU (1 − x) 5
(1)
Let A denote intermediate feature maps of a certain layer, Ak denote the k-th feature map (the k-th ∂y c kc c feature dimension), gij = ∂A k represent the gradient of y with respect to the spatial location (i, j) ij
in Ak , w represent the weight of the feature map. kc kc wij = relu(gij )
(2)
kc k Âkc ij = wij · Aij
(3)
 represents the class activation map.
M represents the final results, which combinning  along the feature dimension. ! X c kc M = ReLU Â
(4)
k
4.1.3
ODAM
ODAM is specifically proposed for object detection models and is essentially an application form of LayerCAM in object detection models. Unlike image recognition models, object detection models can distinguish between different instances. For an instance p, the model outputs category information as well as spatial coordinate information of the instance. During the calculation process of ODAM, unless otherwise specified, the representation of variables remains the same as that in LayerCAM. For an instance p, a feature layer is selected, and its calculation process is identical to that in LayerCAM. In object detection models, such as DETR, each instance has five outputs: category information y, and bounding box coordinates x1 , y1 , x2 , y2 . Driven by a misguided intuition, ODAM interprets the box coordinates as taking the values of x1 , y1 , −x2 , −y2 . This might be because, when the feature region is centered, the expansion or contraction of the feature region has opposite effects on x1 , y1 and x2 , y2 . However, this line of thinking is incorrect, ∂Y k please refer to Section 6.1.1. These outputs are uniformly represented as Y . Here, gij = ∂A k , where ij
Y can be, depending on the context of interpretation, the score y c for a certain category c, or the coordinates of the bounding box x1 , y1 , −x2 , −y2 . ODAM’s computation process for interpreting a single output is almost identical to that of LayerCAM, with the only difference being the removal of the relu in Equation (2). ODAM provides a scheme that integrates the interpretation results from all outputs to produce the final interpretation (we do not endorse this scheme, refer to Section 6.1). In this context, Y represents all outputs, and Φ denotes the operation of taking the maximum value of the interpretation results of all model outputs. (For specific details, please refer to the ODAM code: https://github.com/CyangZhao/ODAM).
Y Mij = Φ Mij
(5)
However, we believe that merging the interpretation results of multiple different outputs can only bring about visual changes. The true interpretation still corresponds to a specific output. 4.1.4
DM
DM being an extension of Equation (5) of ODAM. DM aims to integrate interpretation results from different outputs to obtain the final interpretation that better align with traditional evaluation metrics. We do not present the detailed computation process, as we oppose integrating different results as the final result and do not endorse traditional evaluation methods for model interpretation. We present DM primarily to illustrate that flawed evaluation metrics have misguided researchers’ directions. LayerCAM was published in TIP 2021, ODAM was published in ICLR 2023 and TPAMI 2024, and DM was published in ICCVW 2025. These publications sufficiently demonstrate that gradient-based methods that disregard baselines have gained recognition from mainstream authoritative researchers 6
in the field, while the flawed evaluation methods have not attracted any scrutiny. When something wrong is used long and widely enough, it becomes the standard. During some submission processes, we were required to adopt traditional evaluation methods for experimental comparisons. These issues constitute the significance of undertaking this work. 4.2
Integrated Gradients
The emergence of IG addresses the issue that gradients fail to satisfy SensitivitySundararajan et al. [2017], and IG satisfy Implementation InvarianceSundararajan et al. [2017] at the same time. The naive principle of IG is to focus on every step of the input contribution’s change from the input baseline to the current input, rather than merely concentrating on the final state. This approach helps avoid situations where gradients become ineffective in the case of F (x) in Figure 1. IG is defined as the path intergral of the gradients along the straightline path from the baseline x′ ∈ Rn to the input x ∈ Rn . The integrated gradient along the ith dimension is defined as follows. IGi (x) ::= F (x) − F (x′ ) Z 1 ∂F (x′ + α(x − x′ )) ∂(x′i + α(xi − x′i )) dα ::= ′ ′ ∂α α=0 ∂(xi + α(xi − xi )) Z 1 ∂F (x′ + α(x − x′ )) = (xi − x′i ) dα ′ ′ α=0 ∂(xi + α(xi − xi ))
(6)
The calculation process adopts a straightline path from the baseline input to the current input; for the advantages of this approach, please refer to the original paper. The integral of IG can be efficiently approximated via a summation (Riemann Sum). We simply sum the gradients at points occurring at sufficiently small intervals along the straightline path from the baseline x′ to the input x. IGapprox (x) i ::=
k (x − x′ )) ∂F (x′ + m (xi − x′i ) × Σm k=1 k m (xi − x′i )) ∂(x′i + m
(7)
Here m is the number of steps in the Riemman approximation of the integral. 4.2.1
IG in Model Interpretation
TAM, BT, and DIX are model interpretation method based on IG, but all have been proven to contain significant errors. Please refer to our another research Integrated Gradients in Transformer Interpretation. 4.3
Unification of Gradient Methods and Integrated Gradient Methods
In Equation (7), we can find that the gradient method is actually a special case of IG when m = 1, and the baseline x′ is all-zero. Formally, it simply performs a simple linear approximation of a complex nonlinear operation, as show in Equation (8). Gi (x) ::= xi × 7
∂F (x) ∂(xi )
(8)
4.4
Unification of Integrated Gradient Methods and Taylor Expansion
When approximating Integrated Gradients, if we start from the gradient at the baseline, that is, we also set the number of steps as m, but begin with k = 0, IGapprox (x) i ::=
k (x − x′ )) ∂F (x′ + m (xi − x′i ) × Σm−1 k=0 k m ∂(x′i + m (xi − x′i ))
(9)
When m = 1, and the baseline x′ is all-zero, IGapprox (x) i ::= F (x) − F (x′ ) ∂F (x′ ) ::= xi × ∂(x′i )
(10)
This is precisely the first-order term in the Taylor expansion. The Taylor expansion itself inherently features a well-defined baseline, representing a manifestation of baseline-oriented thinking. Taylor expansion also offers an approach for model interpretation. The issue, however, lies in the fact that for multivariate functions, Taylor expansions beyond the first order require the computation of a large number of mixed gradients, imposing excessively high computational demands and making implementation difficult. Therefore, we have abandoned this approach in our study. Our introduction of the relationship between Taylor expansion and Integrated Gradients here is primarily aimed at emphasizing that a baseline is essential in rigorous model interpretation work.
5
Our Method
Our method rigorously performs model interpretation with clear baseline based on IG. We correct the principles for baseline selection, and employ attribution error to rigorously evaluate the interpretation results. 5.1
Baseline Selection Principles
There are errors in the baseline selection of the original IG. The principles for selecting baselines in the original IG method are as follows: 1. IG recommend that developers check that the baseline has a near-zero score, this allows developers to interpret the output as a function of the input. 2. IG would additionally like the baseline to convey a complete absence of signal, so that the features that are apparent from the attributions are properties only of the input, and not of the baseline. Regarding the first principle, it essentially involves selecting an input baseline based on the output to a certain extent. The second principle determines the baseline through the input. We have already elaborated on these in Section 3.2. The baselines for input and output are almost impossible to align. The reason why the IG paper proposed these two principles and managed to justify them is primarily because IG is mainly applied to object recognition models. In such models, the final output is the softmax output, which amplifies the relative differences among input values, allowing high-probability categories to dominate the probability output while compressing low-probability categories, making it relatively easy to achieve near-zero scores. However, when we apply this method to object detection models, where one of the outputs is the coordinates of the detection bounding box and thus lacks the softmax transformation, we find it challenging to implement the two baseline selection principles proposed in the IG paper. This can be seen in Figures 5 to 7. When the input is a completely black image, the model’s output bounding boxes for objects fail to meet the first principle of IG’s baseline selection.The first principle for baseline selection in IG is actually incorrect, but this flaw may not be exposed in specific scenarios. Here, we correct the principles for baseline selection as follows: 8
1. The baseline for the entire model interpretation is solely determined by the input. This differs from one of IG’s baseline selection principles, as we completely disregard the output. For details, please refer to Section 3.2. 2. The baseline can be chosen arbitrarily, but the final attribution is based on the difference between the input and the baseline. For instance, for an image input, the baseline can be any image, depending on the user’s needs. However, selecting an all-zero image aligns most closely with how humans and models perceive images. Although this may seem similar to IG’s second baseline selection principle, it actually offers more possibilities for baseline selection. Developers can compare the impact of differences between any two images on the output results according to their needs. Moreover, we do not approve of the situation where the IG paper appears to satisfy its first principle by choosing random noise images as baseline inputs. Such a scenario requires a rigorous interpretation approach and should not be directly treated as an interpretation for the outputs, refer to Section 5.2. 5.2
Correct Interpretation of Model Interpretation Results
We have introduced baselines into model interpretation through IG, and accordingly, the interpretation of model interpretation results requires adjustments. The correct approach to interpreting model interpretation results is as follows: our objective to interpret is the difference between the model’s current output and the baseline output, the attribution results rely on the difference between the current input and the baseline input. As shown in Figures 5 to 7, the object to interpret is the difference between the current output and the baseline output. The baseline output is determined by the baseline input. In our experiments, we selected a completely black image as the baseline input. Therefore, the attribution results rely on the difference between the current input and the completely black image, which essentially corresponds to the current input itself. 5.3 5.3.1
Evaluation of Model Interpretation Results Traditional Evaluation Methods
There are currently two main approaches to evaluate the performance of model interpretation methods: qualitative and quantitative. The qualitative approach involves comparing model interpretation results with human understanding of target features for consistency. There are two types of quantitative approaches. One is similar to qualitative evaluation, where the consistency between model interpretation results and feature regions is compared in a batch manner. The other involves perturbing the input according to model interpretation results—for example, removing corresponding elements in order of importance and observing the magnitude of change in the model output. The essence is to evaluate based on marginal effects. Consistency-based evaluation relies on a default assumption: that the way humans perceive things is the optimal solution, and that the model under test is already good enough to obtain this optimal solution. Under this assumption, comparing model interpretation results with human understanding of features—where higher consistency indicates a better interpretation method. First, we cannot be entirely certain that human cognition truly represents the optimal solution; the model may also learn other patterns on its own. Another issue is that we cannot guarantee the test model is sufficiently good. In that case, deviations in model interpretation results may stem either from the interpretation method itself or from the model’s insufficient performance. The marginal-effect-based evaluation approach ignores data patterns. The information conveyed by data depends on the relationships among elements. The marginal-effect-based evaluation approach disrupts these data interdependencies and introduces uncertainty, making it difficult to ensure the accuracy and reliability of evaluation results. For instance, the artifact phenomenon Xie et al. [2023] confirms this problem. Readers are referred to the cited article for details on the artifact case. Regarding the limitations of marginal effects, we can illustrate this with a very simple everyday example. A man and a woman get married and have a child. When evaluated using marginal effects, if we assume there is no man in this relationship, then the child could not exist. Thus, the marginaleffect-based evaluation would attribute the entire contribution of having the child to the man. This is clearly incorrect. The deficiency of marginal effects was already identified as early as 1953Shapley 9
[1953], yet more than seventy years later, using marginal effects for result evaluation has become virtually an established gospel in the field of explainable AI. 5.3.2
Attribution Error
The introduction of IG has enabled more reasonable and accurate model interpretation. However, it is regrettable that the evaluation of model interpretation results in the original IG paper still relies on traditional methods. As shown in Figures 5 to 7, we can precisely assess the quality of model interpretation results by the attribution error between the attribution results and the attribution targets, thereby eliminating the drawbacks of traditional evaluation methods. Particularly, as shown in Figure 7, when evaluated by traditional methods, the interpretability performance of our method at low stages appears to be very poor. However, from the perspective of attribution error, these interpretation results precisely reflect the contributions of features from that specific layer. This discrepancy arises from the varying degrees of feature extraction at different stages—the closer a layer is to the output, the more its features resemble semantic representations. This demonstrates that our approach of assessing interpretability through attribution error can circumvent the drawbacks inherent in traditional evaluation methods. 5.4
Model Interpretation through Features from different Layers
Feature-based model interpretation methods typically first select a feature layer for interpretation. For example, CAM and GradCAM rely on deep features—more precisely, the features from the last layer—while LayerCAM considers fusing interpretation results from different layers. The reason for this practice is that model interpretation results differ across feature layers. According to traditional evaluation methods, interpretation results derived from different layers vary in quality, so researchers usually select a specific feature layer for model interpretation. Regarding the attribution error, we hold a completely opposite view. In the experiments shown in Figure x, we demonstrate that the attribution errors of interpretation results obtained from different feature layers are all very small. Under traditional evaluation methods, using different layers for interpretation yields results of varying quality, but under attribution error, there is no difference in using features from different layers for model interpretation. The hierarchical processing of data by the model is essentially a continuous process of feature extraction. Shallow feature integration is incomplete, so the resulting interpretation results intuitively appear poor. As the model goes deeper, feature extraction becomes increasingly refined and closer to semantics, which better aligns with human intuition—and according to traditional evaluation methods, this means the interpretation results are better. However, from a numerical perspective, we can treat the intermediate feature A as the input and regard the subsequent network as a function f ′ , so the output is y = f’(A). When using IG for model interpretation, we can always accurately attribute the output to the input. IG brings an objective evaluation standard to model interpretation. LayerCAM attributes part of the reason for the differing interpretation results from features of different layers to the different spatial resolutions across layers. In Figure 5, we present experimental results of our method based on ResNet features and Encoder features. Even when the two features have the same spatial resolution, the results still show that interpretation results based on deeper features better align with semantics. The essence of this phenomenon lies in the differing degrees of feature extraction across layers, rather than differences in spatial resolution. In short, features from any layer can sufficiently explain the model. The differences arise from the varying degrees of feature extraction at different depths, with no distinction in quality. In Section 5.3, we pointed out the limitations of traditional evaluation methods. The attribution errorbased evaluation method we advocate is more objective. When the attribution error is sufficiently small, we have sufficient reason to believe that the model interpretation method is effective. If the interpretation results do not align with the feature regions recognized by humans, the reason may lie in the model’s own performance being insufficient, or the model having learned a cognitive pattern different from that of humans, or the feature layer relied upon representing a feature extraction stage that still has limitations. However, there is a hidden assumption here: we assume that model interpretation should reflect the model’s complete extraction of data features, such as the understanding of image semantic 10
content. We can attribute the model output to intermediate features at any layer, which is reasonable, but not every layer’s features represent the model’s complete extraction of semantic features. The widely adopted qualitative and quantitative experiments are actually built on the basis of human understanding of image semantics. Our human semantic features are the final features we have fully extracted. However, some methods perform model interpretation based on shallow features of the model. For instance, ODAM’s interpretation of the DETR model is based on features extracted by ResNet rather than the features extracted by the final Transformer, which is unreasonable. The model’s complete extraction of image semantic features lies in the features fed into the discriminator, not the intermediate features. Therefore, to fully reflect the features upon which the entire model makes inferences, selecting deep features is necessary. Consequently, ODAM’s choice of using ResNet-output features when explaining DETR is inappropriate. Additionally, the experimental results verify another phenomenon: the closer the features are to the output, the lower the degree of nonlinearity between the features and the output, and thus the attribution error is more likely to be smaller under the same conditions. We will provide a detailed analysis in the subsequent experiments. 5.5
Method
We follow the parameter definitions used in LayerCAM and ODAM, and the computational process of our final method is as follows: Mkij (A, A′ ) ′
::=
′
′
k (Akij − Akij ) ∂F (Ak + m (Ak − Ak )) × Σm k=1 ′ ′ m ∂(Akij + k (Akij − Akij ))
(11)
X
(12)
m
Mij =
Mkij
k
Here m is the number of steps in the Riemman approximation of the integral. To emphasize the flexibility in baseline selection, we express our process as Mkij (A, A′ ). Here, A represents an intermediate feature from a certain layer, and A′ denotes the baseline for that intermediate feature. Using the intermediate feature A as a demarcation, we divide the entire network into two parts: f1 and f2 , where A = f1 (x) and y = f2 (A). The baseline is determined as A′ = f1 (x′ ), and the output baseline is y ′ = f2 (A′ ). The input baseline x′ can be arbitrarily defined within a reasonable range of the input. The final result is an attribution interpretation for y − y ′ with respect to x − x′ . We allow x′ to be arbitrarily defined within the reasonable input range, but when we only explain x, we still define x′ as an all-zero vector. The all-zero baseline is also adopted in subsequent experiments. It is particularly worth emphasizing here that, although ODAM can be regarded as a single-step IG, ODAM does not have an explicit input baseline. For example, when interpreting DETR, ODAM is implemented based on features output by ResNet, meaning that ODAM’s baseline is an all-zero feature map with exactly the same shape as ResNet’s output features, i.e., the baseline is A = 0. However, in this case, we cannot confirm a clear and unique input baseline x′ via A′ = f1 (x′ ). This is a hidden drawback of ODAM: ODAM can never perform precise attribution explanation with baseline interpretation method (Section 5.2).
6
Experiments
After unifying the gradient method and the IG method within the same framework, we can find that the gradient method is actually a single-step IG. In fact, there is no longer a need to compare these two methods in terms of interpretation accuracy, as IG method is bound to be more precise than the gradient method. Consequently, there is little need to conduct extensive experiments to validate the performance of these methods, as this would only waste computing power and energy. 11
The main purpose of our subsequent experiments is not to compare the performance of ODAM and our method, but to demonstrate how to implement our method and how to correctly interpret model interpretation results using a baseline-oriented mindset. We select the DETR model and the VGG model from the ODAM and LayerCAM papers for experimental demonstration. The environmental configurations and code for our experiments will be made publicly available. To replicate these experiments, it is preferable to have a GPU, as our experiments were conducted on an NVIDIA A100 GPU. The version of ODAM published in TPAMI has added some application scenarios for the model’s interpretability results compared to its ICLR version, but this is not our focus. Here, we only discuss and compare the methods themselves, while the applications of the model’s interpretability results are beyond the scope of our discussion. Additionally, it should be noted that ODAM has certain defects and even errors (Section 6.1), and the application of ODAM’s results in the original paper also needs to be re-examined. To better present the results of model interpretation, we choose a more reasonable approach when creating visualizations. We denote the interpretation reslut as S, and then normalize S using S max(abs(S)) , which can maintain a linear relationship between scores and preserve their positive and negative attributes. As shown in Figure 5, red represents positive contribution, blue represents negative contribution, and white represents no contribution. A color bar is used for matching color and value. Colors of an image represent relative value within a single image, and cannot be used for comparing numerical values between images. Other images in this article follow these rules. In some figures, like Figure 3, in order to better demonstrate the allocation of overall contribution, we use the heatmap as the bottom layer, and then overlay the original image with a certain transparency on the heatmap layer for indication. In all experiments based on the baseline analysis method advocated in this paper, we annotate the baseline detection box (black) and the model’s output detection box under the current input (green), along with the corresponding baseline analysis data beneath each case image. When interpreting a target box, we use cyan to highlight the specific edge being explained, as shown in Figure 4. 6.1
Interpretation of DETR
6.1.1
Debunking ODAM
Among the methods capable of providing model interpretation for DETR, the process-based method is GAE, while in feature-based methods, the ODAM paper demonstrates that ODAM outperforms other intermediate feature-based methods (such as GradCAM, GradCAM++Chattopadhyay et al. [2018], and D-RISEPetsiuk et al. [2021]). GAE is also the primary method that ODAM paper compares with in the context of DETR interpretation, abbreviated as GAME in the ODAM paper. The ODAM authors evaluate the two as follows: "The ODAM combo maps highlight information consistent with the Generic Attention-model Explainability (GAME)," and "in contrast to GAME, ODAM is also able to highlight the important regions for predicting each coordinate of the bounding box." The following are our arguments refuting ODAM. • GAE can similarly switch the target interpretation object from category output to coordinate output and is also capable of interpreting coordinate information, as shown in Figure 2. GAE is also a gradient method based on attention mapps from all layers, and the attention mapps is generated from features, meaning attention mapps and features are not independent , that is, the contribution of the input to the output is captured in both the attention maps and the intermediate features. All of these render baseline analysis methods inapplicable. Therefore, Figure fig. 2 is solely used to refute the viewpoint of ODAM. GAE can switch its interpretation object to explain any output, and from this perspective, ODAM does not hold an advantage. Given that GAE cannot be analyzed using baseline methods, we refrain from evaluating its interpretation results. Moreover, since ODAM’s interpretation of coordinates is incorrect (refer to next point), there is even no value in making a comparison. • ODAM’s approach to interpreting coordinates is incorrect. Based on a flawed intuition, ODAM assigns the interpretation values for box coordinates as x1 , y1 , −x2 , −y2 . The 12
Figure 2: GAE coordinate interpretation. reasoning behind this is that when considering the feature region as the center, the expansion or contraction of the feature region has opposite effects on x1 , y1 and x2 , y2 . However, this line of thinking is erroneous. When viewed through the lens of baseline analysis methods, what we are actually interpreting is the difference between the current output and the baseline output. This seemingly reasonable idea, coupled with some misleading results, can easily lead to misunderstandings. As shown in Figure 6, ODAM has a hidden baseline, and the interpretation results represent the difference between the current output and the baseline output. • It is clearly unreasonable to compare ODAM’s interpretation results, which integrate both category and coordinate information, with GAE’s pure category interpretation results. In Figure 3, we have re-presented the results focusing solely on category interpretation. Judging from the perspective of the qualitative evaluation criteria used in both the ODAM and GAE papers, ODAM’s interpretation results fall significantly short of those of GAE. Please note that this evaluation is conducted using the qualitative evaluation criteria from the ODAM and GAE papers. Since baseline analysis methods cannot be applied to evaluate GAE’s results, we refrain from making any assessments regarding the effectiveness of GAE’s model interpretation. • The experiment in ODAMZhao et al. [2024a] employs DETR_demo instead of the standard DETR model. The DETR_demo was originally intended by the author’s team as a simple demonstration of the model’s underlying principles, and demo models are not used in formal research. For instance, the research in the original GAE paper utilizes the standard DETR model. Moreover, the team that proposed DETR has long since removed DETR_demorelated content from their GitHub project. Judging from the evaluation criteria of ODAM and GAE, ODAM’s results on DETR_demo are inferior to those of GAE, and its results on the standard DETR model are even worse (Figure figs. 3 and 4).
13
Figure 3: Interpretation of category outputs (logits and probabilities) of the DETR_demo model by ODAM and GAE. (ODAM strictly adheres to the original settings and provides interpretation based on the output features from ResNet.)
Figure 4: Interpretation of category outputs (logits and probabilities) of the DETR model by ODAM and GAE. (ODAM strictly adheres to the original settings and provides interpretation based on the output features from ResNet.)
14
6.1.2
Comparative Experiments
Since GAE is not applicable to the baseline analysis, or more specifically, the attribution error analysis, we only compare with ODAM in subsequent experiments. Based on the above refutations, we commence a comparison between ODAM and our method using the baseline analysis approach. It should be noted that we conduct experiments solely on the standard DETR model, as the DETR_demo model has a simple structure and is merely for demonstration purposes; thus, we do not expend effort on the demo model. We revise ODAM’s method of interpreting coordinates, ensuring that the positive and negative attributes of coordinates are no longer altered. In addition, we implement our method based on the output features of both ResNet and the encoder separately. Furthermore, combined with the case illustration of the VGG model in Section 6.2, we demonstrate that the closer the features are to the output, the more semantically relevant the feature extraction results are, and the closer the qualitative analysis results are to human cognition. However, it is important to note that features at any stage of the information processing can serve as a complete attribution for interpreting the output. Meanwhile, the closer the features are to the output, the lower the degree of nonlinearity and the higher the degree of linearity between the features and the output, resulting in fewer steps required when applying Integrated Gradients (IG) for model interpretation. This point becomes more evident in the VGG experiments in Section 6.2. Our interpretation results for categories are shown in Figure 5, and those for bounding boxes are presented in Figure 6. In all our experiments, the baselines are determined by all-zero inputs. Below each case illustration, we have indicated the difference between the output to be interpreted and the baseline output, the attribution results, the error. When interpreting categories, we display two sets of results: one using logit as the output for interpretation and the other using probability. In the figures, the green detection boxes represent the model’s current output, while the black detection boxes denote the baseline output. We highlight the detection boxes being interpreted using cyan. In our experiments, we set the number of integration steps to 500. Additionally, we implemented our method based on features extracted from both the ResNet network output and the Encoder output. Upon a inspection of traditional evaluation methods, for category interpretation, the feature regions highlighted by our method align more closely with the correct feature regions. Upon a inspection of attribution error, our method is much better. Furthermore, it is noteworthy that, with the same number of integration steps, the interpretation accuracy based on Encoder features surpasses that based on ResNet features. This is because the closer the features are to the output being interpreted, the lower the degree of nonlinearity between the features and the output, and consequently, the fewer integration steps required. This point is more clearly demonstrated in Section 6.2. Additionally, it is particularly worth noting that, in the interpretation of detection boxes, our conclusion differs from that of ODAM. The highlighted conclusion in the original ODAM paper is that the model determines the corresponding coordinates of detection boxes based on partial regions or contours of the target’s feature regions. In reality, the model’s generation of detection box coordinates does not rely solely on the contours of feature regions, but rather on the combined effect of all elements. For example, in the interpretation results for the bottom border using our method implemented based on encoder features (as shown in the bottom-right case of Figure 6), the model does not output coordinates based merely on the tips of the cat’s paws; the upper parts of the cat’s feet still play a very significant role. More precisely, every image element contributes to the final result, and the final output is the combined effect of all elements. We emphasize once again that there can be countless case demonstrations. The primary purpose of the case studies presented here is to illustrate the analytical process of our baseline-based method. Because ODAM is a single-step IG without a clear baseline input(please refer to Section 5.5). Although a performance comparison is not deemed necessary, we have designed batch experiments for further testing. We uniformly sampled 2,000 samples from 5,000 samples in the COCO datasetLin et al. [2014]. We set the batch size to 6, and if a batch contained images requiring resizing (stretching to alter their shapes), we discarded that batch. Ultimately, our batch experiments were conducted on a total of 1,194 samples. The reason for discarding images requiring resizing is that adapting the relevant code for such cases would be cumbersome and irrelevant to our experimental conclusions. We did not conduct the experiments on the entire COCO dataset due to the excessive computational time required. Instead, we employed uniform sampling, which does not affect the validity of our experimental 15
Figure 5: Interpretation of category outputs (logits and probabilities) of the DETR model. Table 1: Batch experiments on DETR ODAM Error Class (Prob) Class (Logit) Box (Left) Box (Top) Box (Right) Box(Bottom)
0.9834 15.8498 344.9110 160.3399 176.7152 341.4271
Runtime (min) 2.9643 2.8856 2.8563 3.0020 2.8510 2.9388
Ours (resnet) Error 0.2711 2.6590 95.9336 58.9888 98.6844 48.6525
Runtime (min) 168.5809 115.8652 237.4421 255.5348 265.6704 244.6687
Ours (encoder) Error 0.0286 0.5000 29.2509 19.0098 30.0953 14.9996
Runtime (min) 35.5282 29.9284 78.2876 68.0288 77.6250 73.7890
conclusions. Our method consists of two stages: in the first stage, we set the number of integration error steps to 256, and then selected samples with a relative error (| output−output |) greater than 0.5 baseline for separate model interpretation with 2,000 integration steps. Finally, we compiled the experimental results and recorded the computational time. The results are presented in Table 1. Please note that since ODAM always has an unknown input reference, it is inherently imprecise in principle. From this perspective, there is no need for us to analyze its model interpretation effectiveness. Purely from a mathematical standpoint, we can observe from Table X that ODAM performs poorly. Given that ODAM and our method have different baselines, strictly speaking, it is not appropriate to compare the two methods. If we forcibly compare them based on the data attribution results, we find that in terms of error, our methods (ours (encoder) and ours (resnet)) perform the best, while ODAM performs the worst. Regarding RE, ours (encoder) performs the best. Although the error of ours (resnet) is smaller than that of ODAM, the difference is not significant in terms of relative error. This is because relative error is calculated with the difference between the current output and the ultimate output as the denominator. As a result, even though our method performs better in terms of error, it does not show an advantage in relative error. When we increase the combination of integration steps to 500 and 5000, the results are completely superior to those of ODAM. Table 1 shows that our method is much more accurate than ODAM, and it also verifies that the closer the features used for model interpretation are to the output being explained, the better the explanation accuracy under the same conditions. This is because the closer the features are to the output, the 16
Figure 6: Interpretation of category outputs (logits and probabilities) in the DETR model by ODAM and GAE. (ODAM strictly adheres to the original settings and provides interpretation based on the output features from ResNet.) lower the degree of nonlinearity between the features and the output, and thus the higher the accuracy of IG. Additionally, we emphasize again that the drawback of ODAM is not merely its poor performance shown in Table 1; more importantly, ODAM conceals a subtle yet fundamentally flawed weakness — ODAM lacks an explicit input baseline (refer to Section 5.5), which makes ODAM inherently imprecise from the perspective of baseline analysis. 6.2
Interpretation of VGG
We select VGG for experiments which is also used in LayerCAM paper. Since our interpretation within the DETR model already includes category interpretation, it is actually unnecessary to continue conducting experiments on the VGG model. However, the VGG network is relatively simpler and can more clearly reveal certain patterns. It is worth noting that, like ODAM, LayerCAM also lacks an explicit input baseline, which is an easily overlooked drawback. In the calculation process of LayerCAM, only the positive values of gradients are taken, which further increases the error in the interpretation results. Therefore, we present the original LayerCAM results, along with the interpretation results of a modified LayerCAM model that retains negative gradient values, as well as the results of our method. The experiments were designed in accordance with the original LayerCAM paper and were conducted in five stages. The experimental case 17
Figure 7: VGG category logits interpretation.
demonstrations are shown in Figure 7. Similarly, we uniformly sampled 5,000 ImageNet images for batch experiments, and the experimental results are presented in Table 2. In the experiments depicted in Figure 7, the number of integration steps for our method was set to 500. In the batch experiments presented in Table 2, the number of integration steps was set to 256. Unlike the experiments in DETR, we no longer perform integration with more steps for samples with larger errors in this context. Upon observing Figure 7, it can be seen that the modified LayerCAM exhibits smaller errors, with our method achieving the smallest errors. Although the interpretation results of the modified version appear visually similar to our method, our approach outperforms it in terms of error reduction. Additionally, it is crucial to note that neither the original LayerCAM nor the modified LayerCAM has a definitive baseline image, rendering the interpretation results of both methods less rigorous. Furthermore, it can be observed that the model interpretation results based on deep-level features yield smaller errors. We believe this is because features closer to the output exhibit lower degrees of nonlinearity between the features and the outputs, thereby requiring fewer integration steps for IG and reducing the error of single-step IG. This manifests as a gradual reduction in errors in the two special cases of single-step integration (original LayerCAM and modified LayerCAM). This observation is largely validated in batch experiments, with the only imperfection being that our method does not support this conclusion based on the results from the fourth and fifth stages. This may require further exploration, and what we present here is merely a possible conjecture. 18
Table 2: Batch experiments on VGG
Stage 1 Stage 2 Stage 3 Stage 4 Stage 5
Original LayerCAM Runtime Error (min) 287.67038 0.1870 164.09166 0.1482 91.092766 0.1491 45.233196 0.1545 17.100424 0.1529
Modified LayerCAM Runtime Error (min) 9.277087 0.3714 8.107285 0.1535 7.3210387 0.1501 5.9659696 0.1507 0.8659283 0.1548
Ours Error 0.18437183 0.10293762 0.08678595 0.07759022 0.13428052
Runtime (min) 7.7305 5.5729 3.3586 1.5628 0.8807
Additionally, please note that while the interpretation results of the modified LayerCAM in the VGG model show comparable effectiveness to our method, its performance is significantly inferior when applied to the DETR model.
7
Limitations
Here we discuss a limitation of our baseline methods. We note that ODAM also conducted experiments on FCOSTian et al. [2019] and Faster R-CNNRen et al. [2015]. Unlike DETR, these two models do not have stable outputs. The outputs of FCOS and Faster R-CNN are not fixed — the number of detected bounding boxes may vary across different images. However, Integrated Gradients requires a fixed output, as without stable outputs, there is no consistent integral. This is a limitation of applying the IG method. Nevertheless, this limitation is not entirely insurmountable. For instance, FCOS makes continuous predictions for each pixel, and Faster R-CNN produces fixed convolutional features. Interpreting these fixed features and, in turn, interpreting the model’s inference may be a viable path forward. The focus of this study, as indicated by the title, is to emphasize the baseline that has been overlooked in the field of model interpretation, rather than to implement corresponding engineering applications on all models. Our future work will focus on applying our method to models without fixed outputs, such as FCOS and Faster R-CNN. It is important to note that ODAM, or LayerCAM, can be applied to such models, but the uncertain baseline renders them imprecise. From another perspective, LayerCAM and ODAM are essentially using instantaneous states to evaluate a process, or in other words, using single-step integration to approximate the complete integral. The stronger the nonlinearity of the network, the larger the error — these are unavoidable issues.
8
Conclusion
Our work emphasizes the long-overlooked baseline issue in the field of model interpretation. In this paper, we reformulate the task of model interpretation and the interpretation principles for model interpretation results to demonstrate the importance of the baseline. We further unify gradient-based methods, Integrated Gradients (IG) methods, and Taylor expansion, clarifying the connections among them and explicitly identifying the baseline for each method. On this basis, we analyze the flaws and errors in related model interpretation methods (IG, LayerCAM, ODAM, Difference Map). We advocate evaluating the quality of model interpretation results precisely through the attribution error between the attribution result and the attribution target, rather than adopting flawed evaluation methods, such as those based on marginal-effect or the assumption of perfect model performance. We revise IG and develope a model interpretation method with a clear and reasonable baseline, achieving better results. Our method supports model interpretation based on features from any layer. Interpretation based on features from different layers are all reasonable, and the differences among these results reflect varying degrees of feature extraction at different feature extraction stages. This work is not intended to target any researchers or their achievements; any reference to researchers or achievements is solely for the needs of academic research. 19
References Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. In Jian Su, Xavier Carreras, and Kevin Duh, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1955–1960. The Association for Computational Linguistics, 2016. doi: 10.18653/V1/D16-1203. URL https://doi.org/10.18653/v1/d16-1203. David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and KlausRobert Müller. How to explain individual classification decisions. J. Mach. Learn. Res., 11: 1803–1831, 2010. doi: 10.5555/1756006.1859912. URL https://dl.acm.org/doi/10.5555/ 1756006.1859912. Oren Barkan, Yehonatan Elisha, Jonathan Weill, Yuval Asher, Amit Eshel, and Noam Koenigstein. Deep integrated explanations. In Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhang, and Rodrygo L. T. Santos, editors, Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023, pages 57–67. ACM, 2023. doi: 10.1145/3583780.3614836. URL https://doi.org/10.1145/3583780.3614836. Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In Alessandro E. P. Villa, Paolo Masulli, and Antonio Javier Pons Rivero, editors, Artificial Neural Networks and Machine Learning - ICANN 2016 - 25th International Conference on Artificial Neural Networks, Barcelona, Spain, September 6-9, 2016, Proceedings, Part II, volume 9887 of Lecture Notes in Computer Science, pages 63–71. Springer, 2016. doi: 10.1007/978-3-319-44781-0\_8. URL https://doi.org/10.1007/978-3-319-44781-0_8. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, Lecture Notes in Computer Science, pages 213–229. Springer, 2020. doi: 10.1007/978-3-030-58452-8\_13. URL https://doi.org/10.1007/978-3-030-58452-8_13. Aditya Chattopadhyay, Anirban Sarkar, Prantik Howlader, and Vineeth N. Balasubramanian. Gradcam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, Lake Tahoe, NV, USA, March 12-15, 2018, pages 839–847. IEEE Computer Society, 2018. doi: 10.1109/WACV.2018. 00097. URL https://doi.org/10.1109/WACV.2018.00097. Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 387–396. IEEE, 2021. doi: 10.1109/ICCV48922.2021.00045. URL https://doi.org/10.1109/ICCV48922.2021. 00045. Jiamin Chen, Xuhong Li, Lei Yu, Dejing Dou, and Haoyi Xiong. Beyond intuition: Rethinking token attributions inside transformers. Trans. Mach. Learn. Res., 2023, 2023. URL https: //openreview.net/forum?id=rm0zIzlhcX. Alexandre Englebert, Sédrick Stassin, Géraldin Nanfack, Sidi Ahmed Mahmoudi, Xavier Siebert, Olivier Cornu, and Christophe De Vleeschouwer. Explaining through transformer input sampling. In IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Workshops, Paris, France, October 2-6, 2023, pages 806–815. IEEE, 2023. doi: 10.1109/ICCVW60793.2023.00088. URL https://doi.org/10.1109/ICCVW60793.2023.00088. Jindong Gu, Yinchong Yang, and Volker Tresp. Understanding individual decisions of cnns via contrastive backpropagation. In C. V. Jawahar, Hongdong Li, Greg Mori, and Konrad Schindler, editors, Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III, volume 11363 of Lecture Notes in Computer Science, pages 119–134. Springer, 2018. doi: 10.1007/978-3-030-20893-6\_8. URL https://doi.org/10.1007/978-3-030-20893-6_8. 20
Derek Hoiem, Yodsawalai Chodpathumwan, and Qieyun Dai. Diagnosing error in object detectors. In Andrew W. Fitzgibbon, Svetlana Lazebnik, Pietro Perona, Yoichi Sato, and Cordelia Schmid, editors, Computer Vision - ECCV 2012 - 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part III, volume 7574 of Lecture Notes in Computer Science, pages 340–353. Springer, 2012. doi: 10.1007/978-3-642-33712-3\_25. URL https: //doi.org/10.1007/978-3-642-33712-3_25. Brian Kenji Iwana, Ryohei Kuroki, and Seiichi Uchida. Explaining convolutional neural networks using softmax gradient layer-wise relevance propagation. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pages 4176–4185. IEEE, 2019. doi: 10.1109/ICCVW.2019.00513. URL https: //doi.org/10.1109/ICCVW.2019.00513. Peng-Tao Jiang, Chang-Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Exploring hierarchical class activation maps for localization. IEEE Trans. Image Process., 30: 5875–5888, 2021. doi: 10.1109/TIP.2021.3089943. URL https://doi.org/10.1109/TIP. 2021.3089943. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1885–1894. PMLR, 2017. URL http:// proceedings.mlr.press/v70/koh17a.html. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740–755. Springer, 2014. doi: 10.1007/ 978-3-319-10602-1\_48. URL https://doi.org/10.1007/978-3-319-10602-1_48. Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognit., 65:211–222, 2017. doi: 10.1016/J.PATCOG.2016.11.008. URL https://doi.org/ 10.1016/j.patcog.2016.11.008. Vitali Petsiuk, Rajiv Jain, Varun Manjunatha, Vlad I. Morariu, Ashutosh Mehra, Vicente Ordonez, and Kate Saenko. Black-box explanation of object detectors via saliency maps. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 11443– 11452. Computer Vision Foundation / IEEE, 2021. doi: 10.1109/CVPR46437.2021.01128. URL https://openaccess.thecvf.com/content/CVPR2021/html/Petsiuk_Black-Box_ Explanation_of_Object_Detectors_via_Saliency_Maps_CVPR_2021_paper.html. Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 91–99, 2015. URL https://proceedings.neurips. cc/paper/2015/hash/14bfa6bb14875e45bba028a21ed38046-Abstract.html. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 618–626. IEEE Computer Society, 2017. doi: 10.1109/ICCV.2017.74. URL https://doi.org/10.1109/ICCV.2017.74. L. S. Shapley. A value for n-person games. Contributions to the Theory of Games, 1953. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014. URL http://arxiv.org/abs/1312. 6034. 21
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR, 2017. URL http://proceedings.mlr. press/v70/sundararajan17a.html. Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: fully convolutional one-stage object detection. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 9626–9635. IEEE, 2019. doi: 10.1109/ ICCV.2019.00972. URL https://doi.org/10.1109/ICCV.2019.00972. Shujun Xia, Chenyang Zhao, and Antoni Chan. Explaining object detection through difference map. In IEEE/CVF International Conference on Computer Vision, ICCV 2025 - Workshops, Honolulu, HI, USA, October 19-20, 2025, pages 724–733. IEEE, 2025. doi: 10.1109/ICCVW69036.2025.00080. URL https://doi.org/10.1109/ICCVW69036.2025.00080. Weiyan Xie, Xiao-Hui Li, Caleb Chen Cao, and Nevin L. Zhang. Vit-cx: Causal explanation of vision transformers. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, pages 1569–1577. ijcai.org, 2023. doi: 10.24963/IJCAI.2023/174. URL https://doi.org/10.24963/ijcai.2023/174. Tingyi Yuan, Xuhong Li, Haoyi Xiong, Hui Cao, and Dejing Dou. Explaining information flow inside vision transformers using markov chain. In eXplainable AI approaches for debugging and diagnosis., 2021. URL https://openreview.net/forum?id=TT-cf6QSDaQ. Chenyang Zhao and Antoni B. Chan. ODAM: gradient-based instance-specific visual explanations for object detection. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/ forum?id=kJWcI39kXY. Chenyang Zhao, Janet H. Hsiao, and Antoni B. Chan. Gradient-based instance-specific visual explanations for object specification and object discrimination. IEEE Trans. Pattern Anal. Mach. Intell., 46(9):5967–5985, 2024a. doi: 10.1109/TPAMI.2024.3380604. URL https://doi.org/ 10.1109/TPAMI.2024.3380604. Chenyang Zhao, Kun Wang, Xingyu Zeng, Rui Zhao, and Antoni B. Chan. Gradient-based visual explanation for transformer-based CLIP. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b. URL https: //openreview.net/forum?id=WT4X3QYopC. Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2921–2929. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.319. URL https://doi.org/10.1109/ CVPR.2016.319.
22