arXiv CS open access

Improving Context-Shift Robustness of Convolutional Networks via Context-Regularized Cross-Entropy

Jinen Setpal et al. · arxiv_cs

arXiv CS · Papers

deep-learning machine-learning neural-networks

Image classification models often encounter objects in diverse and unpredictable contexts at test time, which may differ substantially from those seen during training. This phenomenon, termed context shift, exposes a key fragility in neural networks: when models rely on non-causal contextual cues fo

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Improving Context-Shift Robustness of Convolutional Networks via Context-Regularized Cross-Entropy

arXiv:2611.08157v1 [cs.CV] 5 Nov 2026

Jinen Setpal Elmore Family School of Electrical and Computer Engineering Purdue University jsetpal@purdue.edu Chaoyue Liu Elmore Family School of Electrical and Computer Engineering Purdue University cyliu@purdue.edu

Abstract Image classification models often encounter objects in diverse and unpredictable contexts at test time, which may differ substantially from those seen during training. This phenomenon, termed context shift, exposes a key fragility in neural networks: when models rely on non-causal contextual cues for prediction, shifts in context can lead to significant performance degradation. In this paper, we propose a novel training framework to improve the robustness of convolutional neural networks under context shift. Our approach leverages spatial feature attribution to guide models toward making predictions that rely less on contextual regions and more on object-relevant features. As a first step, recognizing a theoretical limitation of existing feature attribution methods we introduce a new variant, ContrastiveCAMs, which produces more faithful attribution maps of model predictions. Building on ContrastiveCAMs, we further propose Context-Regularized Cross-Entropy (CRCE), a modification of the standard cross-entropy loss that regularizes the model’s attention by suppressing the influence of contextual regions, thereby improving robustness to context shift. We evaluate the effectiveness of our approach on several medium-to-large scale datasets (Waterbirds, Spawrious, ImageNet/ImageNet-BG, Hard-ImageNet), and report consistent improvements in context-shift robustness.

Introduction

The vast applications of convolutional neural networks in safety-critical domains such as medical imaging [38, 33], forensic investigation [26] and self-driving [21] make trustworthy and robust predictions essential for large-scale adoption. It has been demonstrated that existing neural networks, despite achieving remarkable performance on large-scale datasets, heavily rely on supporting context (i.e., background area within the images) to determine their predictions [41, 23, 30]. This sensitivity to surrounding context can degrade performance when the same object appears in a different environment [46, 15, 3, 4], whether due to novel object-context combinations in the test data or adversaries manipulating target-agnostic regions to mislead the model. We highlight this concern in Figure 1 using the Waterbirds-100 dataset [37] which only has the object-context combinations of waterbird-waterfront and landbird-land. When a waterbird is shown in the context of land, the model predicts the class landbird with high confidence, due to the influence of the context dominating over the influence of the object. The converse also occurs when a landbird appears in a waterfront context. This concerning lack of robustness has been observed across safety40th Conference on Neural Information Processing Systems (NeurIPS 2026).

Attention Maps (HiResCAMs)

Original Image

Original Image Cross-Entropy (Baseline)

Class: Waterbird Context: Land

Prediction: Landbird Confidence: 99.4%

❌

CR-CE (Ours)

Prediction: Waterbird Confidence: 93.9%

Cross-Entropy (Baseline)

✅

Class: Landbird Context: Waterfront

Prediction: Waterbird Confidence: 95.8%

❌

CR-CE (Ours)

Prediction: Landbird

✅

Confidence: 97.5%

Figure 1: Context Contributions Can Mislead Predictions. Spatial feature attribution maps reveal that cross-entropy trained models rely heavily on context, causing misclassifications in novel environments. Our approach encourages focus on the object thereby improving context-shift robustness.

critical applications spanning the diagnosis of skin cancer [27], pneumothorax [36], as well as in the generalization capabilities of robot foundation models [48]. In this paper, we aim to improve the context robustness of convolutional neural networks by leveraging spatial feature attributions. We begin by noting a theoretical limitation of existing feature attribution methods: they permit an arbitrary constant shift applied uniformly across all output logits, which undermines their ability to provide faithful explanations. As a result, the attributed contributions from context and object regions may not accurately reflect their true influence, as they can be distorted by this constant shift. We address this limitation by proposing a new variant ContrastiveCAMs, based on a prior method, HiResCAMs [10]. We further theoretically confirm that ContrastiveCAMs provide faithful explanations to the neural networks final predictions. To address sensitivity to context shifts, we propose a new loss function, Context-Regularized CrossEntropy (CR-CE), which leverages the proposed feature attribution method, ContrastiveCAMs, to encourage the model to focus on the primary object by suppressing contributions from the surrounding context. This approach requires object-context segmentation during training (crucially, with no segmentation required during inference), which can be readily obtained using existing pretrained segmentation models such as SAM3 [5] and Masked AutoEncoders (MAE) [13, 18]. Notably, the segmentation need not be perfect. In our experiments on the ImageNet dataset, we use MAEs to generate object-context segmentation. Experimental results demonstrate that our method effectively reduces the context contribution to model predictions and improves convolutional networks’ robustness to context shifts consistently across a broad class of datasets. Our key contributions are summarized as follows: • We identify a theoretical limitation of existing spatial feature attribution methods that hinders explanation faithfulness. We address this limitation by proposing a new method ContrastiveCAMs. • We propose a new loss function, Context-Regularized Cross Entropy (CR-CE), that encourages model predictions to focus on the object region rather than the context. • Experimental evaluations demonstrate that the proposed training approach effectively attracts the focus of feature attribution towards the object region, thus enhancing context-shift robustness. Notation. We denote vectors using bold lowercase letters (e.g., v), matrices using uppercase letters (e.g., M ), and tensors using bold uppercase letters (e.g., T), with partial indexing implying selection of the subtensor across the remaining subsequent dimensions (e.g., Ti ∈ Rb×c for T ∈ Ra×b×c ). We use the operator ⊙ to represent elementwise multiplication, and define [K] := {1, 2, . . . , K}. 1.1

Related Work

Feature Attribution Methods in Convolutional Networks. Class Activation Mappings (CAMs) [49] is a prominent family of methods that identify attributions of each (spatial) feature, including GradCAM [39], GradCAM++ [6], Score-CAM [45], and HiResCAM [10]. These methods typically compute feature attributions with respect to the model’s output logits, prior to the final softmax activation. However, due to the contrastive nature of the softmax function [28, 31], this can lead to inaccurate attributions for the post-softmax predictions. DiffGradCAM [31] shares a similar approach 2

to ContrastiveCAMs. But it is based on GradCAM and its connection to the model’s prediction is not theoretically clear. Context-Shift Robustness. One approach to improving robustness is through explanation-guided methods, which leverage feature attribution or saliency maps [40] to direct the model’s attention toward relevant objects and away from contextual distractions in the images [14]. This is typically achieved by augmenting the standard cross-entropy loss with additional regularization terms that penalize discrepancies between the model’s feature attribution map and a target explanation [35, 29, 34, 11]. The regularization term of Right for the Right Reasons (RRR) [35] penalizes the gradients of log-prediction w.r.t. the irrelevant input features. GALS [29] regularizes the Grad-CAM of the model’s predictions toward a target feature attribution map generated using training labels and auxiliary vision-language models. The regularization term of EBL [34] penalizes the positive attributions in contextual regions of the feature attribution map. ABNs [11] introduce an auxiliary attention module to find key regions of images to assist convolutional networks in making predictions. In contrast, our method directly modifies the cross-entropy loss rather than introducing an additional regularization term, so that robustness to context shifts is built in intrinsically. An orthogonal approach is worst-group generalization [2, 7, 37]. Rather than optimizing for overall accuracy, this line of work partitions the dataset into groups defined by object-context combinations and seeks to balance performance across groups by improving the worst-performing group.

Motivation

In image classification tasks, deep neural networks make their predictions based on both the object and the context, i.e., the backgrounds in images. It has been observed that irrelevant, or even inversely correlated, context can make a substantial contribution to the model predictions, lowering model’s confidence on correct predictions, or even overturning the prediction into incorrectness with high confidence [47, 15, 30]. This reliance renders the model less robust to context shifts, such as when a trained model is deployed in a different environment, and more vulnerable to adversarial manipulation; for instance, when an attacker deliberately replaces the context with one commonly associated with other classes. In contrast, humans are able to distinguish between key features and spurious context; where deemed necessary, they can focus on the object region, neglecting the context, so as to accurately classify the relevant object in question. As an illustration of this issue, consider the Waterbirds dataset [37], which consists of two classes of birds: waterbirds and landbirds. During training, most birds appear in their natural habitat: waterbirds are present by waterfronts, whereas landbirds are present surrounded by land. However, during testing, a context shift may occur, where a landbird (or a waterbird) appears in a waterfront context (or a land context, respectively). Neural networks often make incorrect predictions on images with context shift. For example, with a fine-tuned ResNet-50, the context region on average accounts for 46% of the model’s prediction score compared to the influence from the object. Moreover, the model’s accuracy drops from 98% for normal images down to 41% on images with context shift, which is even worse than the random guess accuracy of 50%. Using HiResCAMs [10], a member of the CAM (short for Class Activation Mapping) family that produces a heatmap highlighting the contribution from spatial locations of the image to the network output [49], we can visualize the context’s contributions to the model prediction, see Figure 1 for example. We see that when the image context is correlated with a different class, the misleading context makes large contributions to the prediction resulting in misclassifications with high confidence. This shows the pervasiveness of the context robustness concern, supported by broader statistics across other datasets in Table 1. Our observation is that the neural networks, together with its optimization problem, do not possess an intrinsic mechanism to distinguish objects from context. To see this, consider a neural network f (before the softmax), such as a ResNet-50, composed of a convolutional backbone g and a bias-free linear classifier h: f = h ◦ g. Given an input image X, denote the feature embedding g(X) = A ∈ Rd0 ×d1 ×d2 and the final logit f (X) ∈ RK , where K is the number of classes, d0 is the number of features (a.k.a. channels) and d1 and d2 are the spatial dimensions of the final convolutional layer. HiResCAMs, defined as follows, Ak =

d0 X (∇Aj fk ) ⊙ Aj , j=1

∀k ∈ [K],

(1)

have the following relation with the network output logits [10]: fk (X) =

dX 1 ,d2

(Ak (X))ij ,

∀k ∈ [K].

(2)

i,j=1

Remark 2.1. Other CAM methods, such as GradCAM [39] and GradCAM++ [6], do not possess the above relation with output logits. Let H ∈ Bd1 ×d2 be the object indicator function, i.e., Hij = 1 if the (i, j)-patch corresponds to the object and Hij = 0 if the (i, j)-patch corresponds to the context. Then, the network’s final output logits can equivalently be written as: fk (X) =

dX 1 ,d2

(H ⊙ Ak (X))ij +

i,j=1

dX 1 ,d2

((1 − H) ⊙ Ak (X))ij ,

∀k ∈ [K],

(3)

i,j=1

where the first and second terms correspond to contributions from the object and context regions respectively. Here, we can see that the context contributes to the prediction in the same way as the object regions, and as seen in the demonstration in Figure 1, the network does not learn to suppress the context contributions. In this paper, we seek to alleviate this issue of sensitivity to context shift with a focus on convolutional neural networks. Specifically, we will utilize a variant of HiResCAMs to design a new loss function that is capable of suppressing context contributions, encouraging the models to make predictions by primarily focusing on the objects thus improving their robustness to context shift.

From HiResCAMs to ContrastiveCAMs

In this section, we first present a fundamental issue of HiResCAMs in the multiclass setting, which degrades the faithfulness of feature attribution maps. We then propose a variant, ContrastiveCAMs, which produces accurate, class-specific attribution maps even in the multiclass setting. HiResCAMs Admit Spurious Shifts. A key drawback of HiResCAMs emerge as a consequence of applying softmax activation on the network’s output logits f to obtain the final predictions f˜ = σ(f ), which are typically interpreted as class probabilities. Proposition 3.1 (Redundancy in the softmax function). ∀x ∈ RK , a ∈ R.

σ(x) = σ(x + a1K ),

(4)

Proof. The proof is deferred to Appendix A. The above proposition shows that, given a prediction f˜, the logit vector f is not unique: if each logit is shifted by the same value, the final probability predictions f˜ remain unchanged. Therefore, the value of each individual logit fk alone does not determine the final prediction. This redundancy has also been observed recently by Oh and Noh [28], Piland et al. [31], and discussed in the context of GradCAMs. Here, we show that this redundancy is inherited by HiResCAMs, and further pronounced in terms of degrees of freedom, via Eq. (2). Theorem 3.2. Let A ∈ RK×d1 ×d2 be a HiResCAM. Given an arbitrary matrix M ∈ Rd1 ×d2 , define a new HiResCAM as: Ak := Ak + M, ∀k ∈ [K]. (5) Then, the two HiResCAMs correspond to the same network prediction f˜(X):     dX dX 1 ,d2 1 ,d2 f˜(X) = σ  A:ij  = σ  A:ij  . (6) i,j=1

i,j=1

From this theorem, we can see that the feature attributions rendered by HiResCAMs can be shifted arbitrarily by a matrix M ∈ Rd1 ×d2 . As a result, feature attributions can spuriously correspond to any 4

-5

-1

-4

-2

-4

0 4

-8

-5

-1

-4

-5

-1

-4

-2

-3

-4

-8

.98

.02

Figure 2: Illustration of Theorem 3.2. Shifting A by arbitrary matrix M results in new explanations A. Although the explanation changed significantly, it corresponds to an identical final prediction f˜.

given region without affecting the final prediction. This is illustrated in Figure 2. These additional degrees of freedom hinder the effectiveness of regularizing against object-context segmentation during training, since the explanation can be modified without influencing the overall prediction. To address this issue, we propose to remove the redundant degrees in the feature attribution maps using a contrastive variant of HiResCAMs, termed ContrastiveCAMs. Definition 3.3 (k-ContrastiveCAMs). Given a class of interest k ∈ [K] and a set of HiResCAMs A for a given image X, k-ContrastiveCAMs are defined as follows: Ck := C(k,k′ ) : k ′ ∈ [K] , where C(k,k′ ) := Ak − Ak′ . (7) The aggregated version of k-ContrastiveCAM is defined as Ckag :=

k =1

1 X 1 X C(k,k′ ) = Ak − Ak′ . K ′ K ′

(8)

Here, C(k,k′ ) is also referred as (k, k ′ )-ContrastiveCAM. Contrary to HiResCAMs Ak , which focus on single-logit explanations, ContrastiveCAMs provide spatial feature attributions that explain why the model favors one class over a competing class, making them class-discriminative. Remark 3.4. For binary classification tasks where there is a single output logit and sigmoid applied to obtain the final predictions, HiResCAMs are intrinsically contrastive. Remark 3.5. The idea of subtracting the average CAM has been applied in DiffGradCAMs [31]. However, DiffGradCAMs are built upon GradCAMs, whereas our ContrastiveCAMs are built upon HiResCAMs. With this distinction, both the following key properties hold for ContrastiveCAMs but not for DiffGradCAMs (c.f. Eq.(2) and Remark 2.1). Proposition 3.6 (Uniqueness of ContrastiveCAMs). Let A and A be two HiResCAMs corresponding to probability predictions f˜(X) ∈ RK such that: Ak = Ak + M,

∀k ∈ [K],

(9)

with M ∈ Rd1 ×d2 being an arbitrary matrix. Then, the ContrastiveCAMs C and C, derived from A and A respectively, are the same: C = C,

and

C ag = C .

(10)

Thus ContrastiveCAMs remove the extra degree of freedom, resulting in faithful feature attribution maps. ContrastiveCAMs also have a close connection to the model’s class probabilities. Specifically, any input-dependent change to probability predictions f˜ (e.g., as a result of updating model parameters) is reflected precisely by a proportionate change to ContrastiveCAMs. 5

Proposition 3.7 (Correctness of ContrastiveCAM Explanations). Given an input X and a class index k ∈ [K], the prediction f˜k (X) is determined by the k-ContrastiveCAMs:   −1 dX K 1 ,d2 X f˜k (X) =  exp − C(k,k′ ) ij  , ∀k ∈ [K]. (11) k′ =1

i,j=1

Moreover, if for some k ∈ [K], i ∈ [d1 ], j ∈ [d2 ], C(k,k′ ) ij = 0, ∀k ′ ∈ [K], then the (i, j)-patch does not influence the final prediction. Comparing Eq. (11) with Eq. (2), one can easily find that, given any class index k, the corresponding HiResCAM only determines the output logit value fk but not the final prediction f˜k ; on the other hand, ContrastiveCAMs directly determines the final prediction f˜k , which makes ContrastiveCAMs a more faithful tool to explain the model’s prediction process. Moreover, the above proposition also shows that, iff the ContrastiveCAMs are zero in a certain spatial region, the contribution from this region of the image to the prediction is indeed zero. HiResCAMs do not share this property.

Learning with Context-Regularized Cross-Entropy Loss

We now introduce a learning framework designed to be robust to context shifts. In particular, we propose a novel loss function, called Context-Regularized Cross-Entropy (CR-CE), which inhibits the context from contributing to the model prediction. This regularization is guided using the feature attributions produced by ContrastiveCAMs. To effectively regularize against context contributions during training, prior knowledge of the contextual regions in the input images is generally required. Such information can be obtained in two main ways: (i) through segmentation annotations provided by the dataset (e.g., PASCAL VOC [16], MS-COCO [22]), or (ii) by leveraging pretrained object detection or segmentation models (e.g., Segment Anything Model [5], Masked AutoEncoders [18]) to estimate bounding boxes or object masks (see experiments on ImageNet, Spawrious datasets in Section 5). In practice, the first option is often unavailable or prohibitively expensive. We adopt the second approach which is more accessible, as pretrained models can provide approximate context localization with minimal effort. Note that, our framework does not require precise object segmentation or localization, and rough estimates are often sufficient to effectively regularize contextual influence. In Appendix C.4, we show numerical evidence for the robustness of our framework to noisy object-context segmentation. Given an image X, denote H ∈ Bd1 ×d2 as the binary mask that distinguishes object regions from context. Specifically, for each (i, j)-patch where i ∈ [d1 ], j ∈ [d2 ], Hij = 1 if the patch corresponds to the object, and Hij = 0 if it corresponds to the context. We define CR-CE as follows: Definition 4.1 (Context-Regularized Cross-Entropy Loss). Given a training dataset D and convolutional network f , the CR-CE loss is defined as: LCRCE (D) = E(X,y,H)∼D [ℓ(X, y, H)], with    d ,d K 1 2 X X  ℓ(X, y, H) = log  exp  −Hij C(kt ,k) ij + (1 − Hij ) |( C(kt ,k) )ij | . | {z } i,j=1 k=1

(12)

context regularization

where kt is the index of the ground-truth class. For simplicity of notation, we do not explicitly write the dependence of H and ContrastiveCAMs C on the input image X. The context regularization term in the CR-CE loss regularizes the absolute values of the ContrastiveCAMs |(C(kt ,k) )ij | at context regions, driving them towards zero, thus minimizing the influence of context in the final prediction f˜ (c.f. Proposition 3.7). We also extend context regularization to binary classification settings; we defer this discussion to Appendix B. Comparison with standard cross-entropy loss. The only difference between the standard crossentropy (CE) loss and CR-CE lies in the context regularization term. In particular, if the absolute values of the ContrastiveCAM, |(C(kt ,k) )ij |, are replaced with their negated values, −(C(kt ,k) )ij , the 6

CR-CE loss reduces to the standard cross-entropy loss, noting that cross-entropy loss on one sample ℓCE (X, y) = − log f˜kt as well as the relation in Proposition 3.7. Despite this seemingly minor modification, the effect on model behavior is substantial. The standard cross-entropy loss implicitly encourages the context to cooperate with the object regions in making predictions. While this is beneficial when the context is consistent with the object, it can lead to an over-reliance on contextual cues and thus reduce robustness under context shift. In contrast, CR-CE explicitly penalizes excessive reliance on context, encouraging the model to base its predictions primarily on the object itself. As a result, when the context is altered or adversarially manipulated, models trained with CR-CE exhibit improved robustness. Further, for models using a linear classifier head (e.g., ResNet [17], MobileNet [19], EfficientNet [42]), ContrastiveCAM can be computed in closed form, see Eqs. (1) and (7); this results in a minor computational overhead when CR-CE is used in place of cross-entropy.

Experimental Results

In this section, we numerically evaluate the effectiveness of training with the proposed CR-CE loss function in both promoting object-focused learning and improving robustness to context shifts. 5.1

Datasets and Experimental Setup

Datasets.

We evaluate our method using the following four datasets:

• Waterbirds-100 dataset [37] consists of images of waterbirds and landbirds. In the training set, each class of bird appears in the context of their natural habitat–waterfront and land, respectively. However, in the test set, context shifts may occur, with birds appearing in the opposite environment. • Spawrious dataset [23] consists of AI-generated images of four classes of dog breeds appearing in various contexts. The test set contains combinations of dog breed and context that are not seen during training, thereby assessing models’ robustness to context shift. • ImageNet [8] with ImageNet-BG [30] as test set. We train the model using ImageNet-1k and test it using ImageNet-BG, the latter of which features images with manually replaced backgrounds, designed to decouple object recognition from context cues. • Hard-ImageNet [25] is a subset of ImageNet, consisting of 15 selected classes of images that have high context influence. It is used to investigate the context’s influence on the classification by comparing the model outputs with and without the objects of the images. For all the datasets above, context shifts only occur in the test set, which is not accessed during training. Ground-truth object-context masks are available for the training sets of Waterbirds-100 and Hard-ImageNet. For Spawrious and ImageNet where object-context segmentation is not available, we use existing segmentation models such as SAM3 [5] and Masked AutoEncoders (MAE) [18, 13] to generate these masks. Our approach allows for imperfect segmentation, enabling the effective use of off-the-shelf vision models when ground truth masks are unavailable. Experimental Setup. We follow the methodology in the benchmark datasets [37, 23, 30, 25] and fine-tune ResNet-50 models pre-trained on ImageNet-1k. In our experiments, we also evaluate a modified ResNet architecture where the final downsampling convolution layer’s stride is set to (1, 1), thus doubling the resolution of ContrastiveCAMs, i.e., d1 = d2 = 14. The higher-resolution attribution maps provide more precise localization of object regions. More experiments with different architectures can be found in Appendix C.5. We compare our proposed CR-CE framework with the following prior approaches: • The standard training procedure with cross-entropy loss (CE), which does not implement any context-shift robustness technique. It serves as a baseline for comparison. • Explanation-guided methods, which improve context-shift robustness by guiding models’ feature attributions, i.e., the heatmaps, towards the target object region. Our proposed method falls in this category. We compare with existing methods: attention branch networks (ABN) [11], GALS [29], and training with energy-based loss (EBL) [34]. 7

Original Image

Class: Class: Baseball Bobsled Player

Attention Attention Maps (ContrastiveCAMs) Maps Cross-Entropy Cross-Entropy (Standard) (Baseline)

CR-CE CR-CE (Ours) (Ours)

Context Context Util.: Util.: 61.4% 67.38%

Context ContextUtil.: Util.: 15.25% 0.3%

Confidence: Confidence: 99.9% 92.79%

Confidence: Confidence: 99.9% 86.40%

Original Image

Class: Class: Gondola Bee

Attention Attention Maps (ContrastiveCAMs) Maps Cross-Entropy Cross-Entropy (Standard) (Baseline)

CR-CE CR-CE (Ours) (Ours)

Context Context Util.: Util.: 55.4% 68.26%

Context Context Util.: Util.: 12.7% 7.91%

Confidence: Confidence: 96.0% 99.99%

Confidence: Confidence: 98.7% 99.91%

Figure 3: Qualitative Evaluation. Models trained using cross-entropy rely heavily on context: in the Hard-ImageNet (left) and ImageNet-BG (right) examples shown above, the baseball player and bee are predicted using the playing field (61.4% of the prediction) and the flower (55.4% of the prediction) respectively. On the other hand, models trained using CR-CE leverage object-specific regions and localize the target without the use of object-context segmentation during inference.

Table 1: Context Contribution Ratios. On the test set of each context-shift robustness benchmark we consider, CR-CE trained models primarily rely on object-specific regions to determine predictions. Context Contribution Ratio R% (↓) CAM Approach Size Waterbirds-100 Spawrious‡ ImageNet-BG Hard-ImageNet 7×7

CE (Baseline) CR-CE (Ours)

46.08±0.43 12.86±0.09

44.79±0.3 14.89±0.2

31.83±0 14.85±0

68.53±0.04 35.85±0.26

14 × 14

CE (Baseline) CR-CE (Ours)

58.75±0.12 9.54±0.28

47.32±0.09 10.98±0.12

28.27±0.27 9.32±0.06

66.5±0.07 31.12±0.1

† ‡

No std. dev. is reported; we use pre-trained weights from torchvision without modification. We report metrics on the hardest subtask and difficulty (M2M-hard) within the Spawrious benchmark.

• Other methods, including Context Masking training [1] and Causal Feature Alignment [44]. Both methods use object-context masks to transform the input/hidden features in order to discourage context reliance. Particularly, Context Masking only uses the object regions of images for training by masking out all context regions. For fairness of comparison, we use identical object-context masks across approaches. For all our metrics, we report the mean and standard deviation computed over 3 independent runs. Details of the experimental setup are described in Appendix E.

5.2

CR-CE Suppresses Context Contributions

First, we evaluate the proportion of context contributions using the following metric: Pd1 ,d2 ag

R(C , H) =

ag i,j=1 |(Ckt )ij |(1 − Hij ) , Pd1 ,d2 ag i,j=1 |(Ckt )ij |

(13)

where kt ∈ [K] is the target class index. This metric represents the portion of the absolute value of the ContrastiveCAM heatmap from the context region. A comparison of the proposed CR-CE training framework with the standard CE training is shown in Table 1, with Figure 3 and Figure 5 presenting examples. Comparisons with the other methods assessed in this work are included in Table 5. We observe that training with CR-CE effectively guides ContrastiveCAMs onto the object regions and reduces the model’s reliance on the context region. Additional evaluation of the effectiveness of CR-CE in suppressing context contributions using the Hard-ImageNet evaluation suite can be found in Appendix C.2. 8

Table 2: Context-Shift Robustness on Waterbirds-100, ImageNet, and Spawrious datasets. Accuracy across various benchmarks, with the best context-shifted performance highlighted in bold and the highest in-distribution performance underlined. Models trained with CR-CE generalize effectively to novel environments at a minor cost to in-distribution performance. Waterbirds-100

CAM Size

Approach

Spawrious‡

ImageNet/ImageNet-BG

No Shift (↑)

Shifted (↑)

No Shift (↑)

Shifted (↑)

Easy (↑)

Medium (↑)

Hard (↑)

7×7

CE (Baseline) GALS EBL CFA Context Masking CR-CE (Ours)

98.66±0.10 97.54±1.01 96.35±0.49 93.43±0.13 91.16±3.18 94.38±1.21

41.74±4.15 54.45±6.75 51.31±4.70 72.71±0.64 61.67±6.17 73.30±4.43

82.92† 82.68±0.06 82.88±0.08 82.71±0.03 80.21±0.07 81.85±0.03

74.83† 73.82±0.10 74.79±0.01 73.85±0.09 72.87±0.10 75.75±0.08

90.86±0.54 93.47±0.36 92.43±1.31 89.14±0.89 92.08±1.04 95.55±0.16

79.46±0.48 84.58±0.88 87.23±0.84 85.68±0.68 85.03±2.37 91.25±0.18

81.52±1.03 78.74±0.97 81.21±0.74 81.77±1.73 78.46±1.12 87.49±0.14

14 × 14

CE (Baseline) ABN GALS EBL CFA Context Masking CR-CE (Ours)

98.48±0.09 97.39±0.55 97.39±0.28 96.33±0.95 93.09±0.74 92.27±1.18 94.36±0.12

38.40±1.79 51.87±6.14 63.21±4.90 51.49±5.31 77.51±3.89 64.39±1.79 79.13±0.77

82.33±0.10 79.91±0.06 82.43±0.04 82.72±0.09 79.54±0.09 80.70±0.07 81.25±0.03

74.72±0.06 69.07±0.15 75.21±0.02 75.79±0.20 73.48±0.03 75.49±0.10 76.84±0.12

88.98±0.14 89.65±0.44 95.03±1.35 93.46±0.73 87.22±0.78 90.96±0.24 96.47±0.06

73.38±0.87 72.81±1.32 87.63±3.80 89.05±0.47 85.68±0.68 85.70±2.99 92.41±0.29

79.76±1.01 63.96±6.54 84.08±1.87 79.36±2.83 75.88±1.12 80.80±0.71 88.66±0.52

† ‡

No std. dev. is reported; we use pre-trained weights from torchvision without modification. The reported metrics correspond to the many-to-many spurious correlation setting; the complete benchmark is assessed in Appendix C.1.

5.3

Reduced Context-Contributions Improve Robustness to Context Shifts

Next, we demonstrate the improved performance of CR-CE trained models under context shift by comparing them against the standard cross-entropy training, existing explanation-guided learning approaches [29, 11, 34] and other methods [44, 1]. We report the numerical results in Table 2. We observe that across all datasets, the CR-CE training framework consistently improves the robustness to context shift and further outperforms models trained using cross-entropy loss and the other compared methods. This demonstrates that the proposed method successfully suppresses contribution from irrelevant or misleading context, yielding correct and robust model predictions. We also note that, when the context is relevant (as in the setting without context shift), there is a slight degradation in the prediction accuracy across tasks. This is expected because when the test dataset and training dataset have the same set of object-context combinations, the model can often infer the correct class with the help of the context. Indeed, such correlations are often not desirable if context-shift robustness is the target. Additionally, we observe that some existing explanation-guided methods may underperform the standard cross-entropy baseline on complex classification tasks, such as ImageNet, while outperforming it on simpler tasks such as the Waterbirds-100 dataset. We hypothesize that this behavior is largely due to their reliance on classical feature attribution methods (e.g. Grad-CAM and HiResCAM), which suffer from the unfaithfulness issue theoretically described in Theorem 3.2. In the binary classification setting of Waterbirds-100, HiResCAM coincides with our ContrastiveCAMs (see Remark 3.4) and does not exhibit this limitation. More discussion of explanation-guided methods on this matter can be found in Appendix C.3. Notably, we emphasize that our approach does not use object-context segmentation during inference; the promising performance of our approach suggests that the CR-CE trained models have the ability to distinguish objects from context while making predictions. Finally, we also note that the generalization gap between in-distribution and context-shifted images shrinks as explanation granularity is increased (comparing the 7 × 7 and 14 × 14 CAM resolutions).

Conclusion

In this work, we advance explanation faithfulness and context robustness in convolutional networks. We introduce ContrastiveCAMs, a feature attribution method that addresses limitations of existing CAM methods and has a strong connection to model predictions. We further propose a new loss function, Context-Regularized Cross-Entropy (CR-CE), which penalizes contributions from context regions, thereby improving models’ robustness to context shifts. Our experimental evaluations show that our method effectively attracts the feature attributions towards the object region, and enhances context-shift robustness. 9

Limitations. We identify two limitations of this work. First, the method does not naturally extend to transformer architectures, such as Vision Transformers (ViTs) [9]. This is because CAM-based spatial feature attribution methods rely on the local structure induced by convolutional layers, whereas attention mechanisms [43] mix patch tokens in a way that loses explicit spatial locality. Second, the proposed CR-CE training requires additional object-context segmentation information (albeit only during training), which requires additional cost to obtain in practice. We note that existing explanation-guided methods also rely on this information.

Acknowledgment This research used both the DeltaAI advanced computing and data resource, which is supported by the National Science Foundation (award OAC 2320345) and the State of Illinois, and the Delta advanced computing and data resource which is supported by the National Science Foundation (award OAC 2005572) and the State of Illinois. Delta and DeltaAI are joint efforts of the University of Illinois Urbana-Champaign and its National Center for Supercomputing Applications.

References [1] Ananthu Aniraj, Cassio F Dantas, Dino Ienco, and Diego Marcos. Masking strategies for background bias removal in computer vision models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4397–4405, 2023. [2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019. [3] Pedro RAS Bassi, Sergio SJ Dertkigil, and Andrea Cavalli. Improving deep neural network generalization and robustness to background bias via layer-wise relevance propagation optimization. Nature Communications, 15(1):291, 2024. [4] Christopher Boland, Keith A Goatman, Sotirios A Tsaftaris, and Sonia Dahdouh. There are no shortcuts to anywhere worth going: Identifying shortcuts in deep learning models for medical image analysis. In Medical Imaging with Deep Learning, pages 131–150. PMLR, 2024. [5] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris Coll-Vinent, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. In International conference on learning representations, volume 2026, pages 138846–138923, 2026. [6] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Gradcam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, March 2018. doi: 10.1109/wacv.2018.00097. URL http://dx.doi.org/10.1109/WACV.2018.00097. [7] Mathieu Chevalley, Charlotte Bunne, Andreas Krause, and Stefan Bauer. Invariant causal mechanisms through distribution matching. In International Conference on Learning Representations, 2022. [8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009. [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. [10] Rachel Lea Draelos and Lawrence Carin. Use hirescam instead of grad-cam for faithful explanations of convolutional neural networks. arXiv preprint arXiv:2011.08891, 2020. [11] Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention branch network: Learning of attention mechanism for visual explanation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10705–10714, 2019. [12] Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Out-ofdomain robustness via targeted augmentations. In International conference on machine learning, pages 10800–10834. PMLR, 2023. 10

[13] Shanghua Gao, Zhong-Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. IEEE transactions on pattern analysis and machine intelligence, 45(6):7457–7476, 2022. [14] Yuyang Gao, Siyi Gu, Junji Jiang, Sungsoo Ray Hong, Dazhou Yu, and Liang Zhao. Going beyond xai: A systematic survey for explanation-guided learning. ACM Computing Surveys, 56 (7):1–39, 2024. [15] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020. [16] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In 2011 international conference on computer vision, pages 991–998. IEEE, 2011. [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. [18] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022. [19] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019. [20] Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi. Improving deep learning interpretability by saliency guided training. Advances in Neural Information Processing Systems, 34:26726–26739, 2021. [21] Jinkyu Kim and John Canny. Interpretable learning for self-driving cars by visualizing causal attention. In Proceedings of the IEEE international conference on computer vision, pages 2942–2950, 2017. [22] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. [23] Aengus Lynch, Gbetondji Jean-Sebastien Dovonon, Jean Kaddour, and Ricardo Silva. Spawrious: A benchmark for fine control of spurious correlation biases. In Workshop on Spurious Correlation and Shortcut Learning: Foundations and Solutions, 2025. [24] Mazda Moayeri, Phillip Pope, Yogesh Balaji, and Soheil Feizi. A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19087–19097, 2022. [25] Mazda Moayeri, Sahil Singla, and Soheil Feizi. Hard imagenet: Segmentations for objects with strong spurious cues. Advances in Neural Information Processing Systems, 35:10068–10077, 2022. [26] Jamuna S Murthy and GM Siddesh. Ai based criminal detection and recognition system for public safety and security using novel criminalnet-228. In International Conference on Frontiers in Computing and Systems, pages 3–20. Springer, 2023. [27] Meike Nauta, Ricky Walsh, Adam Dubowski, and Christin Seifert. Uncovering and correcting shortcut learning in machine learning models for skin cancer diagnosis. Diagnostics, 12(1):40, 2021. [28] Yoojin Oh and Junhyug Noh. Beyond softmax: Dual-branch sigmoid architecture for accurate class activation maps. arXiv preprint arXiv:2511.05590, 2025. [29] Suzanne Petryk, Lisa Dunlap, Keyan Nasseri, Joseph Gonzalez, Trevor Darrell, and Anna Rohrbach. On guiding visual attention with language specification. Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 11

[30] Łukasz Piekarek and Kamil Szyc. Imagenet-bg: A toolkit and dataset for evaluating vision model robustness against background variations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 231–240, October 2025. [31] Jacob Piland, Christopher Sweet, and Adam Czajka. Diffgradcam: A class activation map using the full model decision to solve unaddressed adversarial attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1201–1210, 2026. [32] Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5):1–17, 1964. [33] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017. [34] Sukrut Rao, Moritz Böhle, Amin Parchami-Araghi, and Bernt Schiele. Studying how to efficiently and effectively guide models with explanations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1922–1933, 2023. [35] Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. Right for the right reasons: Training differentiable models by constraining their explanations. In Proceedings of the TwentySixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2662–2670, 2017. doi: 10.24963/ijcai.2017/371. URL https://doi.org/10.24963/ijcai.2017/371. [36] Khaled Saab, Sarah Hooper, Mayee Chen, Michael Zhang, Daniel Rubin, and Christopher Ré. Reducing reliance on spurious features in medical image classification with spatial specificity. In Machine Learning for Healthcare Conference, pages 760–784. PMLR, 2022. [37] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. [38] DR Sarvamangala and Raghavendra V Kulkarni. Convolutional neural networks in medical image understanding: a survey. Evolutionary intelligence, 15(1):1–22, 2022. [39] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017. [40] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations, 2014. [41] Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2022. URL https: //openreview.net/forum?id=XVPqLyNxSyh. [42] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR, 2019. [43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [44] Rahul Venkataramani, Parag Dutta, Vikram Melapudi, and Ambedkar Dukkipati. Causal feature alignment: Learning to ignore spurious background features. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4666–4674, 2024. [45] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 24–25, 2020. [46] Zifan Wang. On the Feature Alignment of Deep Vision Models Explainability and Robustness Connected at Hip. PhD thesis, Carnegie Mellon University, 2023. [47] Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09994, 2020. 12

[48] Youguang Xing, Xu Luo, Junlin Xie, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Shortcut learning in generalist robot policies: The role of dataset diversity and fragmentation. In Conference on Robot Learning, pages 3239–3266. PMLR, 2025. [49] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929, 2016.

Proofs

Proposition 3.1 (Redundancy in the softmax function). ∀x ∈ RK , a ∈ R.

σ(x) = σ(x + a1K ),

(14)

Proof. σ(x+a1K ) = PK

k=1

exk +a

(ex1 +a , ex2 +a , · · · , exK +a ) = PK

xk k=1 e

(ex1 , ex2 , · · · , exK ) = σ(x). (15)

Theorem 3.2. Let A ∈ RK×d1 ×d2 be a HiResCAM. Given an arbitrary matrix M ∈ Rd1 ×d2 , define a new HiResCAM as: Ak := Ak + M, ∀k ∈ [K]. (16) ˜ Then, the two HiResCAMs correspond to the same network prediction f (X):     dX dX 1 ,d2 1 ,d2 (17) A:ij  . f˜(X) = σ  A:ij  = σ  i,j=1

i,j=1

Proof. The proof for this statement follows from the relation between the two HiResCAMs. For a given shift M , the predictions from the corresponding HiResCAM A can be expressed as:     dX dX 1 ,d2 1 ,d2 σ A:ij  = σ  (A:ij + Mij ) . (18) i,j=1

Let a =

i,j=1

Pd1 ,d2

i,j=1 Mij . We can equivalently write the above statement as:



dX 1 ,d2

 A:ij + a1K  .

(19)

Applying Proposition 3.1, we have:     dX dX 1 ,d2 1 ,d2 A:ij  = σ  A:ij  = f˜(X). σ

(20)

= σ

i,j=1

Proving the desired statement. Proposition 3.6 (Uniqueness of ContrastiveCAMs). Let A and A be two HiResCAMs corresponding to probability predictions f˜(X) ∈ RK such that: Ak = Ak + M,

∀k ∈ [K],

(21)

with M ∈ Rd1 ×d2 being an arbitrary matrix. Then, the ContrastiveCAMs C and C, derived from A and A respectively, are the same: C = C,

and

C ag = C .

(22)

Proof. First, consider k-ContrastiveCAMs. We have that: C k = {C (k,k′ ) : k ′ ∈ [K]}

(23) ′

= {Ak + M − Ak′ − M : k ∈ [K]}

(24)

′

(25)

= {Ak − Ak′ : k ∈ [K]} ′

= {C(k,k′ ) : k ∈ [K]} = Ck , 14

∀k ∈ [K].

(26)

Similarly, in the case of aggregated k-ContrastiveCAMs, we have: K

C k = Ak −

1 X Ak′ K ′

(27)

k =1

= Ak + M −

1 X (Ak′ + M ) K ′

(28)

k =1

= Ak + M −

K ·M 1 X − Ak′ K K ′

(29)

k =1

= Ak −

1 X Ak′ = Ckag , K ′

∀k ∈ [K].

(30)

k =1

Eqs. (26) and (30) together prove the desired equalities. Proposition 3.7(Correctness of ContrastiveCAM Explanations). Given an input X and a class index k ∈ [K], the prediction f˜k (X) is determined by the k-ContrastiveCAMs: −1   dX K 1 ,d2 X f˜k (X) =  exp − C(k,k′ ) ij  , ∀k ∈ [K]. (31) k′ =1

i,j=1

Moreover, if for some k ∈ [K], i ∈ [d1 ], j ∈ [d2 ], C(k,k′ ) ij = 0, ∀k ′ ∈ [K], then the (i, j)-patch does not influence the final prediction. Proof. For a class index k ∈ [K], the prediction probability given a logit vector f is expressed as: efk f˜k = σk (f ) = PK

k′ =1 e

fk′

(32)

We define the logit vector in terms of the elementwise difference to a candidate class k: d := f − fk =⇒ f = fk + d.

(33)

Based on this definition, class probabilities can equivalently be computed as: efk f˜k = PK

= PK

fk′ k′ =1 e

efk

fk +dk′ k′ =1 e

efk P K f

dk′ k′ =1 e

= PK

dk′ k′ =1 e

(34)

This re-contextualizes softmax as a direct function of the differences between class logits. We can further deconstruct the difference by logit values: dk′ = fk′ − fk =

dX 1 ,d2

Ak′ ij −

i,j=1

dX 1 ,d2

Akij ,

∀k, k ′ ∈ [K].

(35)

i,j=1

Applying Definition 3.3, we have: dk′ =

dX 1 ,d2

(Ak′ − Ak )ij = −

i,j=1

C(k,k′ ) ij ,

∀k, k ′ ∈ [K].

(36)

i,j=1

Substituting dk′ from Eq. (36) into Eq. (34), we have: 1 P d1 ,d2 C(k,k′ ) ij − i,j=1   −1 dX K 1 ,d2 X = exp − C(k,k′ ) ij  ,

f˜k (X) = P

(37)

K k′ =1 exp

k′ =1

i,j=1

∀k ∈ [K].

(38)

We can thus compute class probabilities as a direct function of ContrastiveCAMs. Next, we show that zero-patches do not influence predictions. For a given class k ∈ [K], denote the set of zero-patches as Zk , and its complement as Zk′ : n o Zk = (i, j) ∈ [d1 ] × [d2 ] : C(k,k′ ) ij = 0, ∀k ′ ∈ [K] , and Zk′ = ([d1 ] × [d2 ]) \ Zk . (39) Applying this partition to Eq. (38), we have that:  −1  dX K 1 ,d2 X f˜k (X) =  exp − C(k,k′ ) ij  k′ =1

 =

K X

−1



=

K X

=

K X

k′ =1

C(k,k′ ) ij −

i,j∈Zk′

C(k,k′ ) ij 

(41)

i,j∈Zk

−1

 X

exp −

C(k,k′ ) ij −

i,j∈Zk′

k′ =1



exp −

k′ =1



(40)

i,j=1

0

(42)

i,j∈Zk

−1

 X

exp −

C(k,k′ ) ij 

(43)

i,j∈Zk′

Note that, for some k ∈ [K], C(k,k′ ) ij = 0 for all k ′ ∈ [K] is equivalent to (Ak )ij = (Ak′ )ij for all k ′ ∈ [K], which further implies that C(k,k′ ) ij = 0 for all k, k ′ ∈ [K]. This means that Zk = Zk′ for all k, k ′ ∈ [K]. Therefore, Eq. (43) holds for all k ∈ [K], implying all the logit outputs are not affected by the patch (i, j). Remark 3.4. For binary classification tasks where there is a single output logit and sigmoid applied to obtain the final predictions, HiResCAMs are intrinsically contrastive. Proof. For K = 2, i.e., binary classification tasks, one can obtain class probabilities using two setups: (i) f ∈ R2 with softmax σ activation, or (ii): f ′ ∈ R with sigmoid ϕ activation. Consider ContrastiveCAMs C ∈ R2×d1 ×d2 for softmax activation, and the corresponding HiResCAMs A ∈ Rd1 ×d2 for sigmoid activation. We will prove that HiResCAMs in the sigmoid setup are equivalent to ContrastiveCAMs in the softmax setup. For the softmax case, we apply Proposition 3.7 and obtain class probabilities as:     −1 dX dX 1 ,d2 1 ,d2 pf (y = 1|X) = f˜1 (X) = exp − C(1,1) ij  + exp − C(1,2) ij  . i,j=1

C(k,k) = 0d1 ×d2 , ∀k ∈ [K], hence we have:   pf (y = 1|X) = 1 + exp −

(44)

i,j=1

dX 1 ,d2

−1

C(1,2) ij 

(45)

i,j=1



= 1 + exp 

dX 1 ,d2

−1

C(2,1) ij 

(46)

i,j=1

and similarly, 



pf (y = 2|X) = 1 + exp −

dX 1 ,d2 i,j=1

−1

C(2,1) ij 

(47)

For the sigmoid case, we apply Proposition B.1 and obtain class probabilities as:   −1 dX 1 ,d2 pf ′ (y = 1|X) = 1 − ϕ(f ′ ) = ϕ(−f ′ ) = 1 + exp  Aij  ,

(48)

i,j=1

and 



pf ′ (y = 2|X) = ϕ(f ′ ) = 1 + exp −

dX 1 ,d2

−1 Aij 

(49)

i,j=1

Comparing Eq. (46) with Eq. (48) and Eq. (47) with Eq. (49), we observe that HiResCAMs’ connection to the model prediction is identical to that of ContrastiveCAMs, hence HiResCAMs in the binary setting are intrinsically contrastive. Proposition B.1 (Correctness of HiResCAMs in Binary Setting). Given an input X, the prediction f˜(X) of a binary classifier is determined by the corresponding HiResCAM:   −1 dX 1 ,d2 f˜(X) = 1 + exp − Aij  . (50) i,j=1

Proof. In the binary classification setting, f (X) ∈ R, and f˜ = ϕ(f ), where ϕ denotes sigmoid activation. Applying Eq. (2), we have:   −1 dX 1 ,d2 f˜(X) = ϕ(f ) = (1 + exp(−f ))−1 = 1 + exp − Aij  . (51) i,j=1

Context-Regularized Binary Cross-Entropy

In the binary classification setting, the neural network outputs a single logit, f (X) ∈ R, to which the sigmoid function, denoted ϕ : R → [0, 1], is applied to produce the corresponding probability predictions: f˜ = ϕ(f ). The corresponding class labels are denoted y ∈ {0, 1}. Additionally, HiResCAMs have shape A ∈ Rd1 ×d2 , and their connection to logits in Eq. (2) extends to class probabilities. This follows from the bijectivity of sigmoid activation: Proposition B.1 (Correctness of HiResCAMs in Binary Setting). Given an input X, the prediction f˜(X) of a binary classifier is determined by the corresponding HiResCAM:   −1 dX 1 ,d2 f˜(X) = 1 + exp − Aij  . (52) i,j=1

We leverage this property to apply context regularization in the binary setting: Definition B.2 (Context-Regularized Binary Cross-Entropy Loss). Given a training dataset D and convolutional network f , CR-BCE is defined as: LCRBCE (D) = E(X,y,H)∼D [ℓ(X, y, H)], with    dX 1 ,d2    (53) ℓ(X, y, H) = log 1 + exp  (−1)y Aij Hij + (1 − Hij )|Aij | . {z } | i,j=1 context regularization

This definition is motivated by noting that the per-sample binary cross-entropy loss ℓBCE can be equivalently expressed as: ℓBCE (X, y) = −y log(f˜) − (1 − y) log(1 − f˜) = log(1 + exp((−1)y f )), (54) when sigmoid activation is used to obtain class probabilities from logits. 17

Table 3: Spawrious Context-Robustness Benchmark. CR-CE outperforms the standard crossentropy training, as well as GALS and ABN, across all difficulty levels and context shift settings of the Spawrious dataset. Spurious Shift Type

Task Difficulty

CAM Size

Approach

Average (↑)

Easy (↑)

Medium (↑)

Hard (↑)

7×7

CE (Baseline) GALS EBL CFA Context Masking CR-CE (Ours)

93.30±0.03 93.42±0.18 93.31±0.69 90.86±0.54 92.36±0.70 95.47±0.24

83.17±0.47 84.39±2.13 83.91±1.46 79.46±0.48 80.97±1.59 90.51±0.16

86.58±1.88 87.11±0.27 86.44±1.45 81.77±1.73 83.67±3.71 91.35±0.34

87.69 88.31 87.89 84.03 85.66 92.44

14 × 14

CE (Baseline) ABN GALS EBL CFA Context Masking CR-CE (Ours)

92.83±0.01 88.01±1.49 94.72±1.32 94.20±0.28 88.98±0.14 93.07±0.40 96.35±0.15

80.36±1.46 80.47±0.61 85.74±3.96 82.20±3.33 73.38±0.87 81.79±2.29 92.24±0.30

87.74±0.37 80.04±2.13 90.57±0.65 87.59±1.54 75.88±1.12 83.91±1.95 92.89±0.20

86.98 82.84 90.35 88.00 79.41 86.26 93.83

7×7

CE (Baseline) GALS EBL CFA Context Masking CR-CE (Ours)

90.86±0.54 93.47±0.36 92.43±1.31 89.14±0.89 92.08±1.04 95.55±0.16

79.46±0.48 84.58±0.88 87.23±0.84 85.68±0.68 85.03±2.37 91.25±0.18

81.52±1.03 78.74±0.97 81.21±0.74 81.77±1.73 78.46±1.12 87.49±0.14

83.95 85.60 86.96 85.65 85.19 91.43

14 × 14

CE (Baseline) ABN GALS EBL CFA Context Masking CR-CE (Ours)

88.98±0.14 89.65±0.44 95.03±1.35 93.46±0.73 87.22±0.78 90.96±0.24 96.47±0.06

73.38±0.87 72.81±1.32 87.63±3.80 89.05±0.47 85.68±0.68 85.70±2.99 92.41±0.29

79.76±1.01 63.96±6.54 84.08±1.87 79.36±2.83 75.88±1.12 80.80±0.71 88.66±0.52

80.71 75.47 88.91 87.29 82.93 85.82 92.51

One-to-One

Many-to-Many

Table 4: Hard-ImageNet Benchmark. Predictions on models trained with CR-CE are least likely to persist when the object is absent from the image. Object-focus increases proportional to the CAM resolution, with some performance cost incurred when the target is present alongside the context. CAM Size

7×7

14 × 14

Accuracy under Target Removal

Rel. Foreground Sensitivity

Approach

No Shift Accuracy (↑)

Gray Mask (↓)

Gray BBOX (↓)

Tile (↓)

ρ = 0.25 (↑)

ρ = 0.5 (↑)

CE (Baseline) GALS EBL CFA CR-CE (Ours)

95.82±0.08 95.91±0.15 95.87±0.23 96.00±0.13 91.42±0.28

82.4±0.13 80.93±0.27 77.64±1.66 82.13±0.35 50.09±0.89

79.38±0.2 76.62±1.21 73.20±2.07 79.24±0.50 45.78±0.41

77.11±0.08 75.69±0.73 70.74±1.67 76.67±0.35 46.98±0.2

−0.21±0.01 −0.18±0.01 −0.11±0.04 −0.20±0.01 +0.10±0.02

−0.36±0.01 −0.31±0.03 −0.19±0.06 −0.35±0.01 +0.16±0.01

CE (Baseline) ABN GALS EBL CFA CR-CE (Ours)

95.82±0.08 95.24±0.54 95.82±0.15 95.24±0.15 96.00±0.13 88.53±0.00

81.2±0.61 80.84±0.15 78.00±3.54 70.31±4.46 81.42±0.50 37.07±0.48

78.36±0.28 73.47±1.62 74.27±4.35 67.16±3.45 78.22±0.56 32.44±0.15

77.64±0.2 73.33±0.23 73.82±3.78 65.91±5.33 76.76±0.73 32.76±0.78

−0.22±0.01 −0.34±0.02 −0.12±0.08 −0.04±0.03 −0.18±0.02 +0.17±0.01

−0.34±0.003 −0.44±0.01 −0.23±0.14 −0.05±0.14 −0.36±0.06 +0.23±0.01

Additional Experimental Results

C.1

Spawrious Benchmark on Various Context-Shift Configurations

The Spawrious dataset comprises of four dog breeds spread across six contexts, with three difficulty levels based on the object-context combinations available during training, and two different configurations of context shift: (i) one-to-one shifts, where each class is correlated with one context, and (ii) many-to-many shifts, where groups of classes are correlated with groups of contexts, the latter being observed broadly in practice [23]. We include results on both one-to-one and many-to-many shifts in Table 3. Consistent with our findings in Section 5.3, we observe that CR-CE trained models 18

Table 5: Context-Contribution Ratios. We report the context contribution ratios across each of the assessed approaches. CR-CE trained models place a greater emphasis on object-specific regions to determine predictions and learn to ignore cues from context regions through the process of training. Context Contribution Ratio R% (↓) CAM Approach Size Waterbirds-100 Spawrious‡ ImageNet-BG Hard-ImageNet

7×7

CE (Baseline) GALS EBL CFA Context Masking CR-CE (Ours)

46.08±0.43 36.2±0.92 56.02±5.17 17.06±0.16 22.48±1.72 12.86±0.09

44.79±0.3 28.06±1.39 21.37±1.45 43.1±2.69 48.11±2.22 14.89±0.2

31.83† 29.75±0.5 25.35±2.06 32.49±0.07 32.86±0.02 14.85±0.00

68.53±0.04 63.64±2.03 57.24±3.91 57.22±0.50 70.63±0.06 35.85±0.26

14 × 14

CE (Baseline) ABN GALS EBL CFA Context Masking CR-CE (Ours)

58.75±0.12 34.92±19.38 35.7±2.42 48.21±2.44 16.08±0.88 25.24±1.27 9.54±0.28

47.32±0.09 49.89±16.99 17.77±2.07 29.82±1.06 46.66±0.58 38.44±1.91 10.98±0.12

28.27±0.27 33.37±1.01 24.14±0.13 22.94±0.83 28.42±0.02 27.64±0.03 9.32±0.06

66.5±0.07 69.06±4.02 56.87±9.37 48.63±9.68 47.89±1.56 69.35±0.2 31.12±0.1

† ‡

No standard deviation is reported; we use pre-trained weights from torchvision without modification. We report metrics on the hardest subtask and difficulty (M2M-hard) within the Spawrious benchmark.

outperform the baseline and existing explanation-guided approaches in context-shifted accuracy across all difficulty levels. C.2

Evaluating Context Dependence using Hard-ImageNet

The Hard-ImageNet benchmark [25] assesses model reliance on context by comparing performance on images with and without the target object. Ideally, the model should predict the true class in the presence of the object, and a random class in the absence of the object. In other words, the prediction should not be based on the context alone. The Hard-ImageNet suite also reports Relative Foreground Sensitivity (RFS) [24], which compares accuracy reduction when gaussian noise (with magnitude denoted ρ) is added to either the object or the context region; higher RFS signifies greater object-dependence. Evaluation of CR-CE on the Hard-ImageNet and comparison with other methods are reported in Table 4. We observe that the CR-CE framework meaningfully outperforms each of the compared methods on key robustness metrics assessed by the Hard-ImageNet benchmark. Example images of Hard-ImageNet demonstrating the effectiveness of CR-CE on suppressing context dependence can be find in Figure 3 and Figure 5, see the classes Baseball Player, Snorkel, and Howler Monkey. Remark C.1. Hard-ImageNet includes a lot of context-heavy images where object regions comprise only 13.96% of the image on average, leading to a bigger reduction of accuracy in the accuracyrobustness tradeoff. C.3

Comparison with Explanation-Guided Learning (EGL) Methods

Typically, explanation-guided learning frameworks [14] construct a multi-task optimization objective with a per-sample loss of the following form: ℓEGL = ℓCE + λ · ℓEG ,

with gradient

∇ℓEGL = ∇ℓCE + λ∇ℓEG .

(55)

Here, ℓEG is a secondary explanation-guided goal designed to suppress context contributions. The specific form of this term varies between approaches [38, 34, 20, 29, 11]. For a given (i, j)-patch corresponding to context, gradients of the form in Eq. (55) create a tug-of-war dynamic: the first term seeks to minimize ℓCE at the cost of ℓEG , and vice versa. To manage this, the additional hyperparameter λ is introduced and must be carefully tuned to prevent poor classification accuracy. In contrast, CR-CE replaces the mechanism in cross-entropy that increases contributions 19

Original Image

True Class: Dachshund Pred. Class: Labrador

❌

Attention Maps

HiResCAM

Original Image

True Class: Bulldog

ContrastiveCAM

Pred. Class: Dachshund

Attention Maps

❌

HiResCAM

ContrastiveCAM

Figure 4: Superficially Aligned Attribution Maps. On a ResNet-50 trained with the GALS framework, we observe that HiResCAMs suggest a focus on the object regions; however, the model predicts the context-correlated class observed during training. ContrastiveCAMs faithfully reveal that the context contribution is in fact the primary driver of the prediction. This highlights the essential role that explanation faithfulness plays in ensuring improvements to context robustness in the explanation-guided learning paradigm.

from context regions, preventing competing gradients (taken w.r.t. the specific patch), hence resulting in a more pronounced object-focus effect. The advantage of this is evident in the Waterbirds-100 dataset. Since it is a binary classification task, the GradCAMs used by GALS coincide with ContrastiveCAMs used by CR-CE. The only difference between the two approaches lies in how the feature attribution method is incorporated in the model training procedure. In this direct comparison, CR-CE continues to outperform GALS, see Table 2.

Explanation Correctness is Key to Robustness Improvements. In Table 2, we observe that some prior explanation-guided methods may even have an inferior performance than the CE training baselines, especially when the tasks become complicated. To investigate this performance drift, we measure the context-contribution ratios for those approaches in Table 5. We observe that models trained by those explanation-guided methods often have greater context dependence, indicating the ineffectiveness of these methods in guiding the feature attributions towards object regions. We hypothesize that this behavior is largely due to their reliance on classical feature attribution method (e.g. Grad-CAM and HiResCAM), which suffer from the unfaithfulness issue theoretically described in Theorem 3.2. Figure 4 shows typical examples where prior explanation-guided methods may be successful in guiding those classical CAMs to focus on the object regions but fail in doing so for the more faithful ContrastiveCAMs.

C.4

CR-CE is Tolerant to Imperfect Object-Context Segmentation

In our experiments with Spawrious and ImageNet datasets in Table 2, we already used AI-predicted object-context segmentation, obtained from SAM3 [5] and MAEs [18], which are approximations and are inevitably imperfect. Here, we further assess CR-CE’s tolerance to noise by adding additional noise to these object-context masks. Specifically, we perform random affine transformations (up to 5◦ of rotation, ±7.5% translation, and ±30% scaling) on each of those masks H. This augmentation results in ≈ 50% mIoU of the original object-context masks to their corrupted counterparts. We train CR-CE using these corrupted masks, and report our results in Table 6. We can see that, even with these noisy object-context segmentation, CR-CE can still predict well with minor degradation in the performance.

C.5

CR-CE is Effective Across Architectures and Scales

Here we provide additional experimental validations of the CR-CE training framework on different network architectures, including EfficientNetV2 [42] and ResNet-18. The results are reported in Table 7 and 8 respectively, demonstrate the effectiveness of CR-CE across variations in architectures and scales. 20

Table 6: Context-Shift Robustness using Noisy Masks (NM). Despite the additional noise introduced to object-context masks, CR-CE remains effective at improving context-shift robustness. CR-CE retains its utility in scenarios where accurate masks are challenging or expensive to obtain. Waterbirds-100

CAM Size

Approach

ImageNet/ImageNet-BG

Spawrious

No Shift (↑)

Shifted (↑)

No Shift (↑)

Shifted (↑)

Easy (↑)

Medium (↑)

Hard (↑)

7×7

CE (Baseline) CR-CE (Ours) + NM CR-CE (Ours)

98.66±0.10 92.10±0.22 94.38±1.21

41.74±4.15 72.61±1.26 73.30±4.43

82.92† 81.14±0.02 81.85±0.03

74.83† 74.89±0.07 75.75±0.08

90.86±0.54 93.07±0.32 95.55±0.16

79.46±0.48 87.75±0.42 91.25±0.18

81.52±1.03 82.19±0.16 87.49±0.14

14 × 14

CE (Baseline) CR-CE (Ours) + NM CR-CE (Ours)

98.48±0.09 90.67±2.42 94.36±0.12

38.40±1.79 76.12±3.24 79.13±0.77

82.33±0.10 79.83±0.06 81.25±0.03

74.72±0.06 75.24±0.02 76.84±0.12

88.98±0.14 93.07±0.51 96.47±0.06

73.38±0.87 85.64±0.17 92.41±0.29

79.76±1.01 80.59±0.84 88.66±0.52

†

No std. dev. is reported; we use pre-trained weights from torchvision without modification.

Table 7: Context-Shift Robustness On EfficientNetV2. CR-CE improves context-shift robustness on varied convolutional backbones. Following EfficientNetV2’s [42] training recipe, we adopt a larger batch size of 4096 and increase the learning rate to 10−2 for ImageNet training. Waterbirds-100

CAM Size

Approach

ImageNet/ImageNet-BG

Spawrious

No Shift (↑)

Shifted (↑)

No Shift (↑)

Shifted (↑)

Easy (↑)

Medium (↑)

Hard (↑)

7×7

CE (Baseline) CR-CE (Ours)

91.08±0.05 94.31±1.93

39.31±3.01 78.55±5.01

83.71† 83.36±0.16

73.58† 74.46±0.08

92.62±0.41 93.86±0.62

81.49±0.29 84.23±0.67

77.50±0.91 88.29±0.22

14 × 14

CE (Baseline) CR-CE (Ours)

84.64±0.14 94.02±1.15

38.88±1.31 80.43±5.01

84.00±0.07 83.67±0.03

74.53±0.07 75.20±0.04

93.11±0.42 94.84±0.33

80.27±2.06 86.98±0.04

79.30±1.17 88.85±0.55

†

No std. dev. is reported; we use pre-trained weights from torchvision without modification.

About Data Augmentation with Object-Context Mixing

A straightforward yet often effective approach to improving robustness to context shifts is to augment the training dataset with synthetic images that combine objects with diverse contexts [12]. This approach is orthogonal to the methods considered in this paper, including explanation-guided methods, as it leaves the training procedure unchanged and simply introduces additional augmented training samples. Therefore, it can be readily combined with our proposed method. However, its effectiveness may be limited by the finite set of object–context combinations observed during training, potentially leaving the model vulnerable to unseen context shifts at evaluation time, particularly adversarial ones. In contrast, our proposed training framework suppresses the influence of context regions, not limited by the artificial object–context combinations seen during training, potentially providing greater robustness to out-of-distribution scenarios. Nevertheless, we empirically evaluate this approach by augmenting each training image with an additional image in which the object is paired with a context uniformly sampled from the nontarget classes, thereby doubling the size of the training dataset. We report the effect of pairing this augmentation with both CE and CR-CE in Table 9. We can see that the data augmentation technique performs well in simple tasks, such as Waterbirds, but becomes worse when the task becomes harder.

Training Details

Hyperparameter Settings. For all methods, we use Momentum SGD [32] with a momentum parameter 0.9. The hyperparameter settings shown in Table 10 are applied across all methods. GALS [29], ABN [11] and Energy loss [34] each have an additional hyperparameter λ which weights the regularization strength of spatial feature attributions, which we keep at their default value of 1. Computational Resources. We train each of our models using a single NVIDIA H100 GPU. The most efficient loss function, cross-entropy, took a maximum of 5.5 hours to train ResNet-50 on ImageNet, and a minimum of 20 minutes on the Waterbirds dataset. Training with CR-CE took a maximum of 6.1 hours per ResNet-50 run on the ImageNet dataset, and a minimum of 20 minutes on the Waterbirds dataset. Each of the other methods approximately took between 8 and 12 hours to train a ResNet-50 on ImageNet. Each of the methods used a maximum of 65 GB of VRAM during training on the ImageNet dataset, with gradient accumulation applied as necessary to stay within the VRAM limit of the GPU. We trained a total of 582 models: 3 independent runs for each approach 21

Table 8: Context-Shift Robustness On ResNet-18. CR-CE improves context-shift robustness on the smaller ResNet-18 backbone. We observe a more pronounced trade-off between in-distribution and context-shifted accuracy due to the smaller size of the backbone. In contrast, this trade-off is minimized by the larger capacity of the EfficientNetV2 backbone, see Table 7. Waterbirds-100

CAM Size

Approach

ImageNet/ImageNet-BG

Spawrious

No Shift (↑)

Shifted (↑)

No Shift (↑)

Shifted (↑)

Easy (↑)

Medium (↑)

Hard (↑)

7×7

CE (Baseline) CR-CE (Ours)

79.96±12.03 88.48±0.09

38.00±3.79 53.55±10.44

73.49† 72.40±0.05

58.38† 62.61±0.02

80.89±0.92 90.77±0.07

59.34±0.73 75.93±0.56

52.16±1.27 68.99±0.29

14 × 14

CE (Baseline) CR-CE (Ours)

97.59±0.36 88.25±0.22

30.41±8.90 59.95±7.97

75.02±0.10 71.44±0.06

60.43±0.06 63.37±0.04

74.84±2.08 91.64±0.26

46.14±2.45 79.42±0.23

37.81±1.14 70.68±1.44

†

No std. dev. is reported; we use pre-trained weights from torchvision without modification.

Table 9: Context-Shift Robustness using Object-Context Augmentation. Generating objectcontext combinations using copy-paste augmentation only improves robustness in the Waterbirds setting, when object-context augmentation simulates the test set. In larger and more realistic contextshift scenarios, augmenting contexts does not produce enough signal to suppress context contributions. Waterbirds-100

CAM Size

Approach

ImageNet/ImageNet-BG

Spawrious

No Shift (↑)

Shifted (↑)

No Shift (↑)

Shifted (↑)

Easy (↑)

Medium (↑)

Hard (↑)

7×7

CE (Baseline) CE + OCM CR-CE (Ours) CR-CE (Ours) + OCM

98.66±0.10 96.01±0.11 94.38±1.21 95.35±1.02

41.74±4.15 90.64±0.81 73.30±4.43 92.63±0.97

82.92† 80.38±0.13 81.85±0.03 79.52±0.03

74.83† 72.80±0.04 75.75±0.08 73.79±0.01

90.86±0.54 79.65±0.39 95.55±0.16 94.40±0.09

79.46±0.48 49.15±2.01 91.25±0.18 87.95±0.61

81.52±1.03 42.99±1.01 87.49±0.14 76.96±0.60

14 × 14

CE (Baseline) CE + OCM CR-CE (Ours) CR-CE (Ours) + OCM

98.48±0.09 95.51±0.22 94.36±0.12 94.88±0.29

38.40±1.79 92.73±1.06 79.13±0.77 93.36±0.88

82.33±0.10 81.08±0.14 81.25±0.03 78.85±0.07

74.72±0.06 73.70±0.05 76.84±0.12 74.75±0.03

88.98±0.14 70.69±2.47 96.47±0.06 95.30±0.24

73.38±0.87 40.84±0.74 92.41±0.29 88.55±0.36

79.76±1.01 32.17±5.06 88.66±0.52 77.51±0.41

†

No std. dev. is reported; we use pre-trained weights from torchvision without modification.

Table 10: Hyperparameters are dataset-specific and kept consistent across approaches. HardImageNet and ImageNet-1k are trained for fewer epochs since we fine-tune ImageNet-trained models. We increase the batch size as number of classes (2, 4, 15, and 1000 respectively) increases. Dataset

Epochs

Batch Size

Waterbirds-100 Spawrious Hard-ImageNet ImageNet-1k / ImageNet-BG

150 150 100 20

64 128 256 512

10−3 10−3 5 · 10−3 5 · 10−4

Momentum

Weight Decay

Scheduler

0.9

10−4 10−4 10−4 10−5

Cosine Annealing

and backbone on 9 different training sets (6 for Spawrious, 1 each for Waterbirds, ImageNet-BG, and Hard-ImageNet), with 2 CAM resolutions for all but ABNs. ABN increases the CAM resolution as part of its architecture design; we do not reduce it to 7 × 7. Reproducibility. The source code for this project can be accessed at https://codeberg.org/ jinensetpal/context-regularized-cross-entropy/. Asset Licenses.

We used the following existing assets, listed with their respective licenses:

• The Waterbirds dataset [37] is licensed under MIT. • The Spawrious dataset [23] is licensed under CC0-1.0. • The ImageNet dataset [8] is licensed only for non-commercial purposes. • The ImageNet-BG dataset [30] is published under a public domain license. • The Hard-ImageNet dataset [25] is published under CC-0. • Model weights for the Masked AutoEncoder (MAE) [18] trained on the Large-Scale Unsupervised Segmentation (LUSS) [13] dataset is licensed under CC BY-NC 4.0. • Model weights for ABNs [11] is licensed under MIT. • SAM3 [5] is published under a custom SAM License which permits the academic use of their tool. • The torchvision library is published under a BSD 3-Clause License.

Attention Maps (ContrastiveCAMs)

Original Image

Class: Bobsled

Cross-Entropy (Baseline)

CR-CE (Ours)

Context Util.: 67.4%

Context Util.: 15.3%

Confidence: 92.8%

Confidence: 86.4%

Class: Snorkel

Class: Gondola

CR-CE (Ours)

Context Util.: 68.3%

Context Util.: 7.9%

Confidence: 99.9%

Class: Howler Monkey

Attention Maps (ContrastiveCAMs) CR-CE (Ours)

Confidence: 99.9%

Cross-Entropy (Baseline)

CR-CE (Ours)

Context Util.: 74.2%

Context Util.: 4.0%

Confidence: 99.9%

Attention Maps (ContrastiveCAMs)

Context Util.: 56.7%

Context Util.: 7.0%

Confidence: 68.4%

Confidence: 91.2%

Class: Airship

Attention Maps (ContrastiveCAMs)

Cross-Entropy (Baseline)

CR-CE (Ours)

Context Util.: 64.3%

Context Util.: 19.3%

Confidence: 98.7%

Confidence: 93.7%

Attention Maps (ContrastiveCAMs) Original Image

Cross-Entropy (Baseline)

CR-CE (Ours)

Context Util.: 66.7%

Context Util.: 11.6%

Confidence: 99.9%

Class: Microwave

Attention Maps (ContrastiveCAMs)

Cross-Entropy (Baseline)

CR-CE (Ours)

Context Util.: 62.4%

Context Util.: 12.4%

Confidence: 98.4%

Confidence: 99.3%

Attention Maps (ContrastiveCAMs)

Original Image

Class: Guitar

Context Util.: 2.0%

Confidence: 99.9%

Original Image Cross-Entropy (Baseline)

Original Image

Class: Space Shuttle

Context Util.: 85.2%

Original Image

Class: Anemone Fish

CR-CE (Ours)

Attention Maps (ContrastiveCAMs)

Attention Maps (ContrastiveCAMs) Original Image Cross-Entropy (Baseline)

Cross-Entropy (Baseline)

Original Image Cross-Entropy (Baseline)

CR-CE (Ours)

Context Util.: 50.5%

Context Util.: 5.7%

Confidence: 96.8%

Confidence: 94.2%

Class: Gondola

Cross-Entropy (Baseline)

CR-CE (Ours)

Context Util.: 55.7%

Context Util.: 8.9%

Confidence: 99.9%

Figure 5: ImageNet-BG and Hard-ImageNet. Extended qualitative evaluations of object-context dependence by cross-entropy and CR-CE trained models. Images belonging to the Snorkel and Howler Monkey classes correspond to the Hard-ImageNet validation set, whereas the rest correspond to the ImageNet-BG test set. Across each of the examples, we observe the context dependence in models trained using cross-entropy loss is significantly mitigated by training with CR-CE.

Improving Context-Shift Robustness of Convolutional Networks via Context-Regularized Cross-Entropy

Analyze & explore related research in Conceptio

Related documents