Shortcut Learning in Geospatial Foundation Model Embeddings for Cross-Disaster Damage Assessment

arXiv:2610.13766v1 [cs.CV] 9 Oct 2026

Shortcut Learning in Geospatial Foundation Model Embeddings for Cross-Disaster Damage Assessment

Ali Shibli, Andrea Nascetti, Yifang Ban Division of Geoinformatics KTH Royal Institute of Technology Stockholm, Sweden {shibli,nascetti,yifang}@kth.se

Abstract Geospatial foundation models can support disaster damage assessment, but high accuracy on familiar disasters may hide shortcut learning: prediction from event or scene characteristics rather than evidence that transfers to a new disaster. We study embeddings from TerraMind, Prithvi-EO-2.0, DOFA, and DINOv3 on the multidisaster datasets xView2 and BRIGHT. Holding out an entire disaster, instead of randomly splitting image tiles, reduces mean balanced accuracy from 0.807 to 0.638 on xView2 and from 0.818 to 0.568 on BRIGHT. Four sets of tests examine what information supports prediction: event and acquisition context, before/after image inputs, building count, and correct image pairing. Pre-event images remain predictive, and a classifier given only the annotated number of buildings reaches balanced accuracy of 0.629 and 0.653. Combining before/after features performs best on average on xView2, but does not improve over pre-only features on BRIGHT. Shuffling image pairs within the same disaster and damage class causes modest losses. Motivated by these findings, we test whether feature transformations learned only from training disasters improve generalization to unseen disasters. These transformations improve performance, with supervised contrastive learning performing best on xView2 and PCA on BRIGHT. Our results highlight the importance of evaluation on unseen domains for assessing model robustness and show that simple featurelevel transformations can improve generalization. Code and experiments can be found at https://github.com/alishibli97/gfm-shortcut-learning.

1

Introduction

Geospatial foundation models (GFMs) provide reusable Earth-observation embeddings when downstream labels are scarce. For disaster response, however, a useful representation must transfer to an event that was not present during training. A random image-level split does not test this: tiles from the same hurricane, earthquake, or wildfire can occur on both sides of the split, sharing geography, sensor properties, building density, and the proportion of tiles labeled as damaged. This setting encourages shortcut learning: using an easy correlation that predicts benchmark labels but is not reliable evidence of damage under the intended deployment shift [Geirhos et al., 2020]. A model may recognize a known event from geography, hazard type, sensor characteristics, land cover, or urban form, then associate it with that event’s fraction of damaged tiles. It may also use either image separately without comparing the same location before and after the disaster. Such context can be legitimate; the central question is whether the resulting prediction transfers to an unseen event. We study embeddings from TerraMind, Prithvi-EO-2.0, DOFA, and DINOv3 on the multi-disaster datasets xView2/xBD [Gupta et al., 2019] and BRIGHT [Chen et al., 2025]. We compare a random 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: 2nd Workshop on Advances in Representation Learning for Earth Observation.

image-level split, denoted IID (independent and identically distributed) because both sets contain the same events, with leave-one-event-out (LOEO), where the test disaster is unseen. Our audit separates four questions: whether prediction uses (i) event prevalence or acquisition metadata, (ii) one temporal view, (iii) annotation-derived building count, or (iv) the correct same-location pre/post match. We then test whether source-only PCA, a supervised MLP, supervised contrastive learning (SupCon) [Khosla et al., 2020], and cross-event SupCon improve unseen-event prediction. No held-out-event sample is used for training or feature transformation. Our contributions are: • an event/context audit showing that random splits overestimate transfer and that event prevalence, acquisition metadata, and building count predict damage; • a complete temporal-view and pairing audit showing dataset-dependent use of pre-, post-, and change information, with only modest losses after correspondence is broken; and • source-only controls showing that event–class reweighting has little effect, while compression and supervised representation learning improve prediction on held-out disasters.

2

Related Work

Robust EO representations Prithvi-EO-2.0, TerraMind, and DOFA provide reusable EO representations with different pre-training and sensor designs [Szwarcman et al., 2026, Jakubik et al., 2025, Xiong et al., 2024]; DINOv3 provides a large self-supervised vision baseline [Siméoni et al., 2025]. Robustness benchmarks nevertheless show sensitivity to real distribution shifts [Doerksen and Kerner, 2026, Li et al., 2025]. We focus on a concrete operational shift: training on known disasters and testing on a different disaster. Shortcut learning and cross-disaster transfer Shortcuts are predictive but potentially unstable signals [Geirhos et al., 2020, Ye et al., 2024]. xBD established paired optical damage assessment, while BRIGHT adds globally distributed optical–SAR pairs [Gupta et al., 2019, Chen et al., 2025]. Cross-disaster adaptation has also been studied directly [Mouradi and Kshirsagar, 2026]. We complement this work with controlled tests of available context and image-pair information. Our transfer baselines use supervised and cross-domain contrastive learning [Khosla et al., 2020, Wang et al., 2021]; our contribution is the evaluation and audit, not a new training loss.

3

Shortcut Audit and Source-Only Mitigation

Task and representation For backbone ϕm , we independently encode the pre- and post-event images and form zi = [zipre ; zipost ; zipost − zipre ], zit = ϕm (xti ). (1) The binary label yi is one when the tile contains at least one damaged building; ei is its disaster event. A class-balanced linear probe (linear classifier) predicts yi . We compare a random image-level split (IID), where the same events occur in training and test sets, with LOEO. In LOEO, all samples from one event are held out, and scaling, transformation, and probe fitting use only the remaining source events. Metrics are averaged over events so large disasters do not dominate. What shortcuts are available? Figure 1 summarizes our four diagnostic families and source-only transfer evaluation. (1) Event/context. Can event or acquisition information predict damage without images? The event-prior assigns each test tile its event’s proportion of damaged training tiles. It is IIDonly because unseen events have no training labels. A separate metadata probe receives hazard type, sensor, source, and ground sampling distance (pixel size), excluding event identity. (2) Temporal view. Which before/after inputs help? Separate probes receive pre-only, post-only, difference-only (z post − z pre ), or complete zi features. Pre-only prediction observes no disaster impact; post-only and difference features may contain direct damage evidence. Differences are between embeddings, not pixels. (3) Building count. Can building count predict the tile label? An image-free probe receives only log(1 + ni ), where ni is the annotated building count for that tile. More buildings create more opportunities for a positive label because yi = 1 when any building is damaged. No segmentation is performed; counts are not image-model inputs. (4) Pair correspondence. How much does location 2

SHORTCUT AUDITS: what information supports prediction? 1 Event / metadata

2 Before / after

3 Building count

4 Location match

Event label frequency or acquisition metadata

Pre, post, difference or full-pair features

One annotated count per tile: log(1 + n)

Shuffle pre or post within event and class

Image-free controls

Image-feature tests

Image-free control

Image-feature test

TRANSFER: can training-only methods help on a new disaster? Source events

Reweight / transform + fit classifier

Test on held-out event

Figure 1: Four questions guide the audit. Image-free controls use metadata or annotations; image tests use fixed-model features. Transfer methods are fitted on source events only.

matching matter? At LOEO test time, we shuffle pre- or post-event features within the same event and label, then rebuild zi . The probe remains trained on correct source pairs. This breaks location matching without reversing time; we average both shuffle directions. Target labels construct this diagnostic only, never training or normal prediction. Source-only controls and mitigation We first train a classifier on the original embeddings with sample weights that give equal importance to each event–label group, preventing larger events or more frequent labels from dominating training. We then compare four transformations fitted on source events: PCA to 256 dimensions; a two-layer MLP trained with binary cross-entropy (CE-MLP); ordinary SupCon, which clusters samples with the same damage label; and cross-event SupCon, which permits same-label positives only from different source events. Both contrastive variants use a 256-D normalized projection. Their purpose is to improve prediction on the held-out event without target-event samples. Comparing ordinary and cross-event SupCon tests whether restricting positive pairs to different source events adds a transfer benefit.

4

Experiments

Data and implementation Our xView2 subset contains 9,168 optical pre/post pairs from 19 events; 35.3% of tiles contain at least one building labeled minor damage, major damage, or destroyed. BRIGHT contains 3,029 pairs from 11 events and five hazard groups; 26.6% contain a damaged or destroyed building. Other tiles are negative. BRIGHT pairs pre-event RGB optical imagery with postevent SAR, repeated across three channels here. This is a compatibility test, not a sensor-optimized SAR pipeline. Images are resized to 224 × 224. TerraMind, DOFA, and DINOv3 ViT-L/16 (LVD-1689M) use model-specific RGB normalization. Prithvi-EO-2.0-300M uses display RGB range-matched to HLS RGB statistics; this does not reconstruct physical reflectance, so its results describe an out-of-domain input setting. We concatenate mean-pooled features from four backbone depths: 4,096 dimensions per image for Prithvi and DINOv3, and 3,072 for TerraMind and DOFA. Backbone weights stay fixed. Learned transformations use a 512-D hidden layer, 256-D output, AdamW, 30 epochs, and three seeds. We report balanced accuracy (BA: mean recall across the two classes) and area under the ROC curve (AUC). Learned-method results average seeds before backbones. Event and context results Table 1(a) shows why random-split performance is an optimistic estimate of unseen-disaster performance. Full-pair BA drops from 0.807 to 0.638 on xView2 and from 0.818 to 0.568 on BRIGHT under LOEO. Event-prior AUC reaches 0.821 and 0.715 without images. Metadata is also predictive within familiar events, but its LOEO BA is approximately chance (0.498 and 0.500). These baselines reveal available context signals; they do not establish that the image probe uses them. Image inputs and pairing On xView2, post-only BA exceeds pre-only (0.617 versus 0.577), and the full pair performs best at 0.638 (Table 1b). On BRIGHT, pre-only has the highest mean BA (0.571), narrowly above the full pair (0.568). Combined inputs therefore help on xView2 but not on average in our BRIGHT pipeline, highlighting the challenge of exploiting optical–SAR pairs. 3

Table 1: Audit means over four backbones: (a) event/context, (b) image inputs, and (c) count/pairing controls. Full pair combines pre, post, and their difference. Shuffled pair averages both shuffle directions. Event prior is random-split only. (a) Event and acquisition context Full pair Dataset

Event prior

Metadata

Random BA LOEO BA Random AUC Random BA LOEO BA

xView2 BRIGHT

.807 .818

.638 .568

.821 .715

.725 .600

(b) Before/after image inputs (LOEO BA / AUC) Dataset Pre only Post only Difference only xView2 .577 / .611 .617 / .665 BRIGHT .571 / .612 .553 / .587

.498 .500

Full pair

.587 / .626 .638 / .699 .561 / .606 .568 / .606

(c) Building count and location matching (LOEO BA) Dataset Building count Full pair Shuffled pair xView2 BRIGHT

.629 .653

.638 .568

.617 .552

Table 2: Source-only LOEO transfer (BA / AUC). Scores average events and backbones, after seeds for learned transformations. Bold marks the highest mean; rankings are descriptive. Method Raw features Event–class weighting PCA-256 Supervised MLP SupCon Cross-event SupCon

xView2 BA / AUC

BRIGHT BA / AUC

.638 / .699 .638 / .700 .654 / .727 .658 / .714 .664 / .730 .661 / .727

.568 / .606 .570 / .616 .594 / .650 .582 / .615 .588 / .643 .585 / .642

Building count alone reaches BA 0.629 and 0.653 (Table 1c). Like pre-only success, this reveals predictive scene information, not necessarily harmful shortcut use. Shuffling pairs reduces BA from 0.638 to 0.617 on xView2 and from 0.568 to 0.552 on BRIGHT. Correct matching helps modestly when event and class are preserved. Transfer improvements We next test source-only controls and transformations. Event–class weighting barely changes BA (Table 2). Feature transformations give larger mean gains: SupCon performs best on xView2 (BA 0.664, AUC 0.730), and PCA on BRIGHT (0.594, 0.650). Cross-event SupCon has slightly lower mean scores than ordinary SupCon on both datasets, giving no observed benefit from restricting positive pairs to different events.

5

Conclusion

High accuracy on familiar disasters does not guarantee reliable prediction on a new disaster. Our experiments show that random splits hide shortcuts in GFM embeddings. Event statistics, scene information, and building count can predict tile-level damage, while correct pre/post matching provides a modest benefit. Feature transformations improve unseen-event performance, although cross-event SupCon does not outperform ordinary SupCon or PCA. Damage models should therefore use held-out-event evaluation and image-free, temporal-view, and correspondence-breaking controls. Future work should evaluate building-level severity and native sensor inputs.

6

Acknolwedgement

Computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), funded by the Swedish Research Council. 4

References Hongruixuan Chen, Jian Song, Olivier Dietrich, Clifford Broni-Bediako, Weihao Xuan, Junjue Wang, Xinlei Shao, Yimin Wei, Junshi Xia, Cuiling Lan, Konrad Schindler, and Naoto Yokoya. Bright: A globally distributed multimodal building damage assessment dataset with very-high-resolution for all-weather disaster response. Earth System Science Data, 17(11):6217–6253, 2025. doi: 10.5194/essd-17-6217-2025. Kelsey Doerksen and Hannah Kerner. Earthshift: A benchmark for measuring robustness to real-world distribution shifts in earth observation. arXiv preprint arXiv:2605.29330, 2026. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2:665–673, 2020. Ritwik Gupta, Richard Hosfelt, Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, and Matthew Gaston. xbd: A dataset for assessing building damage from satellite imagery. arXiv preprint arXiv:1911.09296, 2019. Johannes Jakubik, Felix Yang, Benedikt Blumenstiel, Erik Scheurer, Rocco Sedona, Stefano Maurogiovanni, Jente Bosmans, Nikolaos Dionelis, Valerio Marsocci, Niklas Kopp, Rahul Ramachandran, Paolo Fraccaro, Thomas Brunschwiler, Gabriele Cavallaro, Juan Bernabe-Moreno, and Nicolas Longépé. Terramind: Large-scale generative multimodality for earth observation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661–18673, 2020. Xiang Li, Yong Tao, Siyuan Zhang, Siwei Liu, Zhitong Xiong, et al. Reobench: Benchmarking robustness of earth observation foundation models. arXiv preprint arXiv:2505.16793, 2025. Asmae Mouradi and Shruti Kshirsagar. Robust building damage detection in cross-disaster settings using domain adaptation. arXiv preprint arXiv:2603.14694, 2026. Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, et al. Dinov3. arXiv preprint arXiv:2508.10104, 2025. Daniela Szwarcman, Sujit Roy, Paolo Fraccaro, Thorsteinn Eli Gislason, et al. Prithvi-eo-2.0: A versatile multi-temporal foundation model for earth observation applications. IEEE Transactions on Geoscience and Remote Sensing, 64:1–20, 2026. doi: 10.1109/TGRS.2025.3642610. Rui Wang, Zuxuan Wu, Zejia Weng, Jingjing Chen, Guo-Jun Qi, and Yu-Gang Jiang. Cross-domain contrastive learning for unsupervised domain adaptation. arXiv preprint arXiv:2106.05528, 2021. Zhitong Xiong, Yi Wang, Fahong Zhang, Adam J. Stewart, Joëlle Hanna, Damian Borth, Ioannis Papoutsis, Bertrand Le Saux, Gustau Camps-Valls, and Xiao Xiang Zhu. Neural plasticity-inspired multimodal foundation model for earth observation. arXiv preprint arXiv:2403.15356, 2024. Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, Xia Hu, and Aidong Zhang. Spurious correlations in machine learning: A survey. arXiv preprint arXiv:2402.12715, 2024.

5

Related documents

• Shortcut Learning in Geospatial Foundation Model Embeddings for Cross-Disaster Damage Assessment#1298262
• Physically aligned forest fire risk prediction: A deep learning framework coupling fuel and climate multivariate factors.#636167
• Physically aligned forest fire risk prediction: A deep learning framework coupling fuel and climate multivariate factors#631440
• FL-MHSM: Spatially-adaptive Fusion and Ensemble Learning for Flood-Landslide Multi-Hazard Susceptibility Mapping at Regional Scale#31255
• No One Knows the State of the Art in Geospatial Foundation Models#999791