Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Marcelo Sartori Locatelli 1 2 Fernando Tonucci 2 Jea Kwon 1 Luiz Felipe Vecchietti 1 Bryan Nathanael Wijaya 1 Cheng Yaw Low 3 Virgilio Almeida 2 Meeyoung Cha 1 4

arXiv:2606.07172v1 [cs.CV] 5 Jun 2026

Abstract

foundation models across multiple data modalities and application domains.

Geospatial understanding is a critical yet underexplored dimension in the development of machine learning systems for tasks such as image geolocation and spatial reasoning. In this work, we analyze the geospatial representations acquired by three model families: vision-only architectures (e.g., ViT), vision-language models (e.g., CLIP), and large-scale multimodal foundation models (e.g., LLaVA, Qwen, and Gemma). By evaluating across image clusters, including people, landmarks, and everyday objects, grouped based on the degree of localizability, we reveal systematic gaps in spatial accuracy and show that textual supervision enhances the learning of geospatial representations. Our findings suggest the role of language as an effective complementary modality for encoding spatial context and multimodal learning as a key direction for advancing geospatial AI.

Recent advances, such as CLIP (Radford et al., 2021), include multimodal models that integrate text and vision to learn joint representations within a shared latent space. Another line of research focuses on vision-language models (VLMs), which integrate text and image inputs through a two-stage training pipeline: an initial phase using paired text-image data, followed by instruction tuning (Liu et al., 2024; Bai et al., 2025; Kamath et al., 2025). These models typically employ a frozen vision encoder alongside a language model, enabling multimodal understanding and generation. Vision models increasingly demonstrate the ability to internalize diverse meta information around the world, raising the question of whether they also encode geolocation—even without explicit geospatial supervision. Their internal representations are shaped by architecture, pretraining, and fine-tuning, yet remain difficult to interpret (Ghiasi et al., 2022). This challenge is further amplified in emerging VLMs, where multimodal complexity obscures the underlying mechanisms by which knowledge is encoded. Such opacity can lead to unintended outcomes, including geographic disparities that reflect uneven generalization across regions (Moayeri et al., 2024). To improve fairness and transparency, we examine the capacity of vision-only and vision-language models to encode implicit geospatial information (Figure 1) by asking the following question: To what extent do these models internalize global location knowledge during their training and fine-tuning pipelines?

1. Introduction Vision models have undergone tremendous progress in the last decade, driven by advances in convolutional neural network (CNN) architectures (Simonyan & Zisserman, 2014; He et al., 2016) and Vision Transformers (ViT) (Dosovitskiy et al., 2021). These models are capable of capturing high-level, transferable representations that can be utilized in zero-shot scenarios via their embeddings and be adapted through fine-tuning for specific downstream tasks. Specifically, ViTs benefit from the scalability of Transformers (Vaswani et al., 2017) and enable the development of

Here, we use the term geospatial representations to refer to latent features contained in the inner layers of ViTs or VLMs that encode information relevant to downstream geospatial tasks, i.e., related to physical location on Earth. Learning geospatial representations via supervised training has already been explored by Vivanco Cepeda et al. (2023). Following on their work, we are interested in investigating the kinds of geospatial features that are learned during training without additional supervision. For text-based large language models (LLMs), Gurnee & Tegmark (2024) and Godey et al. (2024) have shown that specific neurons and

Max Planck Institute for Security and Privacy (MPI-SP), Bochum, Germany 2 Universidade Federal de Minas Gerais, Belo Horizonte, Brazil 3 Changwon National University, Changwon, South Korea 4 Korea Advanced Institute of Science and Technology, Daejeon, South Korea. Correspondence to: Meeyoung Cha <[email protected]>, Virgilio Almeida <[email protected]>. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

layers within LLMs implicitly encode latitude and longitude information and that this capacity scales with increasing model size.

This approach enables the model to learn invariant representations across multiple augmented views of the same image, resulting in linearly separable features that implicitly capture semantic structures such as object boundaries and regions. Building upon this foundation, DINOv2 (Oquab et al., 2024) extends this methodology by incorporating the IBOT loss (Zhou et al., 2021), a patch-level objective. This integration facilitates scalable pretraining, enhancing the model’s capacity to learn visual representations from large-scale unlabeled datasets.

Embeddings from pretrained LLMs can also be used to create geospatial embeddings from geolocation-related prompts, as shown in LLMGeovec (He et al., 2025), where its representations improve performance on various downstream tasks requiring geospatial understanding. Additionally, Roberts et al. (2024) has shown that VLMs have spatial reasoning capabilities, being able to complete a variety of tasks through zero-shot settings. We extend these results by focusing on how ViT models separate images spatially in their learned latent space and exploring how these models learn geospatial representations.

2.2. Vision-Language Models A turning point for vision models has been the integration of language for learning shared representations. Pioneering work like CLIP jointly trained a vision encoder and a language encoder to align their representations using a large corpus of web-scraped image-caption pairs (Radford et al., 2021). Subsequent research, such as SigLIP (Zhai et al., 2023), expanded CLIP by replacing softmax loss with a sigmoid objective, decoupling performance from batch size, and allowing for improved scalability. These models are fundamental and are applied as the core vision-encoder for many of the vision-language foundation models presented in our work.

Our main contributions are as follows. • We investigate geospatial representations learned by vision-only architectures, vision-language models, and large-scale multimodal foundation models, finding that the latter two groups exhibit substantially stronger geospatial structure. • We evaluate the performance of different model representations using layer-wise probing for geospatial location prediction. We find that vision-only models tend to exhibit stronger representations on their last layer, while VLMs have better geospatial representations on the early layers of their language model block.

Foundation VLMs utilize specialized training pipelines. For example, LLaVA-1.5 (Liu et al., 2024) employs a twostage training process, first aligning a frozen CLIP visual encoder with an LLM on image-text pairs, and then finetuning the model on a GPT-generated instruction-following dataset to enhance its conversational and reasoning abilities. Qwen2.5 (Bai et al., 2025) follows a similar process, pretraining on image-text pairs and then performing additional supervised fine-tuning (SFT) and direct preference optimization (DPO) to structure instruction-following data. Gemma 3 (Kamath et al., 2025) leverages a frozen SigLIP vision encoder, a pretraining stage similar to previous models, and a post-training stage that includes knowledge distillation from a larger instruction-tuned model and alignment with human feedback via SFT and reinforcement learning with human feedback (RLHF).

• We show that prompting VLMs allows the geospatial information to be propagated to the latter layers, in some cases leading to an increase in representation quality.

2. Related Works 2.1. Vision-Based Models ViTs emerged as a paradigm shift in computer vision, adapting Transformers (Vaswani et al., 2017) to image data by segmenting images into tokenized patches with positional embeddings (Dosovitskiy et al., 2021). The initial ViT model demonstrated that large-scale supervised pretraining on image classification tasks could yield transferable representations across domains. Researchers have also explored selfsupervised approaches to vision. One such method is the Masked Autoencoder (MAE) (He et al., 2022), which trains models to reconstruct randomly masked image patches. This objective encourages the extraction of semantically rich features that can be effectively adapted to downstream tasks via fine-tuning.

2.3. Geospatial Representations in Vision Models Many previous works have approached geo-localization as a supervised learning task. PlaNet (Weyand et al., 2016) uses a CNN architecture to classify images into adaptively generated geographical cells, finding that such an approach leads to a big improvement over regression or retrieval-based approaches. More recently, PIGEON (Haas et al., 2024) combines CLIP features with a hierarchical geo-localization and haversine smoothing to achieve strong performances. GeoCLIP (Vivanco Cepeda et al., 2023) takes a complementary approach, aligning image features from CLIP to a

As an alternative to self-supervised pretraining, (Caron et al., 2021) proposed a self-distillation framework named DINO. 2

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

input image

A(2)

A(3)

A(4)

Linear Probing A(l)

Ridge Regression

W = arg min ||Y – A(l)W||F + 𝜆||W||F W

⊕

Layer L

⊕

Layer 5

⊕

Layer 4

⊕

predictied lat/lon

A(1)

Layer 3

⊕

Layer 2

Layer 1

vision-only/vision-language model “The “Theimage imagedepicts depictsthe the “The image of depicts the Step Pyramid Djoser, imageof depicts the Step“The Pyramid Djoser, Step Pyramid of Djoser, located ininSaqqara.” Step Pyramid of Djoser, located Saqqara.” located in Saqqara.” located in Saqqara.”

⊕

A(L–1)

A(L)

model output

. .. . . . . . . ... . .. .... . . . . .. . .. .. .. . 2 R = 0.75 . true lat/lon

Figure 1. Schematic illustration of our linear probing analysis setup. We fit ridge regression models W on the [CLS] or last token residuals A(l) from each layer l to investigate whether geospatial information (i.e., latitude and longitude) can be extracted from these hidden dimensions based on the R2 value.

location encoder. Their results demonstrate that high-quality geospatial representations transfer beyond geo-localization, reinforcing early intuition on the importance of such features for image understanding tasks (Hays & Efros, 2008).

We provide the details below. Yahoo Flickr Creative Commons 100 Million (YFCC100M) (Thomee et al., 2016). We used a 4M-image subset from the MediaEval 2016 Placing Task competition (Choi et al., 2016), obtained via Kaggle1 . This dataset is a diverse collection of Flickr-sourced images spanning natural scenes, urban environments, and everyday objects with location data. To analyze localizability across semantic categories, we partitioned this subset via unsupervised clustering. First, we extracted image embeddings with ResNet-152 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009), then we applied principal component analysis (PCA) to retain the top-100 components explaining the highest variance and performed k-means clustering (Han et al., 2011). Among 19 tested k values (k = 10, 15, . . . , 100), we selected k = 40 using the elbow method (Thorndike, 1953), with manual inspection confirming semantic coherence. The resulting clusters captured meaningful categories, including people, objects, cliffs, landscapes, and buildings. Complete clustering details are provided in Appendix B.

These works establish that vision models can be trained to geolocate and learn useful features for other tasks in the process. Our goal is different: rather than training a geolocalization model, we are interested in which geospatial structure was implicitly learned by existing vision foundation models, which architectural choices might have influenced its emergence, and what the implications are for downstream tasks.

3. Methods 3.1. Models To examine how geo-localization capabilities emerge in vision models without explicit supervision, we curated a diverse set of architectures spanning multiple modalities and training paradigms. Our selection includes both (i) vision-only encoders, i.e., ViT (Dosovitskiy et al., 2021), ViT Masked Autoencoder (He et al., 2022), DINOv2 (Oquab et al., 2024), and Web-SSL models (Fan et al., 2025), and (ii) vision–language models, i.e., CLIP (Radford et al., 2021), MetaCLIP (Xu et al., 2024), LLaVA-1.5 (Liu et al., 2024), Qwen2.5 (Bai et al., 2025), and Gemma 3 (Kamath et al., 2025), representing supervised and self-supervised approaches. For each model family, we evaluated at least two size variants to assess the influence of scale on learned geospatial representations. Additional information about these models is given in Appendix A.

Google Landmarks (Weyand et al., 2020). This dataset contains over 5M photographs of globally recognized landmarks and tourist sites (e.g., the Eiffel Tower, Mount Fuji) that are available in public data sources. We hypothesize that such iconic scenes may have been encountered during model pretraining, contributing to their high localizability. It also contains non-localizable images, such as particularly close-up shots of people or animals, and generic textures such as soil, which lack distinctive geospatial cues. In our experiments, we use a subset of 580k images2 , hereafter 1 https://www.kaggle.com/datasets/habedi/ large-dataset-of-geotagged-images 2 https://huggingface.co/datasets/ visheratin/google_landmarks_places

3.2. Dataset To build our dataset, we sampled images from established benchmarks, including YFCC100M and Google Landmarks. 3

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

referred to as the Landmarks dataset, with geolocation coordinates extracted from OpenStreetMap.

targets (latitude and longitude), are estimated as: 2

Ŵ = arg min Y − A(l) W F + λ W F .

3.2.1. S AMPLING

Across all datasets, the geospatial distribution of images was imbalanced, skewed toward major cities in Europe and North America. To mitigate this bias, we partitioned the globe into non-overlapping geocells based on global administrative area boundaries and iteratively merged regions with insufficient samples. We merged geocells hierarchically. First within regions (GID 1), then across regions of the same country (GID 0). To avoid oversampling, each geocell was defined to include at least one complete GID 2 (citylevel) administrative unit, even in dense areas like Paris. These geocells were then used to balance the YFCC100M and Landmarks dataset by selecting 5,000 images per source, with at most five images per geocell. For the Landmarks dataset, we also excluded duplicate images of the same landmark to ensure diversity.

Here, ∥ · ∥F denotes the Frobenius norm, and λ > 0 is the regularization hyperparameter controlling the strength of the L2 penalty on W. In all of our experiments, λ is chosen for each probe using Leave-One-Out cross-validation (Golub et al., 1979).

4. Experiments 4.1. Performance on Geolocation Information Prediction We perform layer-wise probes on all the evaluated models and report the geolocation prediction performance in Figure 2. We find that models trained jointly on text and images exhibit measurable geospatial representations: the R2 values reach up to 0.8 for Landmarks and streets (cluster 28), while the average R2 is above 0.4 for the larger models, suggesting some degree of geospatial representations across image types. The average R2 for vision-only models, on the other hand, is mainly below 0.3. Among vision-only models, performance improves with model size, with DINOv2-giant (1B) and the Web-SSL DINO-7B model achieving the best result. This observation suggests that, when trained on a scale, geospatial representation can be learned from images alone. When compared with VLMs, DINOv2-giant is outperformed, on average, even by the much smaller CLIP-base model, suggesting the effectiveness of language pretraining in implicitly learning geospatial representations. This is strengthened by the fact that Web-SSL DINO-7B is outperformed by MetaCLIP-huge (600M) despite the much larger scale and being trained on the same dataset.

3.3. Probing To examine whether the evaluated models encode geospatial information, we perform linear probing (Alain & Bengio, 2017), a standard mechanistic interpretability technique for Transformer architectures (Gurnee & Tegmark, 2024; Kim et al., 2025). Transformers (Vaswani et al., 2017) consist of sequential blocks that iteratively refine token representations within the residual stream x(l) ∈ Rt×dmodel , where dmodel is the hidden dimension of each evaluated model, as an input with t tokens is propagated through the l-th Transformer block (Elhage et al., 2021). This refinement is achieved through successive multi-head attention (MHA) and multilayer perceptron (MLP) layers with residual connections. These layers are often paired with normalization, of which the formulation is omitted for brevity: (l) hattn = x(l) + MHA x(l) (1) (l) (l) hmlp = MLP hattn (l)

(l)

x(l+1) = hattn + hmlp

(4)

The cluster-wise performance presented in Figure 2 indicates that the relative difficulty of each cluster in YFCC100M remains consistent across various model architectures. For instance, the streets cluster, building cluster, and the Landmarks dataset consistently show higher R2 across models, while the objects cluster contains little information that can serve as clues for geo-localization. Interestingly, the signs and text clusters show a degree of localizability for VLMs not observed for vision-only models.

(2) (3)

Although t varies across vision models, downstream tasks typically rely on a single summary representation. Usually, the [CLS] token is used in vision models and the final token representation in VLMs. We fit ridge regression models to predict latitude and longitude (in degrees) from layer-wise token summary representations. We report the models’ predictive performance using the coefficient of determination (R2 ). Formally, given hidden representations A(l) ∈ Rn×dmodel from layer l, number of samples n, and hidden dimension dmodel , the regression weights W ∈ Rdmodel ×2 , corresponding to the two-dimensional

Figure 3 shows image samples positioned according to their R2 values for the largest model of each model family. Highly localizable images tend to be famous landmarks, e.g., pyramids, and open spaces with pieces of architecture or nature. Meanwhile, close-ups of objects and food images are the least localizable. Notice that for CLIP, the cluster of figures containing signs achieves a notable degree of localizability. 4

Textual Supervision Enhances Geospatial Representations in Vision-Language Models 0.8

vit-mae-base vit-mae-large vit-base-patch16-224 vit-large-patch16-224 dinov2-base dinov2-large dinov2-giant webssl-dino7b-full8b-224 clip-vit-base-patch32 Qwen2.5-VL-3B-Instruct gemma-3-4b-it Qwen2.5-VL-7B-Instruct gemma-3-12b-it gemma-3-4b-pt gemma-3-12b-pt clip-vit-large-patch14 metaclip-h14-fullcc2.5b llava-1.5-7b-hf

0.7 0.6 0.5 0.4

R²

Vision-only Vision-language

0.3 0.2

Objects Uncategorized Glowing Lights Musicians Flowers Drinks Soil Food Birds and Insects Animals People Closeup Displays Vehicles Interiors Text Colorful Outdoors Children Poles and Columns Religious Monuments Statues Horizon Wall People 2 Banks/ Beaches People 1 Cold Places Large Gatherings Roads and Railways Cars Sports People Outdoors Lakes Nature Piers Cliffs and Mountains Signs Old Buildings Constructions Buildings Landmarks Streets

0.1

Figure 2. Model performance measured by coefficient of determination R2 across all models. The x-axis shows image clusters based on the YFCC100M dataset and the Google Landmarks dataset, and the y-axis lists the models compared. Higher R2 values (darker colors in the heatmap) indicate better geolocation-prediction accuracy across clusters.

reflects the model’s tendency to deprioritize geographic signals in the absence of a textual prompt, thus neglecting spatial information not essential for text generation. A similar, though less pronounced, effect is observed in LLaVA, where geolocation signals diminish as the image transitions into the language modeling component.

vit-mae-large vit-large dinov2-giant Qwen2.5-VL-7B gemma-3-12b-pt clip-vit-large

4.3. Steering the Models via Text Prompts

llava-1.5-7b Unlocalizable (R² 0)

In prior experiments, we observed that the geospatial information in the model residuals, particularly VLMs, degrades over layers without a textual prompt. This leads to, in some extreme cases, for example, Gemma, a negative R2 , suggesting that there is no linear mapping from the model residuals to geolocation coordinates.

Very Localizable (R² = 1)

Figure 3. Illustration of image cluster localizability for each model, where each image serves as a representative of a cluster (Appendix B). The images to the right represent clusters with better localizability (higher R2 ). For high-performing models, highly localizable landmarks usually achieve the highest performance.

To check if this effect occurs because of the absence of a textual prompt related to geospatial information, we prompt the models with the query “Guess the latitude and longitude of this image. Answer only with the coordinate tuple (lat, long)”. Figure 4(b) shows the per-layer R2 of VLMs when prompted this way. We observe that, in this setting, R2 does not decrease as drastically for Gemma and LLaVA. In fact, for LLaVA, it starts increasing over the textual layers. Interestingly, for Qwen, the performance drastically increases, leading to R2 as high as 0.88 for the Landmarks dataset. This effect suggests that textual and image representations of geospatial representation might be entangled

4.2. Geospatial Representations Across Layers To investigate where geospatial representation emerges within model architectures, we analyze probe performance across layers for both vision-only and multimodal models. In vision models such as ViT and DINOv2, geospatial representations tend to develop progressively with increasing layer depth, as evidenced by a consistent rise in probe R2 values across all settings (see Figure 4(a)). For VLMs, however, an interesting observation emerges: the R2 values increase only up to a certain point before stagnating, and in the case of Gemma, the R2 values decrease throughout the later layers, regardless of image characteristics. This likely 5

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

0.8

0.6

0.4

0.2

0.0

Text layers

Qwen2.5-VL-7B-Instruct 1.0 0.8

R²

0.6

1.0

0.2

gemma-3-4b-pt

0.8 0.6

0.2

0.0

(a)

Layer

Qwen2.5-VL-7B-Instruct

0.8 0.6

Layer

gemma-3-4b-pt

1.0

Text layers

1.0

0.2

0.8 0.6

0.2

0.0

Layer

0.2

Layer

0.2

Layer

landmarks signs objects food streets cliffs

llava-1.5-7b-hf

0.6 0.4

0.8

0.4

1.0

0.4

0.2

0.6 0.4

llava-1.5-7b-hf

0.8

0.4

1.0

0.4

0.2

R²

0.2

Text layers

R²

0.8

vit-mae-large

1.0

0.8

0.2

(b)

clip-vit-large-patch14

1.0

landmarks signs objects food streets cliffs

Text layers

dinov2-giant

1.0

Figure 4. Probe R2 performance by layer of the models for different clusters and datasets with varying levels of localizability. (a) R2 performance when no textual prompt is given. (b) R2 performance when adding a textual prompt to the input, asking the model to predict the image geolocation. The R2 is kept stable throughout the last layers when compared to the decaying performance observed in the non-prompting setup.

We report the predictive performance in terms of R2 in Figure 5 as a function of the retained feature proportion p, showing that R2 increases with p and saturates well before using the entire feature set. Across all models, we find that p ≈ 0.4 (about 40% of dimensions) are sufficient to recover nearly the maximum R2 , indicating that geospatial information is concentrated in a compact subset of dimensions rather than uniformly distributed throughout the embeddings. For Qwen2.5-VL variants, specifically, 90% of their best predictive performance is still observed using only the top 10% of the features.

in the model’s latent space, especially when activated using textual prompts related to geospatial tasks. We expand on this discussion in Appendix G. 4.4. Isolating Geospatial-Specific Components from Embeddings The linear probes operate on high-dimensional representations. In our experiments, the five selected models have dmodel of 768 for CLIP-ViT-large, 1,024 for LLaVA-1.5, 2,048 for Qwen2.5-VL-3B, 3,584 for Qwen2.5-VL-7B, and 1,536 for DINOv2-giant. To explore how latent space contributes to geospatial information, we fit ridge regression probes using only a proportion of the original dimensions p ∈ {0.1, 0.2, . . . , 1.0}, selecting the top p dimensions ranked by the absolute coefficients of the trained probe. 6

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

We show an example in which we can successfully steer the model in Figure 6. By swapping 50% of geospatial representations from an image of the Step Pyramid of Djoser with the geospatial representation from an image of the Trevi Fountain, the model generates the following output text: “The image depicts the Step Pyramid, Rome, Italy”, altering the location of the pyramid from Saqqara to Rome. Even though our experiments show the possibility of successfully steering the model, we observed that as the text generated becomes longer, the generation becomes unstable. In some cases, the model starts to generate repetitive text or descriptions that mix the source and target locations. These results open future avenues for investigating how geospatial representations are coupled with representations related to other types of information during text generation. We discuss more details, including quantitative results, in Appendix H.

1.00

Proportion of R²

0.95 0.90 0.85 0.80

dinov2-giant clip-vit-large-patch14 Qwen2.5-VL-7B-Instruct Qwen2.5-VL-3B-Instruct llava-1.5-7b-hf

0.75 0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Retained dimension proportion p

0.9

1.0

Figure 5. Probe predictive performance R2 as a function of the retained feature proportion p, illustrating the capacity of the embedding subspace needed to reach the maximum R2 for both visiononly and VLMs. Higher values of p correspond to a larger subset of the latent representation.

4.6. Downstream Task Performance Finally, we inspect how the quality of geospatial representations may influence downstream task performance, as these models are usually fine-tuned for specific downstream applications. For this analysis, we investigate a task that requires geospatial awareness: country identification.

4.5. Steering the Model Generation Through Representation Swapping We examine the possibility of steering the text generated by a multimodal foundation model by swapping the top p feature dimensions related to geospatial reasoning. In this case study, the geospatial location in the predicted text should be changed, leaving other semantic information unchanged. We use Qwen2.5-VL-3B, an open-weight VLM that supports a targeted intervention in its residual stream or the additive hidden state passed through the transformer layers. As illustrated in Figure 6, given a source image and a target image, we replace the top geospatially relevant feature dimensions of the source residual-stream summary with the corresponding coordinates from the target, and evaluate the resulting changes in the generated text.

Table 1. Fine-tuning performance for the country identification task.

Model ViT-MAE-large ViT-large DINOv2-large DINOv2-giant CLIP-large

Let t⋆ be the last non-padding input token (the summary to(1) (1) ken for Qwen2.5-VL-3B), and let Asource, t⋆ , Atarget, t⋆ ∈ Rdmodel be the layer-1 residual-stream vectors for source and target images, respectively. Given g ⊆ {1, . . . , dmodel } as the index set of geospatially informative dimensions (with proportion p = |g|/dmodel ) and its complement g c = {1, . . . , dmodel } \ g, we intervene by replacing the source residual with the target residual on the dimensions in g as follows: Ãsource,t⋆ = Asource,t⋆ ⊙ 1gC + Atarget,t⋆ ⊙ 1g , (1)

(1)

Test Acc.

Val. Loss

Train Loss

0.15 0.23 0.29 0.32 0.36

3.35 3.17 2.55 2.78 2.39

2.344 1.346 0.009 0.001 0.009

Using the landmarks dataset, we extract country information for each picture and then subsample the dataset so that at most 100 pictures are selected for each country. Then, we fine-tune one large model from each studied visiononly family (ViT-MAE, ViT, and DINOv2) in addition to CLIP-ViT-large and DINOv2-giant (for the full details, see Appendix I). The models are chosen such that all take the same inputs and have similar model size, making the results comparable. We report the results of each model in Table 1. We observe that the performance of the models follows the order of R2 obtained for our probe analysis, with ViTMAE having the worst performance, while CLIP has the best performance. This corroborates the hypothesis that the presence of geospatial representations in the models is desirable for their use in downstream tasks.

(5)

where ⊙ denotes the element-wise Hadamard product and 1g an indicator vector with entries 1 for indices in g and 0 elsewhere. For brevity, we omit explicit indexing of the selected dimensions g. We then continue the forward pass (1) from layer 2, . . . , L using Ãsource to decode the output text. 7

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

⊕

Layer L

LLM backbone Layer 2

Layer 1

⊕

MLP

⊕

visual projector Layer n

⊕

Layer 2

Layer 1

vision encoder

“The image depicts the Step Pyramid of Djoser, located in Saqqara.”

Layer L

⊕

“The image depicts the Step Pyramid, Rome, Italy.”

Layer L

Layer 2

Layer 2 Layer 1

MLP

Layer n

Layer 2

⊕

Layer 1

⊕

visual projector

⊕

vision encoder

⊕

Feed to the rest of the VLM

Swap the dimensions correlated with the geospatial representation

Layer 3

Extract the last token residuals from the first text layer

source image

“The image depicts the Trevi Fountain in Rome, Italy.”

LLM backbone

target image

Figure 6. Schematic illustration showing that editing the geospatial representations (through dimension swapping) changes the perceived geolocation during token generation of VLMs. We demonstrated this finding on Qwen2.5-VL-3B with the methodology described in Section 4.5.

5. Discussion

in VLMs remains more stable across the post-textual layers (Figure 4(b)).

Our findings demonstrate that the training methodology is crucial for learning geospatial representations in vision-only models and VLMs, as demonstrated in Figure 2. Models that incorporate textual supervision consistently achieve the best performance. Vision-only models improve with scale; larger models like DINOv2-giant outperform both DINOv2-large and DINOv2-base. In contrast, VLMs do not exhibit a clear correlation between model scale and probing performance. This suggests that supervision signals, particularly language, are a primary factor in learning strong geospatial representations in these models. In this way, relating our findings to the Platonic Representation Hypothesis (Huh et al., 2024), even though scaling vision-only models improves performance, textual supervision enhances the efficiency for learning geospatial representations.

The results of our analysis have direct implications for model selection and methodology in a range of geospatial applications. For current pipelines that use traditional vision-only models and small datasets, leveraging representations from VLMs can significantly improve performance. As these representations are implicitly learned from vast datasets, they serve as strong representations for sampleefficient pipelines involving fine-tuning cases where labeled data are scarce. Moreover, our work highlights multimodal learning as a critical direction to build world models, which could be used to improve our understanding of complex social problems and empower new technologies. Finally, the growing capability of these models poses significant privacy risks and fairness implications. These models have spatial imbalance on geolocation performance, with lower performance for underrepresented regions (Appendix C). Additionally, malicious actors could exploit these models to extract precise location data from images, enabling stalking and threats against individuals. The potential for mass surveillance is also a serious concern, where governments and corporations could likewise track individuals’ locations and behaviors through their photos. These ethical risks underscore the need for robust regulatory policies that mandate transparency in model use and enforce explicit user

The optimal layer for extracting geospatial representations depends on the model family and the presence of a textual prompt. In many geo-localization applications, the input image is given without a textual prompt (Figure 4(a)). For these cases, vision-only models perform best when using representations from their deepest layers. However, for VLMs, choosing which geospatial representation to use is not clear and varies between models. Across models, the layers immediately after the textual stream consistently achieve good performance. With a textual prompt, the performance 8

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

consent to ensure safe deployment.

the performance. In Appendix L, we conduct additional experiments after filtering images with captions related to coordinates or countries, finding that performance is generally unchanged, suggesting that memorization is not the only reason for the differences in performance.

6. Conclusion This study demonstrated that textual supervision significantly enhances geospatial representations in visionlanguage models. Through a systematic analysis of visiononly architectures, VLMs, and large-scale multimodal foundation models, we studied how geospatial understanding emerges across model families. Through layer-wise probing, we revealed that multimodal models consistently exhibit high performance for images that are localizable. Furthermore, our analysis indicated that a small subset of hidden dimensions is responsible for encoding critical geospatial features, suggesting a potential pathway for model steering and editing. In summary, our work demonstrated that multimodal learning plays an important role in improving geospatial AI. However, using these models in real-world settings should include safeguards to protect privacy and ensure fairness. As applications involving location-aware image understanding—such as environmental monitoring, urban planning, and disaster response—continue to grow, this use case is expected to become increasingly important. Future research could explore how these models handle other types of images, such as satellite data.

Reproducibility Statement The code for reproducing our results is available through a GitHub repository for validation3 , and all datasets used in this study are publicly accessible, but need to be downloaded separately.

Impact Statement As discussed in Section 5, with increased capability in VLMs, methods that extract geographic coordinates from model outputs, such as the ones used in our probing setup, may start posing privacy concerns, as malicious actors could exploit them to acquire approximate locations from images.

Acknowledgements The authors would like to thank Kyeongjin Ahn, Wagner Meira Jr., and the anonymous reviewers for their valuable feedback. This work was partly supported by the National Science Foundation of Korea (NRF) Grant [RS-202200165347], and research funding from the 2026 Academic Research Promotion Support Project of Changwon National University, South Korea.

7. Limitations Probing Setup. For our probing setup, we use MSE as a loss, as it provides a transparent measure of signal density, while also offering a smooth, convex objective that makes our shallow probes easier to train. For this reason, we opt to use it instead of haversine distance for probing, although the latter is better suited for measuring distances along the surface of the earth.

References Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. ICLR Workshop Track, 2017.

Data Selection. This research relies on pretrained models, meaning that we cannot control which datasets are used to pretrain each model. As a consequence, some of the results may be influenced by the different architectural and data selection choices of each model suite. We attempt to control for this to the best of our ability, and the observed higher performance for VLMs is maintained even in settings where all models are trained on the same data (see Appendix D). Exploring when and how geospatial (and other types of) representations emerge in vision models pretrained from scratch is an interesting direction for future research.

Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660, 2021. Choi, J., Hauff, C., Laere, O. V., and Thomee, B. The placing task at mediaeval 2016. In MediaEval Benchmarking Initiative for Multimedia Evaluation, 2016. URL https://api.semanticscholar. org/CorpusID:6435875.

Memorization. Since the YFCC100M dataset was released before all of the models, and the data used for model pretraining is not transparent, memorization may play a role in the differences in performance. By focusing on the underlying structure of the latent space through probing and also utilizing a dataset with no captions (landmarks), we attempt to mitigate the effect that memorization may have had on

https://github.com/marceloslo/ Textual-Supervision-Enhances-GeospatialRepresentations

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. Ieee, 2009.

He, J., Nie, T., and Ma, W. Geolocation representation from large language models are generic enhancers for spatiotemporal learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 17094– 17104, 2025.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778, 2016. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009, 2022.

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.

Huh, M., Cheung, B., Wang, T., and Isola, P. Position: The platonic representation hypothesis. In International Conference on Machine Learning, 2024.

Fan, D., Tong, S., Zhu, J., Sinha, K., Liu, Z., Chen, X., Rabbat, M., Ballas, N., LeCun, Y., Bar, A., and Xie, S. Scaling language-free visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 370–382, October 2025.

Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025.

Ghiasi, A., Kazemi, H., Borgnia, E., Reich, S., Shu, M., Goldblum, M., Wilson, A. G., and Goldstein, T. What do vision transformers learn? a visual exploration. arXiv preprint arXiv:2212.06727, 2022.

Kim, J., Evans, J., and Schein, A. Linear representations of political perspective emerge in large language models. In The Thirteenth International Conference on Learning Representations, 2025.

Godey, N., De La Clergerie, É. V., and Sagot, B. On the scaling laws of geographical representation in language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 12416–12422, 2024.

Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306, 2024. McInnes, L., Healy, J., and Melville, J. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018.

Golub, G. H., Heath, M., and Wahba, G. Generalized crossvalidation as a method for choosing a good ridge parameter. Technometrics, 21(2):215–223, 1979.

Moayeri, M., Tabassi, E., and Feizi, S. WorldBench: Quantifying geographic disparities in LLM factual recall. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 1211–1228, 2024.

Gurnee, W. and Tegmark, M. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024.

Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.

Haas, L., Skreta, M., Alberti, S., and Finn, C. PIGEON: Predicting image geolocations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12893–12902, 2024. Han, J., Kamber, M., and Pei, J. Data Mining: Concepts and Techniques. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 3rd edition, 2011. ISBN 0123814790.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763. PmLR, 2021.

Hays, J. and Efros, A. A. IM2GPS: estimating geographic information from a single image. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8, 2008. doi: 10.1109/CVPR.2008.4587784. 10

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Roberts, J., Lüddecke, T., Sheikh, R., Han, K., and Albanie, S. Charting new territories: Exploring the geographic and geospatial capabilities of multimodal LLMs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 554–563, 2024.

tokenizer. In International Conference on Learning Representations, 2021.

Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. YFCC100M: The new data in multimedia research. Communications of the ACM, 59(2):64–73, 2016. Thorndike, R. L. Who belongs in the family? Psychometrika, 18(4):267–276, 1953. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. Vivanco Cepeda, V., Nayak, G. K., and Shah, M. GeoCLIP: Clip-inspired alignment between locations and images for effective worldwide geo-localization. Advances in Neural Information Processing Systems, 36:8690–8701, 2023. Weyand, T., Kostrikov, I., and Philbin, J. PlaNet - photo geolocation with convolutional neural networks. In European conference on computer vision, pp. 37–55. Springer, 2016. Weyand, T., Araujo, A., Cao, B., and Sim, J. Google landmarks dataset v2-a large-scale benchmark for instancelevel recognition and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2575–2584, 2020. Wu, N., Cao, Q., Wang, Z., Liu, Z., Qi, Y., Zhang, J., Ni, J., Yao, X., Ma, H., Mu, L., et al. TorchSpatial: A location encoding framework and benchmark for spatial representation learning. Advances in Neural Information Processing Systems, 37:81437–81460, 2024. Xu, H., Xie, S., Tan, X. E., Huang, P.-Y., Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feichtenhofer, C. Demystifying CLIP data. In International Conference on Learning Representations, 2024. Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975–11986, 2023. Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. Image BERT pre-training with online 11

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

A. Model Details Details about the models evaluated in this work are given in Table 2. Table 2. Models evaluated in this work—ViT (Dosovitskiy et al., 2021), ViT Masked Autoencoder (He et al., 2022), DINOv2 (Oquab et al., 2024), CLIP (Radford et al., 2021), LLaVA-1.5 (Liu et al., 2024), Qwen2.5 (Bai et al., 2025), and Gemma 3 (Kamath et al., 2025)—spanning vision-only and vision-language modalities with different training paradigms. Each family is evaluated with at least two size variants to examine the effect of model scale on learned representations.

Model

Modality

Training Methodology

ViT

Vision-Only

ViT Masked Autoencoder DINOv2 CLIP LLaVA-1.5

Vision-Only Vision-Only Vision-Language Vision-Language

Qwen2.5

Vision-Language

Gemma 3

Vision-Language

Supervised pretraining on ImageNet-21k followed by fine-tuning on ImageNet-1k. Self-supervised training using masked autoencoding. Self-supervised learning using a teacher-student framework. Trained on image-text pairs using contrastive learning. Combines a CLIP-based vision encoder with a language model via vision-language alignment, followed by instruction tuning. CLIP-based pretraining enhanced with vision-language alignment and end-to-end instruction tuning. Uses SIGLIP-based vision-text pretraining followed by alignment with instruction-tuned language models.

To extract the inner representations of the model, for the probing experiments, we use a single image as input and no prompt in the VLMs; then we conduct a single forward pass over the entire model, extracting all residuals A(l) for the last token for both the vision and the text components when applicable. A similar setup is adopted for the prompting experiments, but with added prompt tokens. Since we are not interested in generating text, we use no specific parameters (e.g., temperature) for the VLMs for the probing experiments. For the generation experiments, we use a very low temperature (0.0001) to reduce randomness. All experiments were run on a single Nvidia A100 GPU using the HuggingFace implementation of each model. The models that only take images as input were run at full precision, while the VLMs (Gemma, Qwen, and LLaVA) were run in bfloat16 precision.

B. Clustering Details Given the massive size of the YFCC100M dataset, its content was divided into semantic clusters for a better and more comprehensive analysis. For this, we started by extracting embeddings from the final convolutional layer of a ResNet-152 pretrained on ImageNet. Then, the resulting embeddings were L2 -normalized, reduced to 100 dimensions using PCA, and clustered with a standard k-means algorithm. To choose the optimal k, we tested 19 values (k = 10, 15, . . . , 100), and following (Thorndike, 1953), computed the within-cluster sum of squares (WCSS) as follows:

WCSS(k) =

X i

Di =

N X i=1

min c∈{1,...,k}

∥xi − µc ∥2 ,

which corresponds to the sum of squared distances for each point to its nearest cluster center. The value of WCSS(k) decreases monotonically as k increases, meaning that the optimal k is not the one which minimizes WCSS(k), but rather the one at which the decrease plateaus. This point corresponds to the “elbow” of the curve, which in our case was k = 40. A complete overview of the resulting clusters can be seen in Figure 7. In total, eight clusters represented people, including separate clusters for sports, musicians, and children, as well as another cluster for large gatherings. Another 11 clusters were related to man-made structures and architecture, including different types of buildings, walls, streets, and decorations. Animals were represented in two groups: one for birds and insects, and the other for larger animals. Eight clusters depicted different aspects of nature and different landscapes, such as mountains, lakes, beaches, soil, etc. Finally, seven clusters showed different types of objects, mainly food, drinks, displays, and vehicles. The remaining three clusters did not appear to exhibit any clear semantic relationship. 12

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Figure 7. Random samples for each of the 40 clusters obtained for the YFCC100M dataset.

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Figure 8. Two-dimensional visualization of the 40 clusters obtained using UMAP and colored according to macro-category.

For better visualization of the clusters obtained, we projected a small subset of 500 points per cluster into a 2D space using Uniform Manifold Approximation and Projection (UMAP) (McInnes et al., 2018). The generated plot (Figure 8) uses different shades of the same base color to represent clusters belonging to the same macro-category (people, nature, architecture, objects, animals, and others). Clusters associated with nature, architecture, and people are tightly grouped and occupy a distinct region of the latent space, further indicating that our clustering is semantically coherent. The majority of object clusters also lie in a clearly defined area, though there is some variability depending on the image background. Notably, cars and signs—traditionally found in urban settings—appear close to architecture clusters, while general vehicles, e.g., trains and airplanes, appear in between nature and architecture. The two animal clusters are in a subregion of nature, which is consistent with their broader photographic context, as these images are typically taken in green areas. One exception is the “People Outdoors” clusters, which, though grouped together with other people clusters, extend towards both architecture and nature regions, depending on the broader context of the images.

C. Dataset Description C.1. YFCC100M The YFCC100M dataset comprises approximately 99.2M images published on Flickr between 2004 and 2014 under a Creative Commons license (both commercial and non-commercial). It is a highly diverse set of photographs that depict natural and urban environments, people, objects, and everyday events, taken by a mix of professional photographers and casual users. In our experiments, we used a subset of 4,233,900 images, which preserves the diversity of the original set while offering reliable geolocation annotations. The spatial distribution of the data set is highly unbalanced (Figure 9), with the majority of the samples concentrated in a few key regions. Together, the G7 countries account for more than 57% of all samples, a figure that rises to 78% with the addition of the rest of the European countries. Asia is the third most represented continent with close to 500k images (12.1% of the data), more than half of which are from East Asia. In contrast, Central Asia and the Middle East are particularly underrepresented, with no country in either region contributing more than 15k samples. South America accounts for just over 180k images (4.2% of the data), with a sample distribution that closely matches that of the continent’s population. The main outliers are Chile, overrepresented in 17% of the data versus 4.5% of the population, and Venezuela, underrepresented at less than 3% of the samples despite being 6.5% of the total population. Africa contributes approximately 68k images (1.6% of the dataset), with only 12 countries represented by more than 1k samples. Finally, Oceania provides 136k (3.2% of the data) samples, almost entirely from Australia and New Zealand, which together account for 131k. We note that, although there is some variation, all clusters exhibit roughly the same imbalance. After sampling, the balance is marginally improved. Of the 200k total sampled images (5k per cluster), 37k (18.5%) are from Asia, almost 14k (7.0%) are from South America, 8.9k (4.5%) are from Oceania, and 7.4k (3.7%) are from Africa, 14

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Figure 9. Spatial distribution of all samples from our subset of YFCC100M.

Figure 10. World coverage of random sampling (left) and geocells-based sampling (right).

while the participation of G7 countries decreases from 57% to 44%. Figure 10 shows the effects of these improvements, with our sampling methodology leading to better world coverage than a purely random strategy. C.2. Google Landmarks Google Landmarks V2 contains over 5M images of over 200k landmarks across the globe, collected mostly from Wikimedia Commons4 . As the coordinates of images are not typically available on Google Landmarks, we used a subset of 581,215 geolocalized images from 15,453 different labeled landmarks. Though photos of relevant sites are expected to be localizable, the dataset also contains some non-localizable images, such as close-up shots of animals in a national park, as well as paintings and statues. This data set has a pronounced spatial imbalance, as shown in Figure 11. Samples are mainly concentrated in Europe, which alone accounts for over 67% of our subset, followed by Asia (15%), North America (11%), South America (2.5%), Africa (1.1%), and Oceania (1.0%). Outside of Europe, samples are disproportionately concentrated in a few major cities, with less populated areas remaining mostly uncovered. For our experiments, we selected a single random image from each landmark and proceeded to sample 5k images as outlined in Section 3.2.1. The sampling procedure preserved the same continent-level spatial bias present in the original dataset, though the concentration of samples in major cities was reduced. To investigate the effects of spatial imbalance on geo-localization performance, we compute several spatial sensitivity metrics in our subset of Google Landmarks V2. We focus on three complementary indicators: (i) marked spatial self-information (SSI) geo-bias score (Wu et al., 2024), which measures the extent to which a model’s incorrect predictions cluster spatially beyond what would be expected from random errors; (ii) median geodesic error (in kilometers); and (iii) median proximity error (Gurnee & Tegmark, 2024), which quantifies the fraction of candidate predictions that lie closer to the ground-truth location than the model’s own prediction (i.e., a rank-based error). All metrics were computed at the sample level and 4

https://commons.wikimedia.org/

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Figure 11. Spatial distribution of all samples from our subset of Google Landmarks V2. Table 3. Marked SSI geo-bias score per continent for a representative subset of VLMs with a 95% confidence interval (lower is better). NA and SA indicate North and South America, respectively.

Model Qwen2.5-VL-3B Qwen2.5-VL-7B LLaVA-1.5-7B-HF CLIP-ViT-large Gemma-3-12B-PT

Africa

Asia

Europe

Oceania

56.7 ± 5.82 63.9 ± 4.79 52.6 ± 5.02 64.0 ± 5.18 44.7 ± 4.49

59.3 ± 2.05 73.7 ± 2.35 57.3 ± 2.22 66.8 ± 2.07 48.5 ± 1.74

50.1 ± 0.90 66.2 ± 0.91 49.9 ± 0.88 56.4 ± 0.89 43.2 ± 0.77

58.9 ± 1.97 73.1 ± 2.06 56.5 ± 2.04 63.1 ± 1.99 50.2 ± 1.67

56.4 ± 4.19 74.9 ± 4.57 61.1 ± 4.26 77.4 ± 6.77 53.4 ± 4.13

67.2 ± 8.16 84.0 ± 6.44 73.9 ± 9.86 64.8 ± 4.19 53.3 ± 6.68

aggregated both per continent and over the full dataset. For SSI geo-bias, we used the original implementation, made available by the authors 5 , and defined a prediction as good if its geodesic error was smaller than 1,000 km. A value of 0 indicates that prediction errors are spatially random, while higher values reflect increasingly clustered errors. The resulting scores (visually illustrated in Figure 12) show that most vision-only models are heavily spatially biased, as their predictions tend to be clustered around the densest parts of the dataset. A more in-depth look at our best-performing VLMs is provided in Table 3. Overall, SSI values tend to be smaller for samples originally located in Europe, suggesting that errors in these samples behave more randomly compared to those in other regions. However, even within Europe, values remain relatively high, indicating that the model performs better in specific subregions or that they disproportionately succeed in urban areas, where sample concentration is higher. Though SSI is a good indication of how the non-random spatial distribution of prediction errors is, it does not reflect the magnitude of those errors across regions. To assess this, we examine geodesic and proximity errors of our selected VLMs, reported in Tables 4 and 5, respectively. Across all models, geodesic error was consistently and significantly lower in Europe, further reinforcing that performance tends to be better in this region. In contrast, South America and Oceania were the most challenging continents for all models, indicating that the high concentration of samples in the Northern Hemisphere could have a significant impact on the success rate of Southern Hemisphere predictions. As expected, proximity error is the highest in Europe, as the higher density makes retrieval ranking more difficult. On the other hand, South America and Africa had surprisingly low proximity errors, indicating that, though the models may not be good at precisely predicting coordinates in these regions, they generally stay within the correct part of the world. Oceania, by contrast, exhibits much higher proximity errors, indicating that even correctly identifying samples as being in Oceania might be challenging enough for the models. To better understand how these continent-level trends manifest spatially, we also visualize the median geodesic error of Gemma-3-12B-PT on a 4◦ × 4◦ grid (Figure 13, left). As suggested by the aggregated metrics, errors are consistently 5

https://github.com/seai-lab/GeoBS

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Figure 12. Marked SSI geo-bias values for all models on our subset of Google Landmarks V2 Table 4. Median geodesic error (km) per continent for a representative subset of VLMs. Lower is better. Model

Africa

Asia

Europe

N. America

S. America

Oceania

Total

Qwen2.5-VL-3B Qwen2.5-VL-7B LLaVA-1.5-7B-HF CLIP-ViT-large Gemma-3-12B-PT

2185.6 2246.5 2485.8 2440.6 1865.9

1686.2 1688.7 1754.8 1950.1 1546.3

800.1 959.9 799.4 827.9 752.9

1966.5 1890.7 1785.1 2000.7 1768.0

4119.1 4225.0 5032.5 8151.6 3965.3

9212.8 9720.4 7140.4 5257.9 7259.0

1029.1 1173.5 1030.4 1091.9 962.6

lower in Europe and noticeably higher throughout the Southern Hemisphere. North America and Asia exhibit less uniform behavior, characterized by a mix of high- and low-error regions. Figure 13 (right) shows the model’s predicted coordinates, colored by their ground-truth continent. This plot corroborates our earlier observations that, except for Oceania, the model typically places predictions within the correct broad region of the world despite large geodesic errors in certain areas.

D. Effect of Architecture and Scale In our experiments, we employ a variety of models with different training objectives, architectures, scales, and training data. By selecting a large array of models, we find compelling evidence that the use of textual data might be one driving factor for improved geospatial representations inside each model. For instance, even models with more than 1B parameters (DINOv2-giant), when trained solely on visual data in a self-supervised manner, fail to outperform much smaller models, such as CLIP-base, for most of the subsets of our data. Still, it is not clear how much of this can be attributed to the images used to train each model, rather than the text. For this reason, we make a stricter comparison in this section using models with very similar architectures and scale, pretrained on the same dataset, with the major difference being the training objective and the inclusion (or not) of text. We select the MetaCLIP and Web-SSL DINO and MAE models, which are all trained on the MetaCLIP dataset, a collection of image-caption pairs sampled from Common Crawl. The summary of probing results for different model configurations is presented in Table 6. We also show the architectural details for each model in Table 6 for easier comparison. 17

Textual Supervision Enhances Geospatial Representations in Vision-Language Models Table 5. Median proximity error per continent for a representative subset of VLMs. Lower is better. Model

Africa

Asia

Europe

N. America

S. America

Oceania

Total

Qwen2.5-VL-3B Qwen2.5-VL-7B LLaVA-1.5-7B-HF CLIP-ViT-large Gemma-3-12B-PT

0.014 0.016 0.011 0.015 0.012

0.042 0.043 0.043 0.046 0.037

0.224 0.309 0.223 0.242 0.201

0.079 0.080 0.073 0.079 0.072

0.026 0.026 0.027 0.054 0.025

0.120 0.113 0.031 0.028 0.039

0.119 0.144 0.113 0.129 0.104

Figure 13. More in-depth look into Gemma-3-12B-PT predictions in our subset of Google Landmarks V2. Median geodesic error per 4◦ × 4◦ grid (left). All predictions are colored by the ground truth continent (right).

The results for this analysis match our previous observations, with the MAE model, on average, showing the weakest evidence of internal geospatial representations, followed by DINO and finally by CLIP. Among vision-only objectives, we note that the DINOv2 trained model is once again associated with increased quality of geospatial representations when compared to the MAE, suggesting that textual supervision, albeit more effective, is not the only way to introduce this kind of information. Moreover, MetaCLIP large and huge models, trained with a smaller compute budget, still outperform the DINO-7B model, trained on the same dataset, suggesting the usefulness of textual supervision. Further developments in self-supervised learning for ViTs may continue to close this gap in the future, especially as scaling up the number of parameters seems to be correlated with increased performance for this task.

E. Linearity of Geospatial Features In our experiments, we focus on the linear probing for geospatial representation in vision-only and vision-language models, finding that the latter group has internal representations that can be mapped to real-world locations, while the former group is much more limited in this regard. However, it could be the case that this happens only because vision-only models represent this kind of information non-linearly. To explore this possibility, we also train non-linear probes (one hidden layer MLP regression) to check whether vision-only models may be representing geospatial features differently. Figure 14 shows the performance (R2 ) for the non-linear probes for each model. Using non-linear probes did not result in a Table 6. Probe R2 in the landmarks dataset and architectural/training details for models trained on MetaCLIP data. All models have a 224 × 224 image resolution.

Architecture

Objective

Parameters

Patch Size

Train Steps

MetaCLIP-base MetaCLIP-large MetaCLIP-huge Web-SSL DINO-300M Web-SSL DINO-1B Web-SSL DINO-7B Web-SSL MAE-300M Web-SSL MAE-1B

Contrastive Image-Text Pretraining

87M 300M 600M 300M 1B 7B 300M 1B

16 14 14

12.8B

2B 2B 8B

16 14

DINOv2 Loss Masked Autoencoder 18

R2 0.53 0.70 0.73 0.26 0.39 0.56 0.11 0.14

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

0.7 0.6 0.5 0.4

R²

Vision-only Vision-language

0.3 0.2 0.1

Objects Uncategorized Glowing Lights Flowers Musicians Drinks Soil People Closeup Birds and Insects Food Animals Displays Vehicles Interiors Children Poles and Columns Text People 2 Colorful Outdoors Religious Monuments Horizon Statues People 1 Banks/ Beaches Wall Large Gatherings Cold Places Cars Roads and Railways Nature Sports People Outdoors Piers Lakes Cliffs and Mountains Constructions Buildings Signs Old Buildings Landmarks Streets

vit-mae-base vit-mae-large vit-base-patch16-224 vit-large-patch16-224 dinov2-base dinov2-large dinov2-giant clip-vit-base-patch32 Qwen2.5-VL-3B-Instruct Qwen2.5-VL-7B-Instruct gemma-3-4b-it gemma-3-4b-pt clip-vit-large-patch14

0.0

Figure 14. Performance (R2 ) of each model when using a non-linear probe. The x-axis shows different clusters of the YFCC100M dataset and the Landmarks dataset, while the y-axis shows the models evaluated.

significant increase in R2 for any model, thus strengthening the claim that vision-language models’ representations encode geospatial information better.

F. Ablation Studies Our probing setup utilizes the summary representation of the input image, which is either the [CLS] token or the final token representation. However, it could be the case that a geospatial representation emerges across different tokens corresponding to different image patches. To control for this, we also train the same probes using the concatenation of the min and max pooling across all tokens as the inputs for the ridge regression. Figure 15 shows the results across our datasets. It is possible to see that, when compared to the default probing approach, the R2 is actually smaller for the vision-only models, suggesting that the summary token is adequate as a set of features for the probing setup.

G. Additional Prompting Results In Section 4.3, we show that by using specific textual prompts, such as “Guess the latitude and longitude.”, we can strengthen the performance of the probes on the latter text layers of VLMs, suggesting improved representation of geospatial features. In this section, we expand on that by comparing the usage of prompts with varying levels of relevance for coordinate prediction. Figure 16 shows the R2 for the probes trained on models using five different prompt configurations: • None: No prompt, only the image. • Random: 20 tokens sampled randomly for each picture (consistent across models). • Lat/Lon: “Guess the latitude and longitude of this image. Answer only with the coordinate tuple (lat, long)” 19

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

0.7 0.6 0.5 0.4 R²

Vision-only Vision-language

0.3 0.2 0.1 0.0

Objects Uncategorized Glowing Lights Musicians Drinks Flowers Food Displays Birds and Insects Soil People Closeup Animals Interiors Vehicles Text Colorful Outdoors Poles and Columns Religious Monuments Children People 2 Statues Wall Horizon People 1 Banks/ Beaches Cold Places Large Gatherings Roads and Railways Cars Sports People Outdoors Piers Lakes Nature Cliffs and Mountains Signs Buildings Old Buildings Constructions Landmarks Streets

Figure 15. Performance (R2 ) of each model when using a linear probe and the concatenation of max and min pooling across input tokens as features. The x-axis shows different clusters of the YFCC100M dataset and the Landmarks dataset, while the y-axis shows the models evaluated.

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

0.8

0.6

0.2

0.4 0.2

0.0 0.2

Text layers

0.8

0.4

0.4 0.2

0.0 0

Layer

0.2

llava-1.5-7b-hf

1.0

0.8

Text layers

R²

gemma-3-4b-pt

1.0

None Random Lat/Lon City/Country Photo Location

Text layers

Qwen2.5-VL-7B-Instruct

1.0

0.0 0

Layer

0.2

Layer

Figure 16. R2 for different prompting strategies on the landmarks dataset. Note how the use of prompts requiring geospatial awareness leads to increased R2 across studied models towards the latter layers.

• City/Country: “What country and city was this picture taken in? Answer only with the city and country names.” • Photo Location: “Where is this photo?” It is possible to see that, on average, the more explicit prompts are correlated with higher probing performance, while random and no prompting strategies do not show any significant gain as the information propagates through the network. This further suggests that textual information may help the model retrieve and refine geospatial information from the image representations.

H. Additional Representation Swapping Results In this section, we present additional examples of location swap results obtained from the methodology described in Section 4.5. Here, we show that the intervention used the swapping methodology often leads to other changes in the text, e.g., different place names, mixing characteristics from both images, especially if the text generated has very different lengths for the source and target images. Table 7 shows examples of generated text after swapping fraction p dimensions from the source image with the target image. We observe that some figures have strong features, while others have easily edited geospatial information. For example, with the same fraction of replaced dimensions, we can move the Cologne Cathedral to Paris, but we cannot move the Eiffel Tower to Cologne. Additionally, the St. Peter Cathedral in the Vatican overwrites all other information when used as a target image. We contrast these results with randomly swapping a fraction p of the embeddings with no regard for geospatial importance (Table 8). It is possible to see that in all cases, by swapping random dimensions, the text generation changes dramatically when compared to our approach, with most samples either keeping their location or having much more added gibberish. We also evaluate the results quantitatively. For this, we extract a subset of landmarks that are likely known by the model. Namely, we use pictures from the most visited tourist spots from Wikipedia 6 , together with the new seven wonders of the world, totaling 51 places. For each of those, we sample five locations from different countries to serve as targets following our approach. We generate the text for each of these samples using the Qwen2.5-VL-3B model and remove the places that the VLM was unable to correctly localize. After these steps, we were left with 208 (source, target) tuples. We apply our steering methodology from eq. 5 and random swapping to this dataset. The results are evaluated regarding the method’s efficacy, stability, and success rate. We define efficacy as the probability of having the model change its generation for the location from the source to the target location (country, city, or region). Meanwhile, stability refers to the probability of the generated text keeping the original source place name (e.g., Eiffel Tower) after the intervention. Finally, an intervention was successful if both of these conditions applied. These metrics provide a lower bound for the effectiveness of each method as they rely on exact substring matching for identifying whether an intervention was successful or not, and thus, minimal changes such as typos or small amounts of extra gibberish have a big effect on the result. Table 9 summarizes these metrics for different proportion p values. There is a tradeoff between stability and efficacy, with 6

https://en.wikipedia.org/wiki/List_of_most-visited_palaces_and_monuments

Textual Supervision Enhances Geospatial Representations in Vision-Language Models Table 7. Results for embedding swapping using varied landmark pictures as source and target images. The changes are very drastic for some image combinations, despite similar methodology when implementing the interventions.

Source Image

Target Image

Generated Text

0.40

The image depicts the Cologne Cathedral, also known as the Cologne Cathedral, located in Cologne, France

0.40

The image depicts the Cologne Cathedral, also known as the Cathedral of Notre-Dame de Paris, located in the of Paris, France.

0.45

The image depicts the. Peter’s Basilica, Vatican City.

0.50

The image depicts the Taji Palace, Rome, Italy.

0.50

The image depicts the Hagou Basil Mosque, Istanbul, Italy.

the former decreasing with higher p while the latter increases. Note how the efficacy of our method is much higher for small p when compared to random, despite showing similar stability. Maximum success rate is achieved at p = 0.6, where 11% of the generations successfully included the target location while keeping the source place name.

I. Fine-Tuning Details To mitigate the spatial imbalance of our data, which could negatively affect country identification, we constructed a balanced sampling of Google Landmarks across countries. First, we selected a single image for each of the 15,453 labeled landmarks. From these, we retained only those belonging to the 51 countries with over 30 samples, and sampled up to 100 images per country, resulting in a dataset of 3,992 images. Using this data, we fine-tuned CLIP-large, ViT-large, ViT-MAE-large, and DINOv2 (large and giant) for classification. All models were trained for five epochs with a 70% train, 20% validation, and 10% test splits using an AdamW optimizer, a batch size of 16, and learning rates of 1 × 10−5 , 2 × 10−5 , 5 × 10−5 , 1 × 10−4 , 2 × 10−4 , and 5 × 10−4 , with the best chosen based on validation loss. β1 and β2 were set as 0.9 and 0.999, respectively. Figure 17 shows the training and validation loss for the fine-tuned models. We can see that CLIP and DINOv2 models achieve their best validation loss within 400 training steps, with CLIP’s being overall lower, which also translates to better test accuracy.

J. Consistency of Influential Weights Between Models Through our probing experiments, we obtain a large set of regression coefficients for latitude and longitude, one pair for each layer and cluster/dataset. Using these coefficients, we investigate to what extent different types of images are represented 22

Textual Supervision Enhances Geospatial Representations in Vision-Language Models Table 8. Results for swapping random embeddings using varied landmark pictures as source and target images. Note how, when compared to our approach, the locations are no longer swapped.

Source Image

Target Image

Generated Text

0.40

The image depicts the Cologne Cathedral, also known as the Cologne Cathedral, in Cologne, Germany.

0.40

The image depicts the Cologne Cathedral,ptyhek in Cologne, Germany.

0.45

The image depicts the Eiffel Tower, a famous landmark in Paris, France.

0.50

The image depicts the Taj Mah Mahal, a an iconic monument in India.

0.50

The image depicts the Hagia Sophia (Hagia Sophia) in Istanbul, Turkey.

vit-mae-large

vit-large-patch16-224-in21k

dinov2-giant

clip-vit-large-patch14

dinov2-large

Loss

3 2 1 0 4

Loss

train eval

2 1 0

200

400

600

Train Steps

800

200

400

600

Train Steps

800

Figure 17. Training and validation loss for model fine-tuning.

Textual Supervision Enhances Geospatial Representations in Vision-Language Models Table 9. Quantitative results for applying the representation swapping method from Section 4.5 and a random swapping baseline. Qualitatively, higher values of p for both random and our method increase the generation of gibberish and can cause the generated text to mix features from both locations.

p 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Stability

Efficacy

Success Rate

Our Method

Random

Our Method

Random

Our Method

Random

1.00 0.97 0.95 0.93 0.73 0.33 0.12 0.05 0.04 0.04 0.03

1.00 0.98 0.98 0.96 0.76 0.41 0.20 0.10 0.04 0.04 0.03

0.00 0.00 0.00 0.00 0.01 0.12 0.78 0.93 0.97 0.97 0.97

0.00 0.00 0.00 0.00 0.00 0.02 0.41 0.83 0.97 0.97 0.97

0.00 0.00 0.00 0.00 0.01 0.02 0.11 0.04 0.04 0.04 0.03

0.00 0.00 0.00 0.00 0.00 0.01 0.06 0.07 0.03 0.03 0.03

similarly in the models’ latent space. Table 10 shows the average Pearson correlation between each pair of coefficients and the average R2 for a given model on its best performing layer overall. It is possible to see that for all models, correlations are positive, suggesting that some spatial information is common for a variety of different image subjects. However, for most of the vision-only models, the correlation is very weak (ρ < 0.3). Meanwhile, for the models pretrained with language data, we see that correlations are low (0.3 ≤ ρ < 0.5) to moderate (0.5 ≤ ρ < 0.7), indicating some shared neurons for geospatial representation across different image types. These correlations get much higher when we consider only the 40% most important dimensions for predicting coordinates (see Section 4.5), with CLIP and LLaVA achieving values above 0.7, as shown in Table 11. We also check the consistency of the model weights by evaluating how the ranking of the coefficients between embedding dimensions is kept on different seeds. We use Spearman’s rank correlation to measure how well the rank of each coefficient is kept across a set of five distinct seeds when compared to our original run. Additionally, we calculate Jaccard similarity for the overlap between the top 50% of coefficients (p = 0.5) to measure how consistently relevant the top embedding dimensions are for the ridge regression. Table 12 shows the results for this analysis. We note that, generally, models have stable top coefficients, regardless of textual supervision. This is illustrated by the high Spearman correlation coefficients and Jaccard similarities, which suggest that the same features are consistently the most useful for extracting geolocation information across runs. Among all models, the ViT-MAE family has the lowest values for both metrics, possibly due to its lower capability of producing meaningful linearly separable geospatial representations from the images.

K. Full Probing results Tables 13 and 14 show the full results for probing experiments with a 95% confidence interval obtained from the mean R2 over five-fold cross-validation for each dataset. Confidence intervals are generally low, indicating consistency of our probing approach across different subsets of our data.

L. Effect of YFCC100M Caption Memorization The fact that image captions can a priori contain any kind of information, including location coordinates, raises the concern that VLMs may show better performance than vision-only models solely because they have memorized the captions. To investigate this hypothesis, we conduct a new series of experiments on the YFCC100M dataset, explicitly excluding any caption that mentioned country names or coordinates. Out of our initial sample of 200K images across the 40 clusters, 17 images had a latitude or longitude coordinate and 17,634 images had a country name attached. These images were relatively 24

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Table 10. Average correlation (ρ) between regression coefficients for different clusters/datasets for each model. The ± term denotes the 95% confidence interval for the average.

Model ViT-MAE-base ViT-MAE-large ViT-base ViT-large DINOv2-base DINOv2-large DINOv2-giant CLIP-ViT-base Web-SSL DINO-7B Qwen2.5-VL-3B-IT Gemma-3-4B-IT Qwen2.5-VL-7B-IT Gemma-3-12B-IT Gemma-3-4B-PT Gemma-3-12B-PT CLIP-ViT-large MetaCLIP-huge LLaVA-1.5-7B-HF

Average ρ Latitude

Longitude

0.142 ± 0.007 0.150 ± 0.006 0.164 ± 0.005 0.231 ± 0.006 0.283 ± 0.006 0.312 ± 0.007 0.294 ± 0.006 0.410 ± 0.005 0.195 ± 0.004 0.388 ± 0.005 0.426 ± 0.005 0.373 ± 0.005 0.455 ± 0.005 0.421 ± 0.005 0.219 ± 0.005 0.548 ± 0.006 0.522 ± 0.005 0.573 ± 0.006

0.099 ± 0.005 0.116 ± 0.004 0.153 ± 0.004 0.229 ± 0.005 0.294 ± 0.007 0.291 ± 0.006 0.300 ± 0.005 0.500 ± 0.005 0.225 ± 0.004 0.348 ± 0.004 0.372 ± 0.005 0.325 ± 0.004 0.451 ± 0.004 0.378 ± 0.005 0.179 ± 0.004 0.587 ± 0.005 0.580 ± 0.005 0.612 ± 0.005

Average R2 0.034 ± 0.008 0.051 ± 0.010 0.118 ± 0.020 0.145 ± 0.020 0.191 ± 0.033 0.236 ± 0.037 0.262 ± 0.040 0.278 ± 0.034 0.327 ± 0.048 0.344 ± 0.047 0.382 ± 0.046 0.398 ± 0.049 0.421 ± 0.049 0.421 ± 0.050 0.443 ± 0.055 0.482 ± 0.048 0.486 ± 0.049 0.510 ± 0.049

Table 11. Average correlation (ρ) between the 40% most important regression coefficients for different clusters/datasets for each model. The ± term denotes the 95% confidence interval for the average.

Model ViT-mae-base ViT-mae-large ViT-base ViT-large DINOv2-base DINOv2-large DINOv2-giant CLIP-ViT-base Web-SSL DINO-7B Qwen2.5-VL-3B-IT Gemma-3-4B-IT Qwen2.5-VL-7B-IT Gemma-3-12B-IT Gemma-3-4B-PT Gemma-3-12B-PT CLIP-ViT-large MetaCLIP-huge LLaVA-1.5-7B-HF

Average ρ Latitude

Longitude

0.190 ± 0.008 0.210 ± 0.008 0.253 ± 0.007 0.349 ± 0.008 0.407 ± 0.008 0.440 ± 0.008 0.425 ± 0.007 0.586 ± 0.005 0.298 ± 0.005 0.548 ± 0.005 0.455 ± 0.005 0.521 ± 0.005 0.475 ± 0.005 0.463 ± 0.005 0.275 ± 0.005 0.718 ± 0.005 0.695 ± 0.005 0.739 ± 0.005

0.135 ± 0.006 0.160 ± 0.005 0.230 ± 0.005 0.351 ± 0.006 0.421 ± 0.008 0.413 ± 0.007 0.434 ± 0.007 0.674 ± 0.005 0.338 ± 0.005 0.498 ± 0.004 0.399 ± 0.005 0.459 ± 0.004 0.470 ± 0.004 0.416 ± 0.005 0.224 ± 0.005 0.749 ± 0.004 0.745 ± 0.004 0.771 ± 0.004

Textual Supervision Enhances Geospatial Representations in Vision-Language Models Table 12. Spearman correlation and Jaccard similarity for the coefficients for the landmarks dataset across five seeds. Jaccard is calculated for the overlap between the top 50% dimensions across runs.

Model ViT-MAE-base ViT-MAE-large ViT-base ViT-large DINOv2-base DINOv2-large DINOv2-giant Web-SSL DINO-7B CLIP-ViT-base Qwen2.5-VL-3B-IT Gemma-3-4B-IT Qwen2.5-VL-7B-IT Gemma-3-12B-IT Gemma-3-4B-PT Gemma-3-12B-PT CLIP-ViT-large MetaCLIP-huge LLaVA-1.5-7B-HF

Spearman ρ

Jaccard Similarity

Latitude

Longitude

Latitude

Longitude

0.78 0.75 0.88 0.89 0.90 0.93 0.89 0.79 0.89 0.92 0.83 0.88 0.87 0.88 0.90 0.94 0.91 0.94

0.72 0.73 0.87 0.88 0.91 0.94 0.90 0.79 0.91 0.91 0.82 0.85 0.85 0.83 0.87 0.92 0.93 0.94

0.51 0.50 0.61 0.63 0.65 0.71 0.63 0.53 0.63 0.67 0.57 0.62 0.66 0.62 0.66 0.72 0.66 0.69

0.48 0.47 0.59 0.63 0.66 0.71 0.62 0.53 0.67 0.66 0.56 0.59 0.64 0.56 0.60 0.68 0.68 0.70

well distributed across clusters, meaning that any difference in cluster performance is unlikely to be correlated to exact caption memorization. After filtering out these images where data-leakage through memorization was possible, we reran our probing setup. Figure 18 shows the results of probing the models on this new dataset. As it can be seen, results are roughly unchanged, with all performances being at most 0.025 smaller in specific model/cluster combinations. In fact, for the best performing clusters, such as streets, removing these images actually led to an increase in performance. These results suggest that exact memorization of captions is not the reason for the differences in performance observed by the two types of models.

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

ViT-MAE-large

ViT-base

ViT-large

DINOv2-base

DINOv2-large

DINOv2-giant

Web-SSL DINO-7B

ViT-MAE-base

Table 13. R2 scores for vision-only models for each cluster and for the landmarks dataset, with columns and rows sorted by average R2 in ascending order. The ± term indicates the 95% confidence interval.

.01±.01 .01±.01 .02±.01 .02±.00 .02±.01 .01±.01 .02±.00 .03±.01 .02±.01 .02±.01 .03±.01 .01±.01 .01±.00 .01±.01 .01±.01 .02±.00 .04±.01 .01±.00 .03±.01 .03±.01 .02±.01 .03±.01 .03±.01 .05±.01 .05±.01 .05±.00 .02±.01 .04±.01 .06±.01 .04±.01 .06±.01 .04±.01 .06±.01 .03±.01 .04±.01 .07±.01 .09±.00 .08±.01 .09±.01 .09±.01 .07±.01

.01±.01 .01±.01 .02±.00 .02±.00 .04±.01 .01±.00 .04±.01 .04±.00 .04±.01 .04±.01 .06±.01 .02±.01 .03±.00 .03±.01 .01±.01 .03±.01 .08±.01 .03±.01 .05±.01 .05±.01 .03±.00 .05±.00 .05±.01 .06±.01 .07±.01 .07±.00 .05±.01 .06±.01 .07±.00 .06±.01 .08±.01 .05±.01 .08±.01 .04±.01 .06±.01 .09±.01 .12±.01 .11±.01 .13±.01 .11±.02 .10±.01

.01±.02 .01±.01 .03±.00 .05±.01 .10±.01 .05±.01 .08±.01 .09±.01 .10±.01 .10±.02 .13±.01 .05±.01 .07±.00 .07±.01 .07±.01 .09±.01 .16±.01 .06±.00 .08±.01 .11±.02 .10±.01 .10±.01 .14±.01 .11±.01 .18±.01 .12±.01 .13±.01 .13±.01 .14±.01 .17±.02 .20±.00 .17±.02 .20±.01 .12±.01 .17±.02 .19±.01 .21±.02 .22±.02 .21±.01 .22±.03 .21±.01

.02±.01 .03±.01 .04±.01 .06±.00 .15±.01 .06±.01 .12±.01 .11±.01 .17±.01 .14±.01 .15±.01 .08±.01 .08±.01 .09±.01 .08±.01 .12±.00 .18±.01 .10±.00 .13±.01 .13±.01 .12±.00 .13±.01 .17±.01 .14±.01 .20±.01 .14±.01 .15±.01 .15±.01 .16±.01 .19±.01 .22±.01 .20±.02 .25±.01 .15±.01 .20±.02 .22±.01 .25±.01 .25±.01 .25±.01 .26±.02 .24±.01

.02±.01 .04±.01 .08±.01 .06±.01 .11±.01 .06±.01 .12±.01 .11±.01 .14±.01 .13±.01 .13±.02 .09±.02 .10±.01 .12±.01 .11±.01 .14±.01 .15±.01 .15±.01 .20±.01 .18±.01 .20±.01 .21±.01 .17±.01 .16±.02 .17±.01 .23±.01 .19±.01 .22±.01 .19±.02 .17±.01 .22±.02 .28±.02 .30±.02 .26±.02 .25±.02 .33±.01 .39±.01 .36±.02 .43±.03 .39±.03 .45±.01

.05±.01 .07±.01 .12±.01 .09±.01 .15±.01 .09±.00 .15±.02 .14±.02 .19±.01 .17±.02 .17±.01 .12±.01 .14±.00 .17±.02 .13±.01 .17±.01 .19±.01 .21±.01 .24±.01 .22±.01 .25±.01 .28±.02 .22±.01 .21±.01 .22±.01 .28±.01 .25±.01 .28±.01 .25±.01 .25±.01 .28±.02 .34±.02 .36±.02 .32±.01 .31±.02 .41±.01 .47±.01 .42±.01 .53±.02 .44±.02 .56±.01

.04±.01 .08±.02 .14±.02 .11±.01 .17±.01 .10±.00 .18±.01 .15±.02 .23±.01 .20±.02 .18±.02 .14±.01 .16±.01 .18±.02 .15±.01 .19±.01 .20±.01 .23±.01 .27±.02 .25±.01 .28±.01 .31±.02 .24±.01 .23±.02 .23±.02 .32±.02 .27±.01 .31±.01 .28±.01 .27±.01 .32±.02 .36±.02 .38±.01 .35±.01 .35±.02 .45±.01 .52±.01 .46±.01 .59±.02 .48±.02 .60±.01

.06±.01 .08±.02 .18±.02 .17±.01 .14±.01 .16±.01 .18±.02 .18±.02 .17±.02 .21±.03 .25±.01 .22±.02 .22±.01 .29±.02 .24±.01 .26±.01 .27±.02 .30±.02 .30±.01 .31±.01 .37±.02 .37±.03 .30±.02 .34±.01 .34±.02 .42±.02 .41±.02 .40±.01 .33±.01 .45±.02 .41±.01 .49±.01 .46±.01 .50±.02 .48±.01 .57±.01 .58±.01 .59±.02 .66±.01 .56±.02 .69±.01

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

Qwen2.5-VL-3B

Gemma-3-4B-IT

Qwen2.5-VL-7B

Gemma-3-12B-IT

Gemma-3-4B-PT

Gemma-3-12B-PT

CLIP-large

MetaCLIP-huge

LLaVA-1.5-7B-HF

CLIP-base

Table 14. R2 scores for models trained with vision and language data for each cluster and for the landmarks dataset, with columns and rows sorted by average R2 in ascending order. The ± term indicates the 95% confidence interval.

.07±.01 .08±.02 .13±.02 .15±.01 .12±.01 .16±.01 .16±.01 .20±.01 .15±.01 .22±.01 .25±.02 .23±.03 .22±.01 .22±.01 .24±.02 .23±.01 .29±.01 .24±.02 .22±.02 .26±.02 .27±.01 .24±.01 .30±.01 .28±.01 .32±.01 .30±.02 .34±.02 .31±.01 .33±.01 .38±.02 .37±.02 .35±.01 .36±.01 .34±.02 .37±.02 .43±.02 .43±.01 .46±.01 .43±.02 .46±.03 .49±.01

.09±.01 .11±.01 .12±.01 .15±.02 .22±.01 .16±.01 .22±.02 .19±.02 .30±.02 .21±.01 .22±.01 .26±.02 .27±.01 .24±.01 .34±.01 .28±.01 .26±.01 .30±.02 .31±.02 .33±.02 .31±.02 .32±.02 .32±.01 .35±.02 .33±.02 .35±.01 .40±.03 .41±.02 .43±.00 .42±.02 .42±.02 .44±.01 .44±.01 .46±.01 .48±.02 .56±.00 .57±.02 .58±.02 .63±.01 .74±.02 .59±.02

.10±.01 .11±.01 .16±.01 .20±.02 .19±.01 .24±.02 .24±.02 .27±.02 .26±.01 .28±.01 .25±.01 .30±.03 .33±.02 .35±.02 .35±.01 .36±.00 .31±.01 .34±.01 .34±.02 .35±.03 .34±.02 .35±.01 .36±.01 .39±.01 .41±.02 .37±.02 .47±.01 .48±.01 .50±.01 .49±.02 .50±.02 .47±.02 .47±.01 .50±.01 .50±.02 .61±.01 .58±.02 .62±.01 .62±.01 .63±.01 .67±.01

.12±.01 .13±.02 .16±.01 .19±.02 .24±.02 .22±.01 .28±.02 .25±.02 .34±.02 .26±.02 .26±.01 .32±.03 .33±.02 .31±.01 .38±.02 .33±.01 .31±.01 .36±.02 .36±.02 .37±.02 .37±.02 .38±.01 .37±.01 .41±.01 .40±.02 .43±.01 .45±.02 .48±.01 .50±.01 .49±.02 .49±.03 .52±.01 .51±.02 .52±.02 .57±.02 .61±.02 .62±.02 .64±.01 .67±.02 .75±.01 .68±.01

.13±.01 .13±.01 .18±.01 .21±.02 .20±.01 .27±.01 .28±.02 .29±.02 .27±.02 .30±.01 .28±.02 .35±.03 .39±.02 .38±.01 .42±.01 .39±.01 .33±.01 .38±.01 .37±.02 .39±.02 .37±.02 .39±.01 .41±.01 .44±.01 .43±.01 .42±.02 .51±.01 .53±.01 .53±.01 .54±.02 .54±.02 .53±.02 .50±.01 .56±.01 .56±.01 .67±.00 .64±.01 .68±.01 .67±.01 .70±.01 .70±.01

.12±.00 .12±.01 .18±.01 .21±.02 .20±.01 .28±.01 .27±.02 .30±.03 .27±.02 .30±.02 .28±.01 .34±.03 .39±.02 .38±.02 .40±.01 .39±.00 .34±.02 .37±.01 .37±.02 .39±.02 .37±.02 .39±.02 .42±.00 .42±.01 .44±.03 .41±.02 .53±.02 .53±.00 .56±.02 .55±.03 .56±.02 .51±.02 .50±.02 .56±.01 .57±.01 .67±.00 .64±.02 .68±.01 .67±.01 .72±.01 .72±.01

.14±.01 .14±.01 .19±.01 .23±.01 .21±.01 .30±.02 .30±.02 .31±.03 .29±.02 .32±.01 .29±.01 .37±.03 .42±.03 .39±.02 .46±.02 .42±.01 .35±.01 .40±.02 .39±.01 .42±.02 .39±.02 .42±.02 .44±.01 .46±.01 .47±.03 .45±.03 .56±.02 .56±.01 .58±.01 .58±.03 .58±.02 .54±.02 .53±.01 .59±.02 .60±.01 .72±.01 .67±.01 .72±.01 .71±.01 .75±.01 .76±.01

.16±.01 .18±.01 .27±.02 .31±.02 .24±.01 .34±.01 .32±.02 .37±.01 .33±.02 .36±.02 .38±.01 .40±.03 .43±.02 .44±.02 .41±.01 .43±.01 .44±.01 .45±.02 .40±.02 .44±.03 .46±.02 .47±.02 .49±.01 .50±.01 .54±.01 .51±.02 .63±.02 .57±.01 .59±.01 .62±.02 .60±.02 .61±.01 .57±.01 .62±.02 .63±.01 .69±.01 .67±.02 .71±.01 .70±.02 .72±.01 .77±.00

.18±.01 .19±.01 .26±.02 .27±.02 .25±.01 .32±.01 .34±.02 .36±.01 .38±.03 .37±.01 .35±.01 .39±.03 .44±.01 .45±.02 .41±.01 .43±.01 .42±.01 .45±.01 .43±.02 .46±.02 .47±.02 .48±.01 .48±.01 .51±.01 .52±.01 .53±.02 .58±.02 .58±.01 .59±.01 .62±.01 .60±.02 .60±.01 .58±.01 .63±.02 .65±.02 .69±.02 .69±.02 .73±.01 .73±.02 .73±.02 .77±.01

.17±.00 .20±.02 .29±.01 .34±.02 .25±.01 .37±.01 .35±.02 .40±.01 .34±.02 .39±.01 .39±.01 .43±.03 .46±.02 .47±.02 .44±.02 .47±.01 .46±.01 .48±.02 .43±.02 .46±.02 .49±.02 .50±.01 .53±.01 .53±.01 .56±.02 .54±.02 .67±.02 .61±.02 .62±.01 .66±.02 .63±.01 .63±.01 .60±.01 .66±.02 .66±.01 .72±.01 .70±.01 .74±.01 .73±.02 .75±.01 .80±.01

Textual Supervision Enhances Geospatial Representations in Vision-Language Models

0.8 0.7 0.6 0.5 0.4

R²

Vision-only Vision-language

0.3 0.2 0.1

Objects Uncategorized Glowing Lights Musicians Flowers Drinks Soil Birds and Insects Food Animals People Closeup Displays Vehicles Interiors Text Children Colorful Outdoors Poles and Columns Religious Monuments Statues Horizon Wall People 2 Banks/ Beaches Cold Places People 1 Roads and Railways Large Gatherings Cars Sports People Outdoors Nature Lakes Piers Cliffs and Mountains Signs Old Buildings Constructions Buildings Streets Landmarks

vit-mae-base vit-mae-large vit-base-patch16-224 vit-large-patch16-224 dinov2-base dinov2-large dinov2-giant clip-vit-base-patch32 webssl-dino7b-full8b-224 Qwen2.5-VL-3B-Instruct gemma-3-4b-it Qwen2.5-VL-7B-Instruct gemma-3-12b-it gemma-3-4b-pt gemma-3-12b-pt clip-vit-large-patch14 metaclip-h14-fullcc2.5b llava-1.5-7b-hf

Figure 18. Model performance measured by coefficient of determination R2 across all models after the removal of images containing country and coordinate-related captions. The x-axis shows image clusters based on the YFCC100M dataset, and y-axis lists the models compared. Higher R2 values (darker colors in the heatmap) indicate better geolocation-prediction accuracy across clusters.

Record · ID 266227 · SHA-256 8d2362488ec10e91
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.