arXiv CSopen access

CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities

CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities Junyuan Liu1 Xinglei Wang1 Zichao Zeng1,2 Jiazhuang Feng1 Quan Qin1,3 Ilya Ilyankou1 Guangsheng Dong1,4 Tao Cheng1,† 1 SpaceTimeLab, University College London, UK

arXiv:2605.26036v1 [cs.AI] 25 May 2026

2 3DIMPact, University College London, UK 3 School of Resource and Environmental Sciences, Wuhan University, China 4 State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing,

Wuhan University, China † Corresponding author: [email protected]

Abstract Urban representation learning encodes complex urban environments into generalpurpose embeddings for diverse downstream tasks and emerging urban foundation models. However, current evaluations are limited, typically focusing on one or two cities and tasks and relying on random splits that introduce spatial leakage, leading to inflated performance and weak support for cross-location generalization and fair comparison. To address this, we propose CityRep, a unified benchmark that evaluates urban representations across data modalities, cities, and tasks using spatially structured splits. CityRep consists of three key components: (1) a spatial unit-agnostic evaluation framework that supports heterogeneous urban representations through a standardized alignment module; (2) a unified evaluation protocol using block-based spatial splits to mitigate spatial leakage and enable rigorous model comparison; and (3) an extensible multi-city, multi-task benchmark suite spanning 8 cities and 8 tasks across regression, classification, and distribution prediction. We evaluate 11 representative urban representation models. Results show that performance is highly sensitive to the split protocol, with random splits inflating scores and altering model rankings. We also observe substantial variability across cities and tasks, underscoring the need for generalization-aware evaluation. CityRep is released as a reproducible benchmark with datasets, evaluation pipelines, and diagnostic tools to facilitate fair comparison and support future research in urban representation learning towards urban foundation models. Code: https://github.com/inwind0212/CityRep.

1

Introduction

Urban representation learning seeks to turn heterogeneous observations of cities into reusable spatial embeddings. Recent models draw on remote sensing [1–3], street-view imagery [4, 5], and points of interest [6–10] to encode geographic entities, regions, or locations. This motivation parallels the broader shift toward foundation models: representations learned from broad urban data are expected to transfer across tasks, locations, and domains. Yet the evaluation of these representations remains much less unified. Reported results are often tied to a particular model interface or a particular format of downstream task, making it difficult to assess whether an embedding is broadly useful or only effective under a narrow evaluation setup. Existing evaluations are also too narrow to support claims about general-purpose urban representations. Many studies evaluate on one or two cities, a small number of tasks, or a single label type. Such experiments are valuable for demonstrating a specific application, but they do not reveal whether a Preprint.

Figure 1: Framework of CityRep Benchmark. CityRep standardizes the evaluation of heterogeneous urban representations by aligning different spatial supports to common downstream task units, evaluating them across eight cities and eight tasks, and using spatial block splits to mitigate leakage.

representation transfers across urban contexts or across qualitatively different prediction problems. This limitation is especially important for the emerging urban foundation models, whose value depends on broad reuse. A benchmark should therefore cover multiple cities, multiple urban domains, and multiple task types, while remaining extensible to incorporate new cities, tasks, and models. Finally, evaluation must explicitly account for spatial dependence. In urban representation learning, we expect models to leverage information from observed regions to make predictions in unseen areas, rather than merely interpolate among nearby samples. While this challenge has been well recognized in spatial validation studies [11–13], most existing urban representation models are still evaluated using random splits that ignore spatial structure. We address this gap by establishing a unified spatially structured evaluation protocol. Our benchmark provides empirical evidence that random splits can substantially inflate performance and lead to over-optimistic conclusions about model generalization. We introduce CityRep, a unified benchmark for urban representations across modalities, tasks, and cities, as shown in Figure 1. The central goal of CityRep is to move urban representation evaluation beyond narrow, single-setting comparisons. Rather than assessing an embedding on one city, one task, or one type of urban label, CityRep asks whether a representation remains useful across a sufficiently broad set of urban phenomena and geographic contexts. To operationalize this goal, we construct downstream evaluation data from four key dimensions of urban systems: morphology, demographics, economy, and environment. These dimensions are instantiated as eight tasks across eight cities, covering classification, regression, and distribution prediction. CityRep pairs this broad task suite with a common evaluation protocol for heterogeneous representations. Models based on rasters, regions, entities, or coordinates can be evaluated under the same downstream interface, while spatially structured splits are used to reduce leakage between nearby training and test samples. CityRep therefore makes it possible to examine not only average performance, but also how representation quality changes across urban contexts, task domains, label types, and split protocols. This provides a basis for assessing whether urban representations are genuinely general-purpose, rather than effective only under narrow evaluation settings. In summary, our contributions are: • We introduce CityRep, a unified and extensible benchmark for urban representation learning that supports heterogeneous representation types across data modalities, cities, and downstream urban tasks. 2

• We design a spatially structured evaluation methodology, including spatial-unit alignment and block-based spatial splits, to enable fair comparison across heterogeneous urban representations while mitigating spatial leakage. • We conduct a large-scale empirical study of eleven representative urban and geospatial representation models across eight cities and eight tasks, showing that benchmark conclusions are highly sensitive to the evaluation protocol, task domain, and urban context. • We publicly release datasets, evaluation pipelines, processed benchmarks, model manifests, and diagnostic tools to support reproducible research on urban representation learning and urban foundation models.

2

Related Work

Urban Representation Learning Existing urban representation learning methods are highly heterogeneous, differing in the data they utilise, the spatial units they operate upon, and the urban signals they encode. Following geographic information systems (GIS) taxonomy, this heterogeneity is largely shaped by whether models ingest vector data, such as points, polylines, and polygons, or raster data, such as satellite and street-view imagery. Vector-based methods often rely on POI data but produce different outputs: category embedding methods, including Place2Vec [6], POI2Vec [14], and SPPE [7], capture spatial co-occurrence patterns of POI types and require aggregation to represent urban spaces, whereas entity embedding models such as Urban2Vec [4], HGI [15], and CityFM [9] directly encode regions, buildings, or roads. Raster-based methods, by contrast, naturally produce grid-cell embeddings, with earth-observation foundation models such as AlphaEarth Foundation [3] and TESSERA [2] enabling dense large-scale representations, and AETHER further incorporating POI semantics into raster foundations [16]. A related line of coordinate-based encoders, including Space2Vec [17], SatCLIP [1], and CaLLiPer [8], learns representations for continuous locations from POIs, imagery, or language supervision. Consequently, the resulting embeddings operate over disparate spatial supports (e.g., regions, H3 cells [18], raster grids, and coordinates) and capture varying urban information. This creates a central evaluation challenge: heterogeneous methods cannot be fairly compared without spatial alignment to standard task units, and evaluation on only a few downstream tasks is insufficient for representations capturing diverse urban signals. Geospatial Benchmarks and Spatial Evaluation Urban representation learning can be viewed as a fine-grained, city-focused branch of geospatial representation learning. Existing geospatial benchmarks have largely started from image- or raster-centered settings. TorchGeo provides reusable infrastructure for geospatial data loading, sampling, and model development [19], while GEO-Bench, SatlasPretrain, and PANGAEA standardize Earth-observation evaluation and pretraining across tasks, sensors, resolutions, regions, and temporal settings [20–22]. Recent benchmarks move closer to spatial representation learning. TorchSpatial evaluates general-purpose location encoders [23], OBSR evaluates geospatial embedders on regional and trajectory tasks [24], and MoRA introduces human-centric social and economic prediction tasks based on mobility-centered representations [25]. However, they still provide limited evidence on whether representations capture fine-grained intraurban structure and functions. Spatial evaluation is also critical. Prior work shows that random splits can overestimate performance under spatial dependence and recommends spatially structured validation for assessing transfer to unseen areas [11–13, 26]. These gaps motivate a unified multi-city benchmark for heterogeneous urban representations under spatially robust evaluation protocols.

3

CityRep Framework and Benchmark

CityRep aims to make urban representations comparable across spatial units, tasks, and cities. Figure 1 provides an overview of the CityRep benchmark. Urban representations arise in diverse forms, including raster-based embeddings, region-level features, POI or entity representations, and coordinate-based encoders, while downstream labels are defined over heterogeneous spatial units. To enable comparison across such settings, CityRep first aligns each representation to common task units across multiple cities and domains through standardized spatial alignment strategies, constructing unified task datasets. These aligned features are then evaluated under spatially structured split protocols, ensuring that performance reflects generalization to unseen areas rather than interpolation 3

among nearby samples. Finally, results are aggregated across cities using task-appropriate metrics, enabling consistent and robust comparison of different urban representation models. 3.1

Problem Definition

Urban representation learning aims to compress heterogeneous observations of a city into reusable spatial embeddings. Let Dm,c denote the input data used by representation model m in city c, such as satellite imagery, street-view imagery, points of interest, road networks, or geographic coordinates. The model transforms these observations into a city representation Em,c = fm (Dm,c ), (1) where Em,c may be defined on a raster grid, a set of regions or cells, a collection of POIs or spatial entities, or a continuous coordinate domain. CityRep does not prescribe how fm is trained. Instead, it evaluates whether the resulting representation captures transferable urban information that is useful for downstream prediction across multiple task domains. n

c,t For each city c and downstream task t, CityRep defines a set of task units Uc,t = {ui }i=1 and labels yc,t . Because the native unit of Em,c generally differs from the unit of Uc,t , the central benchmark operation is spatial alignment:

Xm,c,t = A(Em,c , Uc,t ), Xm,c,t ∈ Rnc,t ×dm , (2) where A(·) maps the native representation to the task units and dm is the embedding dimension. Each row of Xm,c,t is the feature vector assigned to one task unit and is paired with the corresponding label in yc,t . Given the aligned features, CityRep evaluates each representation with a fixed downstream predictor ŷc,t = gθ,t (Xm,c,t ), (3) where the prediction head is chosen according to the task type: regression, classification, or distribution prediction. This formulation separates representation learning from benchmark evaluation. Models may differ in input data modality, pretraining objective, and native spatial support, but they are compared by the same question: after alignment to the downstream task units, how much task-relevant urban information does the representation provide? 3.2

Spatial Alignment

Spatial alignment is the mechanism that makes heterogeneous urban representations comparable. Representation models and downstream tasks are often defined on different spatial supports, such as raster cells, regions, entities, or coordinates. CityRep therefore treats alignment as a spatial matching problem: for each downstream task unit ui , the benchmark assigns a representation vector that corresponds to the same location or spatial area. The goal is not to force all models and tasks onto a single universal grid, but to preserve each task’s native evaluation unit while mapping every representation to that unit in a consistent way. CityRep implements this principle according to the spatial relationship between the representation unit and the downstream task unit. For raster or region-level embeddings, if the representation units are finer than the task unit, CityRep aggregates the embeddings within the task unit. If the representation unit is coarser than the task unit, all task units covered by the same representation unit share its embedding. When the spatial supports are directly compatible, alignment reduces to raster sampling, cell lookup, or spatial join. For entity-level embeddings, such as POI or map-entity representations, CityRep first aggregates entities to an intermediate support, such as H3 cells, and then applies the same region-matching rules. This avoids directly aggregating sparse and unevenly distributed entities to every downstream task unit, which can otherwise produce many missing features and degrade downstream evaluation; an ablation supporting this H3-first design is provided in Appendix C.3. For coordinate encoders, no stored spatial support is required: CityRep queries the encoder at a representative coordinate of each task unit, such as a point location, raster-cell center, or polygon representative point, and creates an embedding from these sample points. 3.3

Spatial Split

CityRep uses spatial splitting to define the generalization target of the benchmark. Let Uc,t be the task units for city c and task t. Instead of drawing train and test samples independently from Uc,t , we 4

J

c,t first partition the spatial extent of the task into a set of non-overlapping blocks Bc,t = {Bj }j=1 . Each task unit is assigned to one block by spatial containment or by the location of its representative point:

b(ui ) ∈ Bc,t .

(4)

The train, validation, and test sets are then formed by assigning blocks, not individual task units, to disjoint subsets: train val test Bc,t = Bc,t,k ∪ Bc,t,k ∪ Bc,t,k , (5) where k indexes the random seed and the three block sets are mutually disjoint. The corresponding task-unit split is induced by block membership: s s Uc,t,k = {ui ∈ Uc,t : b(ui ) ∈ Bc,t,k },

s ∈ {train, val, test}.

(6)

This formulation makes spatial separation part of the evaluation protocol. Test samples are held out together with spatially proximate samples within the same block, reducing the chance that performance is driven primarily by local interpolation from adjacent training points. The split is task-specific because different tasks may have different spatial extents, valid masks, and label supports, but it is model-invariant: all models evaluated on the same city–task pair use the same block partition and the same seed-specific block assignment. In the current benchmark instantiation, we use a 10 × 10 spatial block partition for the main results and report a block-granularity sensitivity analysis in Appendix D.3. Detailed split configurations, visualization examples, and cross-seed test-block statistics are provided in Appendix C.1. 3.4

Tasks and Dataset

CityRep is built around eight downstream tasks that reflect different dimensions of urban systems. The goal is to test whether an urban representation captures information that transfers beyond a single visual pattern or geographic prior. The tasks span regression, classification, and distribution prediction. Downstream Tasks. CityRep includes eight downstream tasks organized into four urban domains: Morphology (♠), Demographics (♡), Economy (♢), and Environment (♣). Details of the downstream task datasets and raw data sources are provided in Appendix B.1. - Land-use classification (LUC)♠ . This task uses city-specific zoning or land-use datasets from official or public planning sources [27–33]. It evaluates whether representations capture semantic urban functions such as residential, commercial, industrial, transportation, green space, institutional, utilities, water bodies, and mixed-use areas. - Road-density regression (RDE)♠ . This task uses OpenStreetMap road-network data [34]. It evaluates whether representations capture physical street structure and connectivity. - Population regression (POP)♡ . This task uses WorldPop gridded population datasets [35]. It evaluates whether representations capture spatial variation in population intensity. - Age-distribution prediction (AGE)♡ . This task uses WorldPop age–sex datasets [36]. It evaluates whether representations capture demographic composition across age groups. - Gross Domestic Product regression (GDP)♢ . This task uses gridded GDP datasets from Kummu et al. [37]. It evaluates whether representations capture spatial variation in economic output. - Nighttime lights regression (NTL)♢ . This task uses the VIIRS Nighttime Lights Annual V2.2 product [38, 39]. It evaluates whether representations capture spatial patterns of human activity, electrification, commercial intensity, and infrastructure use visible through nighttime illumination. - PM2.5 regression♣ . This task uses SEDAC/CIESIN annual PM2.5 concentration datasets [40]. It evaluates whether representations capture fine particulate pollution exposure. - Land-surface-temperature regression (LST)♣ . This task uses MODIS/Terra MOD11A2 daytime land-surface-temperature datasets [41]. It evaluates whether representations capture surface thermal conditions related to land cover, density, vegetation, and impervious surface. Cities and extensibility. Most sources used in CityRep are global or near-global, including WorldPop, gridded GDP, nighttime lights, PM2.5 , MODIS LST, and OpenStreetMap. As a result, adding a new city mainly requires defining the boundary, extracting the same source layers, and running the standard task construction and alignment pipeline. We instantiate the benchmark on London, New York, Singapore, Sydney, Mumbai, Nairobi, Jakarta, and Cape Town, covering developed and developing urban contexts across Europe, North America, Asia, Africa, and Australia. To further demonstrate 5

Table 1: Number of prediction units for each downstream task across eight cities. Task ♠ Morphology Land use Road density ♡ Demographics Population Age distribution ♢ Economy Gross Domestic Product Nighttime lights ♣ Environment PM2.5 Land surface temperature

London

New York

Singapore

Sydney

Mumbai

Nairobi

Jakarta

Cape Town

100,000 297,314

100,000 187,028

100,000 83,393

100,000 612,027

100,000 58,778

100,000 81,389

100,000 75,724

100,000 343,593

266,183 234,428

114,276 87,736

64,172 32,990

416,254 265,815

43,522 43,455

61,677 55,521

74,885 74,096

167,771 136,093

2,977 2,977

1,812 1,867

806 839

6,121 6,119

575 583

817 818

756 756

3,420 3,434

2,072 295,364

1,038 112,433

551 70,009

4,231 592,033

407 55,394

569 81,389

527 75,117

2,380 339,759

extensibility, Appendix D.4 extends the evaluation of global remote-sensing representations to 26 cities using the same benchmark pipeline. Table 1 reports the number of downstream task units for each city and task. The counts vary because cities differ in spatial extent, valid masks, source resolution, and task support. Dense raster-derived tasks such as population, road density, and land-surface temperature contain many units, while coarser grids such as GDP, NTL, and PM2.5 contain fewer. 3.5

Selected Baselines

We evaluate eleven representative urban and geospatial representation models. The reproduction pipeline uses Foursquare POIs [42], Mapillary street-view imagery [43], OpenStreetMap entities [34], and public remote-sensing embedding products [2, 3] as the main raw data sources. - PE [44] Position encoding (PE) functions encode multi-scale location signals, serving as simple urban representations based entirely on spatial information. We select SphereC [45] as a representative example to illustrate the performance of these methods. - Place2Vec [6] learns place representations from POI context. We reproduce it with Foursquare POIs and aggregate the learned embeddings to H3 cells. - Space2Vec [17] represents locations through a coordinate encoder. We train it with POI category supervision and query the encoder directly at raster-cell centers or land-use point coordinates. - CaLLiPer [8] is a coordinate-based urban representation pretrained via language supervision from POI textual descriptions. Its embeddings are exported through the same interface as Space2Vec. - CityFM [9] learns urban representations from map entity information. We reproduce it using OpenStreetMap entities and export embeddings to H3 cells. - Urban2Vec [4] combines street-view and POIs for learning region representations. We construct its inputs from Mapillary imagery and Foursquare POIs, and root the embeddings in H3 cells. - MuseCL [5] is a multimodal urban representation model. Since consistent mobility data are unavailable across all benchmark cities, we implement a CityRep-compatible variant that replaces the mobility branch with Foursquare POI semantics while retaining the street-view, remote-sensing, and semantic fusion components. - SatCLIP [1] is a pretrained geographic coordinate encoder. We use the model checkpoint and query it directly at downstream task locations. Although not intended for city-level representation learning, it is included for being a representative imagery-based coordinate embedding method. - TESSERA [2] provides pretrained remote-sensing embedding rasters. We use the released embeddings as fixed raster representations and align them to task units by raster sampling. - AlphaEarth [3] is a large-scale pretrained geospatial embedding product. We crop or sample its released raster embeddings for each city and task. - AETHER [16] is a POI-guided alignment framework for pretrained imagery embeddings. We reproduce it by aligning AlphaEarth embedding inputs with Foursquare POI semantics. 3.6

Evaluation

CityRep evaluates each representation after spatial alignment to the downstream task units. For a model m, city c, and task t, the aligned feature matrix Xm,c,t is paired with the task labels yc,t and used to train a lightweight task head. To make model comparison depend primarily on the 6

representation rather than on downstream model engineering, CityRep uses the same predictor family and training protocol for all representation models. After alignment, each representation is evaluated by a task-specific prediction head: ŷc,t = gθ,t (Xm,c,t ), (7) where the output layer and loss are selected according to the task type. Evaluation metrics. CityRep uses nine task-appropriate metrics across the three prediction types. For regression tasks, including road density, population, GDP, NTL, PM2.5 , and land surface temperature, we report R2 , mean absolute error (MAE), and root mean squared error (RMSE), with R2 used as the primary metric. For land-use classification, we report macro F1, macro recall, and macro precision, with macro F1 used as the primary metric because it gives equal weight to each class under imbalanced labels. For age-distribution prediction, we report KL divergence, Chebyshev distance, and L1 distance between the predicted and target distributions, with KL divergence used as the primary metric. Higher values are better for R2 , F1, recall, and precision, while lower values are better for MAE, RMSE, KL divergence, Chebyshev distance, and L1 distance. Formal metric definitions are provided in Appendix C.4. Training and aggregation. All downstream predictors use the same MLP architecture and training protocol across models and tasks. For each model–task–city setting, we run five spatial split seeds, {42, 24, 7, 0, 100}, and average the primary test metric over seeds to obtain a city-level score. Table 2 reports, for each model and task, the mean of city-level scores across cities as Avg., together with the cross-city standard deviation as C Std. Since different tasks use different primary metrics and metric scales, CityRep reports raw task metrics in the main result columns and uses Mean City Rank as a rank-based diagnostic summary for comparing models across tasks and cities. Lower values indicate better overall rank. Additional details on training, aggregation, and rank computation are provided in Appendix C.5.

4

Experiments

We evaluate eleven representation models on eight tasks across eight cities. The experiments examine three questions: which representations are the best under a unified protocol, whether performance is stable across cities and tasks, and how much random splits overestimate generalization compared with spatial splits. 4.1

Main Performance

Table 2 shows that large-scale pretrained geospatial representations achieve the strongest overall transfer. AETHER, AlphaEarth, and TESSERA obtain the best mean city ranks, and they dominate most tasks. This suggests that broad spatial coverage and large-scale pretraining are highly valuable when a representation is expected to support heterogeneous urban prediction tasks. We further report linear-probe results in Appendix D.2, which show broadly comparable model rankings under a lower-capacity downstream evaluator. However, the ranking is not determined by pretraining scale alone. Several specialized or simple representations remain competitive on specific tasks. CityFM achieves strong performance on LST, indicating that map entities can provide useful signals for built intensity and thermal conditions. CaLLiPer is also competitive on AGE. Even the PE baseline performs strongly on the spatially smoother environmental task PM2.5 , suggesting that location-only signals can be informative when the target exhibits broad spatial gradients. These results show that smaller, simpler, or more targeted representations can still be effective when their encoded signals align well with the downstream phenomenon. Another observation is that input modality alone is insufficient to explain performance across tasks. More modalities do not necessarily translate into stronger performance. At the same time, models with similar modalities exhibit distinct task-specific strengths. Among raster or raster-enhanced embeddings, AlphaEarth performs best on LUC, POP, NTL, and LST, TESSERA leads RDE, and AETHER is strongest on GDP, PM2.5 , and AGE. POI- and entity-based models also differ substantially, with CaLLiPer outperforming Place2Vec and Space2Vec overall and CityFM remaining 7

Table 2: Main benchmark results on CityRep. For each task, Avg. reports the mean primary metric across 8 cities and 5 random seeds under the spatial block split. C Std. reports the cross-city standard deviation of city-level performance. Type indicates the main data sources used by each representation: L = location, P = POI, R = remote sensing, S = street-view imagery, and O = other urban/map data. Mean City Rank is computed from the unrounded city-level rankings across all tasks and cities (lower is better). Best, second-best, and third-best results in the Avg. columns are in bold, underlined, and wavy-underlined, respectively. *For AGE, lower KL divergence indicates better performance, and ::::::::::::: results are reported only for the four cities with the most reliable age–sex source coverage. ♠ Morphology Model

Type

LUC

♡ Demographics

RDE

POP

♢ Economy

AGE∗

GDP

♣ Environment

NTL

PM2.5

Overall

LST

Rank

Avg. Avg. Avg. Avg. Avg. Avg. Avg. Avg. C C C C C C C C Mean (F1 ↑) Std. (R2 ↑) Std. (R2 ↑) Std. (KL ↓) Std. (R2 ↑) Std. (R2 ↑) Std. (R2 ↑) Std. (R2 ↑) Std. City Rank PE Place2Vec Space2Vec CaLLiPer CityFM Urban2Vec MuseCL SatCLIP TESSERA AlphaEarth AETHER

L P LP LP PO PRS PRS LR R R PR

Mean over models –

0.149 0.035 0.168 0.025 0.166 0.046 0.189 0.059 0.165 0.043 0.143 0.016 0.153 0.017 0.107 0.031 0.322 0.065 ::: 0.346 0.064 0.343 0.070

0.144 0.217 0.118 0.161 0.187 0.227 0.244 0.072 0.631 0.601 0.566 :::

0.080 0.093 0.060 0.061 0.083 0.100 0.099 0.051 0.085 0.088 0.095

0.176 0.288 0.218 0.298 0.228 0.278 0.306 0.067 0.675 ::: 0.695 0.694

0.088 0.081 0.116 0.131 0.080 0.062 0.078 0.077 0.110 0.118 0.101

0.040 0.043 0.041 0.038 0.041 0.044 0.046 0.043 0.040 0.039 ::: 0.037

0.022 0.022 0.022 0.022 0.024 0.022 0.022 0.022 0.019 0.017 0.019

0.132 0.191 0.205 0.263 0.199 0.316 0.338 -0.003 0.581 ::: 0.612 0.750

0.120 0.134 0.155 0.142 0.083 0.132 0.165 0.084 0.187 0.158 0.125

0.237 0.303 0.305 0.364 0.369 0.319 0.336 0.076 0.595 0.650 0.537 :::

0.269 0.159 0.145 0.172 0.206 0.163 0.153 0.204 0.166 0.132 0.120

0.492 0.063 0.236 0.328 0.348 0.068 0.039 0.032 0.396 ::: 0.390 0.568

0.281 0.090 0.222 0.224 0.254 0.105 0.096 0.121 0.158 0.179 0.154

0.378 0.138 0.320 0.440 0.496 ::: 0.137 0.159 0.184 0.465 0.509 0.500

0.198 0.097 0.186 0.178 0.166 0.082 0.117 0.086 0.158 0.143 0.157

0.205 0.043 0.288 0.081 0.357 0.095 0.041 0.021 0.326 0.135 0.372 0.172 0.269 0.171 0.339 0.143

6.828 7.313 7.438 5.359 6.250 7.469 7.422 9.984 3.141 ::: 2.625 2.172 –

Figure 2: GDP and LST prediction performance across cities. The figure reports R2 , MAE, and RMSE for each model and city.

competitive on LST. These results indicate that benchmark performance is shaped not only by input modality, but also by model architecture, pretraining objective, spatial support, and alignment strategy. 4.2

Performance across Cities and Tasks

The final row of Table 2 provides a complementary task-level view by averaging each column over models. It reveals that different urban phenomena are captured by current representations to different degrees. The cross-city standard deviations also differ substantially across tasks. NTL, PM2.5 , GDP, and LST exhibit larger average C-Std. values, indicating stronger city-dependent variation in model transfer, whereas RDE and POP are comparatively more stable. Figures 2 and 3 show that benchmark difficulty varies strongly across both cities and tasks. Figure 2 provides two representative examples, GDP and LST, while the full city-task results with error bars for all tasks and metrics are provided in Appendix D.1. Model rankings and the relative performance 8

Figure 4: Raw primary-metric changes under random splits compared with spatial splits. Each cell reports random split minus spatial split for the primary evaluation metric. Positive values indicate higher raw metric values.

Figure 3: City-level difficulty profiles across tasks. Lower mean city rank indicates that a city is relatively easier to predict for a given task.

gaps among models change across cities. Thus, results from a single city would give an incomplete picture of model quality. Figure 3 further shows that city-level difficulty is highly task-dependent. The city difficulty profiles cross substantially across tasks, indicating that no city is consistently easy or difficult for all prediction problems. For example, a city that is relatively easy for socioeconomic or built-environment prediction can be difficult for environmental prediction. This suggests that city-level generalization depends on the interaction between the target phenomenon, local urban structure, data quality, and the representation being evaluated. We examine whether these patterns persist beyond the eight cities in an exploratory 26-city extension in Appendix D.4. We further analyze possible factors behind city–task difficulty variation in Appendix E. Therefore, evaluating urban representations on a single city or a single task can obscure important differences in model robustness and transferability. 4.3

The Effect of Spatial Splits

Figure 4 compares random splits with the spatial block split used in the main benchmark. Random splits consistently produce higher apparent performance for most tasks and models. This confirms that random splitting is overly optimistic in urban prediction because nearby training and test samples can share strong spatial autocorrelation. The inflation is largest for densely sampled tasks such as population and LST, where random splits are more likely to place test samples close to training samples. It is smaller for coarser tasks such as GDP. Coordinate-based models such as CaLLiPer and Space2Vec show large gains under random splitting on dense targets, suggesting that smooth coordinate encoders can exploit local interpolation when spatial separation is not enforced. These patterns demonstrate that the effect of split protocol depends on both task resolution and model inductive bias. This result supports the need for spatially structured evaluation. Random splits measure interpolation within observed urban areas, but they do not reliably measure transfer to unseen areas. CityRep therefore uses spatial block splits as the main protocol and treats random splits as a diagnostic for spatial leakage.

5

Conclusion

We presented CityRep, a unified benchmark for evaluating urban representations across heterogeneous spatial supports, varying data modalities, diverse cities, and comprehensive downstream tasks. Our results highlight three key findings: multi-city and multi-task evaluation is necessary because model performance varies substantially across urban contexts and target domains; spatially structured splits are essential because random splits can overestimate generalization through spatial leakage; and future urban representation learning could move toward larger-scale pretrained models that better integrate complementary modalities such as remote sensing, POIs, street-view, map entities, and locations. By making these comparisons systematic and reproducible, CityRep provides a foundation for measuring progress toward general-purpose urban representations and for developing urban foundation models. 9

References [1] Konstantin Klemmer, Esther Rolf, Caleb Robinson, Lester Mackey, and Marc Rußwurm. Satclip: global, general-purpose location embeddings with satellite imagery. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’25/IAAI’25/EAAI’25. AAAI Press, 2025. [2] Zhengpeng Feng, Clement Atzberger, Sadiq Jaffer, Jovana Knezevic, Silja Sormunen, Robin Young, Madeline C Lisaius, Markus Immitzer, Toby Jackson, James Ball, et al. Tessera: Temporal embeddings of surface spectra for earth representation and analysis. arXiv preprint arXiv:2506.20380, 2025. [3] Christopher F Brown, Michal R Kazmierski, Valerie J Pasquarella, William J Rucklidge, Masha Samsikova, Chenhui Zhang, Evan Shelhamer, Estefania Lahera, Olivia Wiles, Simon Ilyushchenko, et al. Alphaearth foundations: An embedding field model for accurate and efficient global mapping from sparse label data. arXiv preprint arXiv:2507.22291, 2025. [4] Zhecheng Wang, Haoyuan Li, and Ram Rajagopal. Urban2vec: Incorporating street view imagery and pois for multi-modal urban neighborhood embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1013–1020, 2020. [5] Xixian Yong and Xiao Zhou. Musecl: predicting urban socioeconomic indicators via multi-semantic contrastive learning. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI ’24, 2024. [6] Bo Yan, Krzysztof Janowicz, Gengchen Mai, and Song Gao. From itdl to place2vec: Reasoning about place type similarity and relatedness by learning embeddings from augmented spatial contexts. In Proceedings of the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, SIGSPATIAL ’17, New York, NY, USA, 2017. Association for Computing Machinery. [7] Weiming Huang, Lizhen Cui, Meng Chen, Daokun Zhang, and Yao Yao. Estimating urban functional distributions with semantics preserved poi embedding. International Journal of Geographical Information Science, 36(10):1905–1930, 2022. [8] Xinglei Wang, Tao Cheng, Stephen Law, Zichao Zeng, Lu Yin, and Junyuan Liu. Multi-modal contrastive learning of urban space representations from poi data. Computers, Environment and Urban Systems, 120:102299, 2025. [9] Pasquale Balsebre, Weiming Huang, Gao Cong, and Yi Li. City foundation models for learning general purpose representations from openstreetmap. In Proceedings of the 33rd ACM international conference on information and knowledge management, pages 87–97, 2024. [10] Junyuan Liu, Xinglei Wang, and Tao Cheng. Enriching location representation with detailed semantic information. In 13th International Conference on Geographic Information Science (GIScience 2025), volume 346 of Leibniz International Proceedings in Informatics (LIPIcs), pages 3:1–3:15, 2025. [11] David R. Roberts, Volker Bahn, Simone Ciuti, Mark S. Boyce, Jane Elith, Gurutzeta Guillera-Arroita, Severin Hauenstein, José J. Lahoz-Monfort, Boris Schröder, Wilfried Thuiller, David I. Warton, Brendan A. Wintle, Florian Hartig, and Carsten F. Dormann. Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8):913–929, 2017. [12] Roozbeh Valavi, Jane Elith, José J. Lahoz-Monfort, and Gurutzeta Guillera-Arroita. blockCV: An R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models. Methods in Ecology and Evolution, 10(2):225–232, 2019. [13] Hanna Meyer, Christoph Reudenbach, Stephan Wöllauer, and Thomas Nauss. Importance of spatial predictor variable selection in machine learning applications: Moving from data reproduction to spatial prediction. Ecological Modelling, 411:108815, 2019. [14] Shanshan Feng, Gao Cong, Bo An, and Yeow Meng Chee. Poi2vec: geographical latent representation for predicting future visitors. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, pages 102–108. AAAI Press, 2017. [15] Weiming Huang, Daokun Zhang, Gengchen Mai, Xu Guo, and Lizhen Cui. Learning urban region representations with pois and hierarchical graph infomax. ISPRS Journal of Photogrammetry and Remote Sensing, 196:134–145, 2023.

10

[16] Junyuan Liu, Quan Qin, Guangsheng Dong, Xinglei Wang, Jiazhuang Feng, Zichao Zeng, and Tao Cheng. Beyond alphaearth: toward human-centered spatial representation via poi-guided contrastive learning. arXiv preprint arXiv:2510.09894, 2025. [17] Gengchen Mai, Krzysztof Janowicz, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao. Multi-scale representation learning for spatial feature distributions using grid cells. In International Conference on Learning Representations, 2020. [18] Uber Technologies, Inc. H3 indexes points and shapes into a hexagonal grid, 2026. https://h3geo.org/. Data available under the Apache 2.0 License. Accessed 06/05/2026. [19] Adam J. Stewart, Caleb Robinson, Isaac A. Corley, Anthony Ortiz, Juan M. Lavista Ferres, and Arindam Banerjee. Torchgeo: Deep learning with geospatial data. ACM Transactions on Spatial Algorithms and Systems, 11(4), 2025. Article 15. [20] Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan David Sherwin, Hannah Kerner, Björn Lütjens, Jeremy Irvin, David Dao, Hamed Alemohammad, Alexandre Drouin, Mehmet Gunturkun, Gabriel Huang, David Vazquez, Dava Newman, Yoshua Bengio, Stefano Ermon, and Xiao Xiang Zhu. Geo-bench: toward foundation models for earth monitoring. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. [21] Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdinando, and Aniruddha Kembhavi. Satlaspretrain: A large-scale dataset for remote sensing image understanding. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 16726–16736, 2023. [22] Valerio Marsocci, Yuru Jia, Georges Le Bellier, Dávid Kerekes, Liang Zeng, Sebastian Hafner, Sebastian Gerard, Eric Brune, Ritu Yadav, Ali Shibli, Heng Fang, Yifang Ban, Maarten Vergauwen, Nicolas Audebert, and Andrea Nascetti. Pangaea: Assessing geospatial foundation models capabilities through a global and inclusive benchmark. IEEE Geoscience and Remote Sensing Magazine, 14:245–285, 2026. [23] Nemin Wu, Qian Cao, Zhangyu Wang, Zeping Liu, Yanlin Qi, Jielu Zhang, Joshua Ni, Xiaobai Yao, Hongxu Ma, Lan Mu, Stefano Ermon, Tanuja Ganu, Akshay Nambi, Ni Lao, and Gengchen Mai. Torchspatial: A location encoding framework and benchmark for spatial representation learning. In Advances in Neural Information Processing Systems, volume 37, pages 81437–81460, 2024. [24] Julia Moska, Oleksii Furman, Kacper Kozaczko, Szymon Leszkiewicz, Jakub Polczyk, Piotr Gramacki, and Piotr Szymański. OBSR: Open benchmark for spatial representations. In Proceedings of the 33rd ACM International Conference on Advances in Geographic Information Systems, SIGSPATIAL ’25, 2025. [25] Ya Wen, Jixuan Cai, Qiyao Ma, Linyan Li, Xinhuan Chen, Chris Webster, and Yulun Zhou. MoRA: Mobility as the backbone for geospatial representation learning at scale. In International Conference on Learning Representations, 2026. [26] Pierre Ploton, Frédéric Mortier, Maxime Réjou-Méchain, Nicolas Barbier, Nicolas Picard, Vivien Rossi, Carsten F. Dormann, Guillaume Cornu, Gaëlle Viennois, Nicolas Bayol, Alexei Lyapustin, Sylvie GourletFleury, and Raphaël Pélissier. Spatial validation reveals poor predictive performance of large-scale ecological mapping models. Nature Communications, 11(1):4540, 2020. [27] New York City Department of City Planning. Primary Land Use Tax Lot Output (PLUTO), version 25v4. NYC Open Data / BYTES of the BIG APPLE, 2026. CSV tax-lot records; current version 25v4; last modified 2026-02-20; accessed 2026-05-06. [28] Urban Redevelopment Authority. Master Plan 2019 Land Use layer (2025). data.gov.sg, 2023. Indicative land-use polygon layer; data from Dec. 2019; extracted from the Master Plan 2019 approved on 2019-11-18; last updated 2025-12-05; accessed 2026-05-06. [29] NSW Department of Planning, Housing and Infrastructure. Environmental Planning Instrument – Land Zoning. NSW Planning Portal Open Data, 2022. Vector land-zoning dataset; update frequency weekly; Creative Commons Attribution; accessed 2026-05-06. [30] City of Cape Town. Zoning. Cape Town Open Data Portal feature service, 2019. Zoning / split-zoning feature service; published 2019-02-10; dataset-specific or custom licence; accessed 2026-05-06. [31] Pemerintah Provinsi DKI Jakarta. Peraturan Gubernur Nomor 31 Tahun 2022 tentang Rencana Detail Tata Ruang Wilayah Perencanaan Provinsi Daerah Khusus Ibukota Jakarta. JDIH Provinsi DKI Jakarta, 2022. Peraturan Gubernur No. 31; ditetapkan 2022-06-27; BD Provinsi DKI Jakarta Tahun 2022 Nomor 63003; accessed 2026-05-06.

11

[32] Municipal Corporation of Greater Mumbai. Development Plan for Greater Mumbai 2014–2034: Existing Land Use Maps and Report. Municipal Corporation of Greater Mumbai, 2012. Existing Land Use 2012 source; cited via UDRI/Login Mumbai data-source register; accessed 2026-05-06. [33] World Bank Data Catalog. Nairobi, Kenya Land Use 2010. World Bank Data Catalog, 2010. Land-use shapefile; source: Columbia University’s Center for Sustainable Urban Development; shapefile last updated 2020-06-25; metadata last updated 2023-01-19; Creative Commons Attribution 4.0; accessed 2026-05-06. [34] OpenStreetMap contributors. OpenStreetMap road-network extract for the CityRep benchmark cities. OpenStreetMap database, 2026. Drivable highway ways extracted for London, New York, Singapore, Sydney, Mumbai, Nairobi, Jakarta, and Cape Town; licensed under the Open Data Commons Open Database License (ODbL) 1.0; accessed/extracted 2026-05-06. [35] M. Bondarenko, R. Priyatikanto, N. Tejedor-Garavito, W. Zhang, T. McKeen, A. Cunningham, T. Woods, J. Hilton, D. Cihan, B. Nosatiuk, T. Brinkhoff, A. Tatem, and A. Sorichetta. Constrained estimates of 2015–2030 total number of people per grid square at a resolution of 3 arc (approximately 100m at the equator), R2024B version v1. WorldPop, School of Geography and Environmental Science, University of Southampton, 2025. Global Demographic Data Project, funded by the Bill and Melinda Gates Foundation, grant INV-045237; Creative Commons Attribution 4.0; accessed 2026-05-06. [36] M. Bondarenko, R. Priyatikanto, N. Tejedor-Garavito, W. Zhang, T. McKeen, A. Cunningham, T. Woods, J. Hilton, D. Cihan, B. Nosatiuk, T. Brinkhoff, A. Tatem, and A. Sorichetta. Constrained estimates of 2015–2030 total number of people per grid square broken down by gender and age groupings at a resolution of 3 arc (approximately 100m at the equator), R2024B version v1. WorldPop, School of Geography and Environmental Science, University of Southampton, 2025. Global Demographic Data Project, funded by the Bill and Melinda Gates Foundation, grant INV-045237; Creative Commons Attribution 4.0; accessed 2026-05-06. [37] Matti Kummu, Maria Kosonen, and Sina Masoumzadeh Sayyar. Downscaled gridded global dataset for gross domestic product (GDP) per capita PPP over 1990–2022. Scientific Data, 12:178, 2025. [38] Earth Observation Group, Payne Institute for Public Policy, Colorado School of Mines. VIIRS Nighttime Lights Annual V2.2. https://eogdata.mines.edu/products/vnl/, 2024. Accessed: 2026-05-06. [39] Christopher D. Elvidge, Mikhail Zhizhin, Tilottama Ghosh, Feng-Chi Hsu, and Jay Taneja. Annual time series of global VIIRS nighttime lights derived from monthly averages: 2012 to 2019. Remote Sensing, 13(5):922, 2021. [40] A. van Donkelaar, M. S. Hammer, L. Bindle, M. Brauer, J. R. Brook, M. J. Garay, N. C. Hsu, O. V. Kalashnikova, R. A. Kahn, C. Lee, R. C. Levy, A. Lyapustin, A. M. Sawyer, and R. V. Martin. Global Annual PM2.5 Grids from MODIS, MISR, SeaWiFS and VIIRS Aerosol Optical Depth (AOD), 1998–2022, V5.GL.04, Version 5.04. NASA Socioeconomic Data and Applications Center (SEDAC), 2024. Data set; accessed 2026-05-06. [41] Zhengming Wan, Simon Hook, and Glynn Hulley. MODIS/Terra Land Surface Temperature/Emissivity 8-Day L3 Global 1km SIN Grid V061. NASA Land Processes Distributed Active Archive Center, 2021. Data set; accessed 2026-05-06. [42] Foursquare Labs, Inc. FSQ OS Places. Foursquare Open Source Places data product, 2026. Open POI dataset; Apache License 2.0; preserve Foursquare attribution and NOTICE file; accessed 2026-05-06. [43] Meta Platforms, Inc. and Mapillary. Mapillary Open Street-Level Imagery. Mapillary platform / API, 2026. Street-level imagery and map-data platform; accessed via Mapillary API on 2026-05-06; images are shared under CC BY-SA and individual image/user attribution should be preserved where required. [44] Gengchen Mai, Krzysztof Janowicz, Yingjie Hu, Song Gao, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao. A review of location encoding for geoai: methods and applications. International Journal of Geographical Information Science, 36:639–673, 2022. [45] Gengchen Mai, Yao Xuan, Wenyun Zuo, Yutong He, Jiaming Song, Stefano Ermon, Krzysztof Janowicz, and Ni Lao. Sphere2vec: A general-purpose location representation learning over a spherical surface for large-scale geospatial predictions. ISPRS Journal of Photogrammetry and Remote Sensing, 202:439–462, 2023. [46] American Planning Association. Land based classification standards. https://www.planning.org/ lbcs/, 2000. Accessed 2026-05-05. [47] James R. Anderson, Ernest E. Hardy, John T. Roach, and Richard E. Witmer. A land use and land cover classification system for use with remote sensor data. Professional Paper 964, U.S. Geological Survey, 1976.

12

Appendix Contents A Discussion

14

A.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

A.2 Broader Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

B Details of Downstream Task Data

14

B.1 Raw Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

B.2 Land Use Label Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

C Additional Experiment Setting Details

21

C.1 Spatial Split . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

C.2 Details of Baselines Reproduction . . . . . . . . . . . . . . . . . . . . . . . . . .

25

C.3 Aggregation for Sparse Entity Representations . . . . . . . . . . . . . . . . . . . .

27

C.4 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

C.5 Training protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29

D Comprehensive Results

30

D.1 Results Across Tasks and Cities . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

D.2 Results with Linear Downstream Task Head . . . . . . . . . . . . . . . . . . . . .

35

D.3 Results with 20x20 Spatial Split . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

D.4 Exploratory Extension to 26 Cities . . . . . . . . . . . . . . . . . . . . . . . . . .

36

E Analysis of Performance Heterogeneity Across Tasks and Cities

13

38

A

Discussion

A.1

Limitations

CityRep is designed to make urban representation evaluation broader and more comparable, but several limitations and scope boundaries remain. First, the benchmark is constrained by data quality in both downstream task construction and baseline reproduction. The downstream labels are derived from public geospatial products and city-specific land-use sources, whose uncertainty, update cycles, spatial resolutions, and semantic definitions vary across regions. CityRep should therefore be interpreted as an evaluation of representations under the released data products, rather than as a definitive measurement of the underlying urban phenomena. At the same time, reproducing existing representation models depends on the availability and quality of their required pretraining inputs. Some original models rely on proprietary, city-specific, or unavailable modalities, so public reproductions may not capture every advantage of the original models under their original data environments. Second, spatial splitting remains a methodological choice rather than a settled standard. CityRep uses block-based spatial splits to reduce leakage and evaluate generalization to held-out urban areas, but other partitioning strategies are possible, such as administrative regions, spatial clustering, distance buffers, or task-specific regionalization. The most appropriate split may depend on task resolution, spatial autocorrelation, city morphology, and the intended generalization target. Further research is needed to understand how different spatial validation designs affect conclusions in urban representation learning. Third, CityRep does not fully investigate how multimodal data scarcity shapes global urban embeddings. Urban representations may depend on remote sensing, POIs, streetview imagery, road networks, mobility data, and administrative records, but these modalities are unevenly available across cities and regions. Such an imbalance may affect what representations learn and may introduce geographic or socioeconomic biases into models intended for broad deployment. In addition, the scope of this study is to evaluate representation at the intra-city level; the zero-shot cross-city generalization of future general-purpose urban models remains to be investigated further. A systematic study of how missing or uneven multimodal data influences global urban representation learning remains an important direction for future work. A.2

Broader Impacts

CityRep is intended as an evaluation tool for understanding what urban representations encode and how reliably they transfer across tasks and cities. A standardized benchmark can reduce duplicated preprocessing effort, make model claims more comparable, and encourage evaluation protocols that better reflect spatial generalization. It may also help researchers identify where existing urban representations fail, especially in cities with limited data availability or urban forms underrepresented in current foundation models. At the same time, urban prediction systems can influence planning, resource allocation, environmental assessment, and infrastructure decisions. Benchmark scores should therefore not be treated as evidence that a model is ready for direct policy deployment. Downstream use requires task-specific validation, uncertainty analysis, stakeholder review, and careful consideration of social context. The benchmark uses aggregated spatial labels, public geospatial layers, and derived task units; nevertheless, urban data can still encode socioeconomic inequalities and uneven data coverage. We therefore release CityRep as a research benchmark rather than a decision-making system, and we encourage users to report failures, biases, and city-specific limitations alongside aggregate scores.

B

Details of Downstream Task Data

B.1

Raw Data

The released task data are processed into a common city–task registry, but the source layers retain different spatial supports, semantics, and access terms. Land use is a point-level classification task derived from city-specific zoning or land-use sources and harmonized into a shared taxonomy [27–33]. Because land-use redistribution terms vary by city, CityRep publicly releases harmonized land-use labels only where permitted by the corresponding source terms. Road density is computed from OpenStreetMap road geometries by aggregating drivable road length to the task grid [34]. Population and age distribution are derived from WorldPop products, with population represented as a scalar 14

Table 3: Raw data sources used to construct CityRep downstream tasks. Task abbreviations follow the main text: LUC = land-use classification, RDE = road-density estimation, POP = population prediction, AGE = age-distribution prediction, GDP = gross-domestic-product prediction, NTL = nighttime-lights prediction, PM2.5 = PM2.5 prediction, and LST = land-surface-temperature prediction. Native support describes the source product before CityRep processing; released support describes the spatial support used by the benchmark task files. Years in the raw-source column denote the actual source or CityRep task version used by the released benchmark files. Task

Native support

Released support

License / terms

♠ Morphology City-specific official or public zoning / landLUC use layers [27–33]

Raw source

Vector polygons or parcellevel records

Harmonized sampled point labels, subject to source terms

RDE

Vector road geometries

Road length density on the task grid

Varies by city; CityRep releases labels only where redistribution is permitted by source terms. ODbL 1.0; OpenStreetMap attribution required.

3 arc-second raster, approximately 100m at the equator 3 arc-second rasters by age and sex group

Same task raster support

CC BY 4.0.

Age-bin distribution on the task raster support

CC BY 4.0.

30 arc-second gridded GDP product Annual cloud-free radiance composite at approximately 15 arc-second resolution

City-clipped task grid

CC BY 4.0 for the public Kummu GDP product. Public domain / open EOG product; cite product page and source paper.

0.01-degree raster

City-clipped task grid

OpenStreetMap drivable highway ways extracted for the CityRep 2026 road-density task version [34] ♡ Demographics POP WorldPop constrained population counts, R2024B, 2024 [35] AGE ♢ Economy GDP NTL

WorldPop age–sex structured population counts, R2024B, 2024 [36] Kummu et al. gridded GDP total, 2024 band [37] VIIRS Nighttime Lights Annual V2.2 average masked radiance composite, 2024 [38, 39]

♣ Environment SEDAC/CIESIN Global Annual PM2.5 Grids, PM2.5 V5.GL.04, 2022 [40] LST

MODIS/Terra MOD11A2.061 8-day daytime land surface temperature annual mean, 2024 [41]

WGS84

8-day, 1km MODIS sinusoidal grid

City-clipped nighttimelights task grid

Annual daytime mean on the CityRep task support

CIESIN open data policy; cite DOI and follow SEDAC/NASA use terms. NASA/LP DAAC data are openly shared without restriction; cite DOI.

count and age represented as a distribution over age bins [35, 36]. GDP uses a gridded economic output product [37], nighttime lights use the VIIRS Nighttime Lights Annual V2.2 average masked radiance composite [38, 39], PM2.5 uses global annual concentration grids [40], and LST uses MODIS/Terra daytime land surface temperature aggregated to an annual mean [41]. Figures 5–12 visualize the processed downstream labels for all eight cities. The visualizations are included to show both the diversity of task supports and the differences in city extent. Some tasks are dense and fine grained, such as road density, population, and LST. Others are coarser, such as GDP, nighttime lights, and PM2.5 . CityRep preserves these task-specific supports instead of forcing all labels onto a single global grid, because the alignment module is responsible for mapping each representation to the target task unit. Table 3 summarizes the raw data sources used to construct the eight downstream tasks. Because the benchmark preserves task-specific spatial supports, the “resolution/support” column reports the native or effective support of the released task labels rather than forcing all sources into a single grid.

15

Figure 5: Land-use (LUC) labels across the eight benchmark cities.

Figure 6: Road-density (RDE) labels across the eight benchmark cities.

Figure 7: Population (POP) labels across the eight benchmark cities.

16

Figure 8: Age-distribution (AGE) labels across the eight benchmark cities.

Figure 9: Gross Domestic Product (GDP) labels across the eight benchmark cities.

Figure 10: Nighttime lights (NTL) labels across the eight benchmark cities.

17

Figure 11: PM2.5 labels across the eight benchmark cities.

Figure 12: Land-surface-temperature labels across the eight benchmark cities.

18

B.2

Land Use Label Mapping

We harmonize city-specific land-use labels into twelve benchmark classes: Residential, Mixed Use, Commercial, Industrial, Transportation, Green / Recreation, Institutional / Civic, Utilities, Water, Agriculture / Rural, Vacant / Reserve, and Other. These classes are informed by established land-use classification schemes, including the Land-Based Classification Standards (LBCS) developed by the American Planning Association and partner agencies [46], and the USGS Anderson land-use/landcover classification system [47]. Both schemes distinguish broad urban functions such as residential, commercial, industrial, transportation, utilities, institutional/public uses, recreation, agriculture, and water. We adapt these high-level categories to a compact benchmark taxonomy so that labels from heterogeneous municipal planning systems can be compared across cities. The mapping is city-specific because local planning systems use different category names and different levels of detail. For example, a source category may explicitly identify utilities in one city but group the same function under public facilities in another. We therefore maintain a mapping table for each city and export both a readable mapping table and audit files with source categories, standardized labels, and counts. We used an LLM-assisted procedure to produce the initial harmonization from source labels to the twelve benchmark classes. The resulting mappings were then manually audited by the authors. Ambiguous or inconsistent categories were corrected before final task reconstruction. Categories that were duplicated or semantically equivalent in the source mapping were also deduplicated. The purpose of this review is not to impose a universal planning ontology, but to ensure that the benchmark classes are consistent enough for cross-city evaluation. Table 4 reports the readable version of the city-specific label mapping used in the benchmark. For compactness, the Sydney mapping is omitted from this appendix table; the complete mapping, including Sydney, is provided in the released code. Table 4: City-specific land-use source labels mapping, excluding Sydney. City

Source label

London

London

City

Source label

High density residential with Residential retail and commercial sites

London

Residential

London

London

Medium density residential with high streets and amenities Business parks

Commercial

London

London London

Retail parks Principle Transport

Commercial Transportation

London London

London

London

London

London

Coniferous and undifferenti- Green / Recreated woodland ation Open or heath and moor Green / Recreland ation Wetlands Green / Recreation Inland Water Water

Low density residential with Residential amenities (suburbs and small villages / hamlets) Urban centres - mainly Mixed Use commercial/retail with residential pockets Large complex buildings var- Commercial ious use (travel/recreation/ retail) Industrial areas Industrial Green / RecreCoastal dunes ation Deciduous woodland Green / Recreation Recreational land Green / Recreation Coastal water Water

London

Agriculture - mainly crops

London

Agriculture - mixed use

London

London

Glasshouses

London

Mining and spoil areas

London London

New York Multi-Family Walk-Up Buildings New York Mixed Residential & Commercial Buildings New York Industrial & Manufacturing New York Transportation & Utility

Mapped class

London

Agriculture / Rural Agriculture / Rural Other

New York

Residential

New York

Mixed Use

New York

Industrial Transportation

New York New York

London

Mapped class

Agriculture / Rural Farms Agriculture / Rural Orchards Agriculture / Rural Multi-Family Elevator Build- Residential ings One & Two Family Build- Residential ings Commercial & Office Build- Commercial ings Parking Facilities Transportation Open Space & Outdoor Green / RecreRecreation ation Continued on next page

19

Table 4: City-specific land-use source labels mapped to the common taxonomy, excluding Sydney (continued). City

Source label

Mapped class

City

Source label

New York Public Facilities & Institutions Singapore RESIDENTIAL

Institutional / Civic Residential

New York Vacant Land

Mapped class Vacant / Reserve

Singapore COMMERCIAL & RESI- Mixed Use DENTIAL Singapore RESIDENTIAL / INSTITU- Mixed Use TION Singapore WHITE Mixed Use

Singapore COMMERCIAL / INSTITU- Mixed Use TION Mixed Use Singapore RESIDENTIAL WITH COMMERCIAL AT 1ST STOREY Singapore COMMERCIAL Commercial Singapore BUSINESS 1 Industrial Singapore BUSINESS 2 Industrial Singapore BUSINESS PARK Industrial Singapore LIGHT RAPID TRANSIT Transportation Singapore PORT / AIRPORT Transportation Singapore TRANSPORT FACILITIES Transportation

Singapore Singapore Singapore Singapore Singapore Singapore Singapore

Singapore CEMETERY

Singapore

Singapore Singapore

HOTEL Commercial BUSINESS 1 - WHITE Industrial BUSINESS 2 - WHITE Industrial BUSINESS PARK - WHITE Industrial MASS RAPID TRANSIT Transportation ROAD Transportation BEACH AREA Green / Recreation OPEN SPACE Green / Recreation SPORTS & RECREATION Green / Recreation EDUCATIONAL INSTITU- Institutional / TION Civic PLACE OF WORSHIP Institutional / Civic WATERBODY Water RESERVE SITE Vacant / Reserve

Mumbai Mumbai Mumbai Mumbai Mumbai Mumbai

Residential Urban Villages Informal Market Other Offices Transport Natural Areas

Mumbai

Swimming Pool

Mumbai

Government Office

Mumbai

Medical Amenities

Mumbai

Municipal Office

Mumbai

Communication

Mumbai Mumbai

Town Duty / Octroi Office Primary Activity

Singapore Mumbai Mumbai Mumbai Mumbai Mumbai

SPECIAL USE Slum / Cluster Commercial Activities Municipal Market Industrial Use Cemetery

Mumbai

Open Spaces

Mumbai

Educational Amenities

Mumbai

Law and Order

Mumbai

Municipal Chowkies

Mumbai

Social Amenities

Mumbai Mumbai

Public Utility and Facity Water

Green / Recreation Green / Recreation Institutional / Civic Institutional / Civic Utilities Agriculture / Rural Other Residential Commercial Commercial Industrial Green / Recreation Green / Recreation Institutional / Civic Institutional / Civic Institutional / Civic Institutional / Civic Utilities Water

Mumbai Mumbai Nairobi Nairobi Nairobi Nairobi

Area Under SPA Vacant res_slum mixed CI commercial transportation

Vacant / Reserve Vacant / Reserve Residential Mixed Use Commercial Transportation

Mumbai Mumbai Nairobi Nairobi Nairobi Nairobi

Under Construction Unclassified residential mixed RC industrial open space

Nairobi

open_space

Nairobi

recreational

Nairobi

institutional

Nairobi

water

Nairobi Jakarta Jakarta

no_structures Zona Perumahan Zona Perdagangan dan Jasa

Green / Recreation Institutional / Civic Vacant / Reserve Residential Commercial

Utilities Agriculture / Rural Vacant / Reserve Other Residential Mixed Use Industrial Green / Recreation Green / Recreation Water

Nairobi Jakarta Jakarta

unknown Zona Pariwisata Zona Perkantoran

Other Commercial Commercial

Singapore PARK Singapore CIVIC & COMMUNITY INSTITUTION Singapore HEALTH & MEDICAL CARE Singapore UTILITY Singapore AGRICULTURE

Singapore Singapore Singapore

Residential Residential Commercial Commercial Transportation Green / Recreation Green / Recreation Institutional / Civic Institutional / Civic Institutional / Civic Utilities

Continued on next page

20

Table 4: City-specific land-use source labels mapped to the common taxonomy, excluding Sydney (continued). City

Source label

Mapped class

City

Source label

Mapped class

Jakarta

Industrial

Jakarta

Zona Badan Jalan

Transportation

Jakarta

Zona Kawasan Peruntukan Industri Zona Transportasi

Transportation

Jakarta

Jakarta

Zona Hutan Lindung

Jakarta

Jakarta

Zona Konservasi

Jakarta

Zona Ruang Terbuka Hijau

Jakarta Jakarta

Zona Pertahanan dan Keamanan Zona Badan Air

Green / Recreation Green / Recreation Green / Recreation Institutional / Civic Water

Jakarta

Zona Pertanian

Green / Recreation Zona Hutan produksi Green / Recreation Zona Perlindungan Setempat Green / Recreation Zona Pelayanan Umum Institutional / Civic Utilities Zona Pembangkit Tenaga Listrik Zona Perikanan Agriculture / Rural 0 Other

Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town

General Residential 1 : Group Housing General Residential 3

Residential

General Residential 5

Residential

Agriculture / Rural Residential

Residential 1 : Conventional Residential Housing Mixed Use 1 Mixed Use Mixed Use 3

Mixed Use

General Business 2

Commercial

General Business 4

Commercial

General Business 6

Commercial

Local Business 1 : Interme- Commercial diate Business General Industrial 1 Industrial Risk Industry

Industrial

Transport 2 : Public Road Transportation and Public Parking Open Space 2 : Public Open Green / RecreSpace ation Community 1 : Local Institutional / Civic Utility Utilities Rural None

Agriculture / Rural Other

Jakarta Jakarta Jakarta Jakarta Jakarta Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town Cape Town

C

Additional Experiment Setting Details

C.1

Spatial Split

Zona Ekosistem Mangrove

General Residential 2

Residential

General Residential 4

Residential

General Residential 6

Residential

Residential 2 : Incremental Housing Mixed Use 2

Residential Mixed Use

General Business 1

Commercial

General Business 3

Commercial

General Business 5

Commercial

General Business 7

Commercial

Local Business 2 : Local Business General Industrial 2

Commercial Industrial

Transport 1 : Transport Use

Transportation

Open Space 1 : Environmen- Green / Recretal Conservation ation Open Space 3: Special Open Green / RecreSpace ation Community 2 : Regional Institutional / Civic Agricultural Agriculture / Rural Limited Use Zone Other

The main benchmark uses spatial block splitting because urban labels are strongly spatially autocorrelated. If nearby samples from the same neighborhood are placed into both training and test sets, a downstream predictor can achieve high performance through local interpolation rather than through 21

Figure 13: Example 10-by-10 spatial block split assignments for AETHER under seed 42. Blocks, rather than individual samples, are assigned to train, validation, and test splits. representation transfer. This is especially problematic for dense raster tasks such as population, road density, and land surface temperature, where adjacent cells often share similar values. For each city–task dataset, we construct a 10 × 10 block grid over the valid task extent. Blocks are assigned, rather than individual task units, to train, validation, and test partitions. The test partition contains 20% of the blocks, and 10% of the remaining training blocks are used for validation. We repeat the procedure with five seeds, {42, 24, 7, 0, 100}, and all models evaluated on the same city–task pair use the same split assignments. Figure 13 illustrates the resulting block layout for the main protocol, and Figure 14 shows how often each block is used as test across seeds. We also run a random split with the same ratios and seeds as a diagnostic comparison. Figures 15 and 16 show that random splitting often increases apparent performance and can alter rankings. This motivates our choice to use spatial splits as the main benchmark protocol and to treat random splits as a measure of spatial leakage sensitivity.

22

Figure 14: Test-block frequency across the five spatial split seeds. The figure shows how often each spatial block is assigned to the test partition.

23

Figure 15: Task-level rank changes between random and spatial splits by model and task. Negative values indicate better rank under random splitting.

Figure 16: Overall mean-rank change between random and spatial splits across tasks. Negative values indicate better rank under random splitting.

24

C.2

Details of Baselines Reproduction Table 5: Reproduction and interface details for the eleven CityRep baselines. The table distinguishes released feature products, official checkpoints, and CityRep-compatible reproductions. “Dim.” is the feature dimension used by the downstream evaluator. “Support” is the spatial interface registered in CityRep before alignment to each task. All models are evaluated with the same downstream predictor, spatial split protocol, and city–task files.

Model

Reproduction status

CityRep input data

Dim. / support

Checkpoint, feature, or code source

Remote-sensing raster representations AlphaEarth

Released feature product

Google AlphaEarth / Satellite Embedding annual raster product, 2024.

64-d raster. Native raster sampled at land-use points; task-grid rasters use mean aggregation or coordinate sampling.

TESSERA

Official package / feature export

TESSERA / GeoTessera 2024 remote-sensing embeddings from Sentinel-1/Sentinel-2 temporal-spectral inputs.

128-d raster. Reference-grid raster for point sampling; task-grid rasters for raster tasks.

AETHER

CityRep-trained checkpoint

AlphaEarth 64-d raster embeddings plus Foursquare POI semantic text; POI text encoded with qwen-3 and projected into the shared space.

128-d raster. Native AETHER rasters and task-specific mean rasters registered for all 8 cities and 8 tasks.

Official feature product from Google Earth Engine / Source Cooperative AlphaEarth Foundations COGs; CityRep wrapper code is used for cropping and task-grid export. Implementation note. AlphaEarth is evaluated as an off-the-shelf released feature product. Results reflect the public remote-sensing embedding together with CityRep’s raster alignment rule. Official GeoTessera code/package: https://github.com/ ucam-eo/geotessera; official pretrained embedding export. Implementation note. TESSERA is evaluated as frozen remote-sensing features exported with the official package/API. CityRep standardizes the task-grid aggregation, point sampling, and downstream evaluator. Official AETHER code: https://github.com/ inwind0212/AETHER; CityRep AETHER code and checkpoints released with the benchmark artifacts; Implementation note. AETHER is the CityRep multimodal representation baseline. It combines remote-sensing and POI semantic signals under the same alignment and downstream evaluation protocol as the other baselines. Map, POI, street-view, and region representations CityFM

Mixed official and reproduced

OSM-derived road, polygon, tag, and surface features prepared in CityFM format.

2560-d H3 region table, resolution 8. Task samples are mapped to H3 cells.

Place2Vec

CityRep-compatible reproduction

Foursquare Open Places, second-level POI categories.

128-d H3 region table, resolution 8. POI category embeddings are pooled into H3 cells.

Urban2Vec

CityRep-compatible reproduction

Mapillary street-view imagery plus Foursquare POI categories aggregated to H3 regions.

128-d H3 region table, resolution 8. Variant registered as street-view plus POI.

Official CityFM code: https://github.com/ PasqualeTurin/ CityFM; official weights used where available, with CityRep-compatible training/export for other cities. Implementation note. We use official CityFM weights where available and produce CityFM-compatible embeddings for the remaining benchmark cities with the same registered input interface. The released export uses components that can be prepared consistently across all CityRep cities. Reproduce code according to the paper; CityRep wrapper prepares Foursquare POIs and H3 exports. Implementation note. CityRep uses Foursquare Open Places as the common POI source across cities. POI counts provide the available activity signal for constructing a reproducible Place2Vec-style region representation. Urban2Vec open-source code: https://github. com/wangzhecheng/ urban2vec_; CityRep wrapper adapts inputs to Mapillary, Foursquare, and H3 regions. Implementation note. Urban2Vec is reproduced with the public street-view and POI inputs available in CityRep. The registered embeddings use the union of available modality support so that regions with one available modality can still be evaluated. Continued on next page

25

Model

Reproduction status

CityRep input data

Dim. / support

MuseCL

CityRep-compatible variant

Mapillary street-view branch, AlphaEarth-based remote branch, and Foursquare POI text branch.

128-d H3 region table, resolution 8. Region embeddings are looked up by H3 cell.

Checkpoint, feature, or code source

Official MuseCL code: https://github.com/ XixianYong/MuseCL; CityRep wrapper provides cross-city asset preparation and H3-region export. Implementation note. MuseCL is implemented as a CityRep-compatible multimodal region encoder using modalities that can be registered consistently across the benchmark. The mobility-related activity signal is represented with available POI-type information. Coordinate and POI-supervised location encoders 192-d entity/coordinate CityRep implementation table. Fixed following the Sphere2Vec-sphereC Sphere2Vec-sphereC position encoding with 64 positional basis: frequencies; task samples https://github.com/ are matched by sample_id gengchenmai/ after evaluating at point sphere2vec; no learned coordinates or raster-cell checkpoint. centers. Implementation note. PE is a non-learned location-only baseline. It is included to measure how much downstream performance can be explained by deterministic coordinate encoding alone under the same spatial split and downstream evaluator. PE

Deterministic location baseline

Longitude–latitude coordinates at each CityRep task unit.

CaLLiPer

CityRep-compatible reproduction

Foursquare POI coordinate–text pairs; POI descriptions include names and categories.

128-d coordinate encoder. Land-use uses point-table inference; raster tasks query the encoder at task pixel centers.

Space2Vec

CityRep-compatible reproduction

Foursquare POI coordinates and second-level category labels.

128-d coordinate encoder. Land-use uses point-table inference; raster tasks query the encoder at task pixel centers.

SatCLIP

Official checkpoint

Longitude–latitude coordinates only at CityRep task locations.

Official CaLLiPer code: https://github.com/ xlwang233/CaLLiPer; CityRep wrapper retrains/exports coordinate embeddings on benchmark POI inputs. Implementation note. CaLLiPer is trained with the common Foursquare input corpus used by CityRep so that the same city coverage and task alignment protocol can be applied to all benchmark cities. Official Space2Vec code: https://github.com/ gengchenmai/ space2vec; CityRep wrapper prepares POI classification data and task-grid exports. Implementation note. Space2Vec is trained under a uniform Foursquare-based POI classification setup for all cities. This provides a consistent coordinate-encoder baseline with the same task sampling and raster-center inference interface used by other coordinate models. 256-d coordinate encoder. Official SatCLIP code: Land-use and raster tasks are https://github.com/ queried directly at sample microsoft/satclip; coordinates or raster-cell official checkpoint centers. microsoft/SatCLIP-ViT16-L40. Implementation note. SatCLIP is evaluated as an off-the-shelf coordinate checkpoint. It uses only location coordinates at CityRep task units and does not use CityRep POI, map, street-view, or task-label data.

26

Table 6: Average feature coverage of H3-first aggregation and direct task-unit aggregation for sparse entity-based representations. Coverage is the fraction of downstream task units with a valid aligned representation. Numbers in parentheses on H3-first rows report ∆ = H3-first minus Task-unit direct. Green indicates higher coverage, and red indicates lower coverage. Model

Variant

LUC

RDE

POP

AGE

GDP

NTL

PM2.5

LST

CityFM

H3-first Task-unit direct

0.817 (+0.488) 0.329

0.775 (+0.479) 0.296

0.912 (+0.554) 0.358

0.958 (+0.545) 0.413

0.781 (-0.003) 0.784

0.776 (+0.134) 0.642

0.797 (-0.039) 0.836

0.814 (+0.505) 0.309

Place2Vec

H3-first Task-unit direct

0.840 (+0.482) 0.358

0.806 (+0.474) 0.332

0.927 (+0.513) 0.414

0.958 (+0.480) 0.478

0.810 (-0.003) 0.813

0.806 (+0.118) 0.688

0.827 (-0.029) 0.856

0.836 (+0.474) 0.362

Table 7: Effect of H3-first aggregation compared with direct task-unit aggregation for sparse entitybased representations. Numbers in parentheses on H3-first rows report ∆ = H3-first minus Task-unit direct. Green indicates improvement according to the metric direction, and red indicates degradation. Model

Variant

CityFM Place2Vec

C.3

LUC F1↑

RDE R2 ↑

POP R2 ↑

AGE KL↓

GDP R2 ↑

NTL R2 ↑

PM2.5 R2 ↑

LST R2 ↑

H3-first Task-unit direct

0.165 (+0.041) 0.125

0.187 (+0.057) 0.130

0.228 (+0.093) 0.135

0.021 (-0.003) 0.024

0.199 (+0.001) 0.198

0.369 (+0.011) 0.358

0.348 (-0.038) 0.386

0.496 (+0.385) 0.111

H3-first Task-unit direct

0.168 (+0.034) 0.134

0.217 (+0.037) 0.180

0.288 (+0.135) 0.154

0.023 (-0.003) 0.026

0.191 (+0.006) 0.185

0.303 (+0.041) 0.262

0.063 (-0.022) 0.085

0.138 (+0.159) -0.022

Aggregation for Sparse Entity Representations

CityRep uses an H3-first aggregation strategy for sparse entity-based representations such as POI and map-entity embeddings. Instead of directly aggregating sparse entities to each downstream task unit, we first aggregate entities to H3 resolution-8 cells and then align the resulting cell embeddings to task units. This intermediate support reduces missing or unstable features when task units are small or when entities are unevenly distributed. Table 7 compares this strategy with direct task-unit aggregation for CityFM and Place2Vec. The H3first strategy improves most task metrics for both models, supporting our choice to use an intermediate spatial support before downstream alignment. Table 6 further compares the average feature coverage of H3-first aggregation and direct task-unit aggregation on dense raster tasks. Coverage is defined as the fraction of task units that receive a valid non-missing representation after alignment. H3-first aggregation substantially improves coverage for both CityFM and Place2Vec, especially on LST and POP, explaining why the intermediate H3 support leads to more stable downstream evaluation. C.4

Evaluation Metrics

CityRep evaluates downstream prediction performance using task-appropriate metrics for regression, classification, and distribution prediction. This appendix provides the formal definitions of all reported metrics. Regression metrics. For regression tasks, including road density, population, GDP, NTL, PM2.5 , and land-surface temperature, we report coefficient of determination (R2 ), mean absolute error (MAE), and root mean squared error (RMSE). The coefficient of determination is defined as Pn (yi − ŷi )2 R2 = 1 − Pi=1 , n 2 i=1 (yi − ȳ) where yi is the ground-truth target, ŷi is the prediction, and ȳ is the mean target value.

(8)

Mean absolute error is defined as n

MAE =

1X |yi − ŷi |. n i=1

(9)

Root mean squared error is defined as v u n u1 X RMSE = t (yi − ŷi )2 . n i=1 27

(10)

Among these metrics, R2 is used as the primary metric because it measures explained variance and is less sensitive to target scale differences across tasks and cities. Classification metrics. For land-use classification, we report macro F1, macro precision, and macro recall. Let C denote the number of classes. For class c, precision and recall are defined as Pc =

TPc , TPc + FPc

TPc , TPc + FNc

Rc =

(11)

where TPc , FPc , and FNc denote true positives, false positives, and false negatives for class c. The class-wise F1 score is F 1c =

2Pc Rc . Pc + R c

(12)

Macro precision, macro recall, and macro F1 are computed by averaging across classes: C

MacroPrecision =

1 X Pc , C c=1

(13)

C

MacroRecall =

1 X Rc , C c=1

(14)

C

MacroF1 =

1 X F 1c . C c=1

(15)

Macro F1 is used as the primary metric because the land-use labels are imbalanced across cities and categories, and macro averaging gives equal weight to each class. Distribution-prediction metrics. For age-distribution prediction, the target for each task unit is a probability distribution over age groups. Let p = (p1 , . . . , pK ) denote the target distribution and q = (q1 , . . . , qK ) the predicted distribution. We report KL divergence: K X

pk , qk + ϵ

(16)

DChebyshev = max |pk − qk |,

(17)

DKL (p∥q) =

pk log

k=1

where ϵ is a small numerical constant added for stability. We also report Chebyshev distance: k

and L1 distance: DL1 =

K X

|pk − qk |.

(18)

k=1

KL divergence is used as the primary metric because it measures distributional mismatch while accounting for relative probability mass differences across age groups. Metric direction. Higher values indicate better performance for R2 , macro precision, macro recall, and macro F1. Lower values indicate better performance for MAE, RMSE, KL divergence, Chebyshev distance, and L1 distance. 28

C.5

Training protocol.

All downstream predictors are trained with the same hyperparameters across models and tasks: a multilayer perceptron with hidden dimension 1024, batch size 512, learning rate 10−3 , a maximum of 100 epochs, and validation early stopping with patience 10. Regression targets are standardized within each city–task dataset during training. These choices keep the downstream model sufficiently expressive to use the embeddings, while limiting task-specific tuning. All experiments were conducted using PyTorch on a workstation equipped with two NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96GB memory, and an AMD Ryzen Threadripper PRO 9975WX CPU with 32 cores and 64 threads. Spatial preprocessing and alignment were performed on the CPU using standard geospatial raster and vector processing libraries, while downstream predictors were trained on GPUs. Most downstream evaluation runs used a single GPU and required less than several hours per city–task pair, depending on task resolution and embedding size. Aggregation across seeds and cities. Let sm,t,c,k denote the primary test score for model m, task t, city c, and seed k. In our benchmark instantiation, we run five split seeds, {42, 24, 7, 0, 100}, and first average the primary metric over seeds to obtain a city-level score: 5

sm,t,c =

1X sm,t,c,k . 5

(19)

k=1

We then report, for each model and task, the mean of these city-level scores across cities as 1 X Avgm,t = sm,t,c , |Ct |

(20)

c∈Ct

together with the cross-city standard deviation s X
2 1 C-Std.m,t = sm,t,c − Avgm,t . |Ct | − 1

(21)

c∈Ct

Here Ct denotes the set of cities included for task t. The average score measures overall task utility, while C-Std. measures how much performance varies across urban contexts. Since different tasks use different primary metrics and metric scales, CityRep uses raw task metrics for the main result columns and uses rank-based summaries only as diagnostic views for comparing models across tasks. Overall rank. The overall rank in Table 2 is computed from city-level rankings rather than from normalized metric values. For each task–city pair (t, c), we rank all models according to the seedaveraged city-level score sm,t,c , with rank 1 assigned to the best model. Higher values are better for R2 and macro F1, while lower values are better for KL divergence in the age-distribution task. Ties receive the best shared rank. This gives a city-level rank rm,t,c . We then average city-level ranks within each task: Rm,t =

1 X rm,t,c , |Ct |

(22)

c∈Ct

where Ct is the set of evaluated cities for task t. For all tasks except age distribution, Ct contains the eight benchmark cities. For age distribution, Ct contains only London, New York, Singapore, and Sydney because the remaining four cities are excluded from the main AGE aggregation due to source-data quality concerns. Finally, the overall rank is the equal-task average: Rm =

1 X Rm,t . |T |

(23)

t∈T

Lower values indicate better overall performance. This aggregation avoids comparing raw metric scales across tasks and prevents tasks with more evaluated cities or denser samples from dominating the overall ranking. 29

Figure 17: Spatial-split primary metric values and rank diagnostics for all evaluated models.

Figure 18: Random-split primary metric values and rank diagnostics for all evaluated models.

D

Comprehensive Results

Figure 17 provides spatial-split diagnostics corresponding to the main benchmark results, and Figure 18 provides the same view under random splitting. These figures are diagnostic rather than replacements for the main table: raw primary metrics remain task-specific, while rank views are used to compare model ordering across heterogeneous metrics. We omit separate mean-city-rank figures here because that heatmap is already included in the right panel of each primary-metric diagnostic figure. D.1

Results Across Tasks and Cities

This section provides the full task-level metric plots for both spatial and random splits. Figures 19 and 20 show the land-use results; Figures 21 and 22 show the road-density results; Figures 23 and 24 show the population results; Figures 25 and 26 show the age-distribution results; Figures 27 and 28 show the GDP results; Figures 29 and 30 show the NTL results; Figures 31 and 32 show the PM2.5 results; and Figures 33 and 34 show the land-surface-temperature results. Unlike the main table, which reports one primary metric per task, these figures include secondary metrics as well: macro precision and macro recall for land use, MAE and RMSE for regression tasks, and L1 distance and Chebyshev distance for age-distribution prediction. Bars report the mean across five seeds, and error bars report the standard deviation across those seeds. These views are useful for checking whether a model’s ranking is driven by one metric alone or whether it is stable across multiple error summaries. The paired layout also shows how the split protocol changes both apparent performance and uncertainty. Spatial splits generally yield lower apparent performance on spatially smooth targets, while random splits can make local interpolation easier. In addition, spatial splits often produce larger error bars than random splits, especially for coarse raster tasks such as GDP, NTL, and PM2.5 . This does not necessarily indicate only optimization instability. Under the spatial protocol, different seeds hold out different geographic blocks, so the standard deviation combines model stochasticity with variation in the difficulty and target variance of the held-out regions. This effect is particularly visible for R2 , whose denominator depends on the variance of the test targets; if a held-out block has a narrow target range, a moderate prediction error can lead to a very low or negative R2 . MAE and RMSE are therefore reported alongside R2 to help distinguish unstable explained-variance estimates from changes in absolute prediction error. The difference is not uniform across tasks or models, which 30

Figure 19: Land-use classification: full metric bars under spatial block splits.

Figure 20: Land-use classification: full metric bars under random splits. is why we report full task-level plots rather than only a single aggregate rank. Spatial and random figures are shown separately to keep labels and city-level bars readable.

31

Figure 21: Road-density regression: full metric bars under spatial block splits.

Figure 22: Road-density regression: full metric bars under random splits.

Figure 23: Population regression: full metric bars under spatial block splits.

Figure 24: Population regression: full metric bars under random splits.

Figure 25: Age-distribution prediction: full metric bars under spatial block splits.

32

Figure 26: Age-distribution prediction: full metric bars under random splits.

Figure 27: GDP regression: full metric bars under spatial block splits.

Figure 28: GDP regression: full metric bars under random splits.

Figure 29: NTL regression: full metric bars under spatial block splits.

Figure 30: NTL regression: full metric bars under random splits.

33

Figure 31: PM2.5 regression: full metric bars under spatial block splits.

Figure 32: PM2.5 regression: full metric bars under random splits.

Figure 33: Land-surface-temperature regression: full metric bars under spatial block splits.

Figure 34: Land-surface-temperature regression: full metric bars under random splits.

34

Table 8: Linear-probe benchmark results on CityRep. Same layout as the main benchmark table, but using a linear downstream probe. For each task, Avg. reports the mean primary metric across 8 cities and 5 random seeds under the spatial block split. C Std. reports the cross-city standard deviation of city-level performance. Type indicates the main data sources used by each representation: L = location, P = POI, R = remote sensing, S = street-view imagery, and O = other urban/map data. Mean City Rank is computed from unrounded city-level rankings across all tasks and cities (lower is better). Best, second-best, and third-best results in the Avg. columns are in bold, underlined, and respectively. *For AGE, lower KL divergence indicates better performance, and wavy-underlined, ::::::::::::: results are reported only for the four cities with the most reliable age–sex source coverage. ♠ Morphology Model

Type

LUC

♡ Demographics

RDE

POP

♢ Economy

AGE∗

GDP

♣ Environment

NTL

PM2.5

Overall

LST

Rank

Avg. Avg. Avg. Avg. Avg. Avg. Avg. Avg. C C C C C C C C Mean (F1 ↑) Std. (R2 ↑) Std. (R2 ↑) Std. (KL ↓) Std. (R2 ↑) Std. (R2 ↑) Std. (R2 ↑) Std. (R2 ↑) Std. City Rank PE Place2Vec Space2Vec CaLLiPer CityFM Urban2Vec MuseCL SatCLIP TESSERA AlphaEarth AETHER

L P LP LP PO PRS PRS LR R R PR

Mean over models –

D.2

0.125 0.027 0.137 0.026 0.128 0.030 0.150 0.048 0.119 0.015 0.119 0.014 0.131 0.016 0.092 0.022 0.258 0.067 0.254 0.076 ::: 0.259 0.079

0.111 0.159 0.107 0.137 0.139 0.192 0.206 0.054 0.538 0.514 0.494 :::

0.050 0.097 0.049 0.055 0.079 0.090 0.094 0.039 0.086 0.087 0.104

0.130 0.217 0.180 0.250 0.141 0.232 0.267 0.022 0.592 ::: 0.615 0.637

0.087 0.062 0.083 0.110 0.084 0.085 0.091 0.050 0.140 0.158 0.108

0.042 ::: 0.046 0.048 0.045 0.053 0.045 0.046 0.048 0.043 0.042 0.040

0.023 0.025 0.027 0.027 0.032 0.022 0.022 0.021 0.020 0.019 0.020

0.183 0.200 0.202 0.243 0.229 0.268 0.321 -0.015 0.596 ::: 0.598 0.741

0.128 0.070 0.161 0.156 0.034 0.147 0.119 0.096 0.134 0.127 0.101

0.220 0.266 0.282 0.315 0.284 0.226 0.347 0.046 0.595 ::: 0.632 0.488

0.204 0.131 0.154 0.134 0.173 0.149 0.113 0.148 0.143 0.105 0.131

0.536 0.084 0.271 0.335 0.348 0.061 0.054 0.033 0.445 ::: 0.436 0.562

0.243 0.069 0.251 0.266 0.304 0.105 0.065 0.087 0.159 0.149 0.171

0.152 0.077 0.135 0.245 0.134 0.126 0.118 0.073 0.373 0.442 0.433

0.124 0.060 0.154 0.138 0.183 0.099 0.100 0.139 0.161 0.139 0.136

0.161 0.038 0.241 0.075 0.298 0.096 0.045 0.023 0.324 0.116 0.354 0.144 0.288 0.170 0.210 0.130

6.484 7.422 7.578 5.734 7.516 7.281 6.500 10.078 2.688 ::: 2.656 1.969 –

Results with Linear Downstream Task Head

Linear-probe evaluation. To examine whether the main benchmark conclusions depend on the capacity of the downstream MLP evaluator, we additionally evaluate all representations with a linear probe under the same spatial block splits, city–task datasets, metrics, and aggregation protocol. Table 8 reports the resulting average performance and cross-city standard deviation. Overall, the ranking is broadly consistent with the main benchmark: AETHER obtains the best mean city rank, followed by AlphaEarth and TESSERA. The three raster or raster-enhanced representations remain the strongest overall, suggesting that their advantage is not solely due to the nonlinear downstream head. At the task level, AETHER performs best on LUC, POP, AGE, GDP, and PM2.5 , AlphaEarth leads on NTL and LST, and TESSERA achieves the best result on RDE. As expected, the linear probe generally yields lower absolute scores than the MLP evaluator, especially on tasks that require more nonlinear decision boundaries or heterogeneous spatial signals. These results indicate that CityRep rankings are reasonably stable under a lower-capacity downstream evaluator, while also showing that some tasks benefit from nonlinear probing.

D.3

Results with 20x20 Spatial Split

To examine whether the benchmark conclusions are sensitive to the spatial block granularity, we additionally evaluate all baselines using a finer 20×20 spatial split while keeping the same downstream models, task files, evaluation metrics, and five random seeds. In Figure 35, the 20×20 split produces higher raw scores than the main 10×10 split for most regression and classification tasks, especially for LST, NTL, and PM2.5 . This is expected because smaller spatial blocks reduce the distance between training and test regions and therefore make the spatial generalization setting less strict. The results are in Table 9. Importantly, the relative ordering of model families remains broadly consistent. Remote-sensing raster representations remain the strongest group overall, while coordinate and region models improve more noticeably on several tasks under the finer split. These results support the main benchmark choice of the 10×10 split as the stricter protocol, and show that the main findings are not an artifact of a single spatial partitioning resolution. 35

Figure 35: Raw primary metric change from the main 10×10 spatial split to the 20×20 spatial split. Values are computed as 20×20 block result minus 10×10 block result. Positive values indicate higher raw metric values; for AGE, the metric is KL divergence, so lower values are better.

Table 9: Benchmark results under the 20×20 spatial block split. For each task, Avg. reports the mean primary metric across cities and 5 random seeds. C Std. reports the cross-city standard deviation of city-level performance. Type indicates the main data sources used by each representation: L = location, P = POI, R = remote sensing, S = street-view imagery, and O = other urban/map data. Mean City Rank is computed from unrounded city-level rankings across all tasks and cities (lower is better). Best, second-best, and third-best results in the Avg. columns are in bold, underlined, and wavy-underlined, respectively. *For AGE, lower KL divergence indicates better performance, and ::::::::::::: results are reported only for the four cities with the most reliable age–sex source coverage. ♠ Morphology Model

Type

LUC

♡ Demographics

RDE

POP

♢ Economy

AGE∗

GDP

♣ Environment

NTL

PM2.5

Overall

LST

Rank

Avg. Avg. Avg. Avg. Avg. Avg. Avg. Avg. C C C C C C C C Mean (F1 ↑) Std. (R2 ↑) Std. (R2 ↑) Std. (KL ↓) Std. (R2 ↑) Std. (R2 ↑) Std. (R2 ↑) Std. (R2 ↑) Std. City Rank PE Place2Vec Space2Vec CaLLiPer CityFM Urban2Vec MuseCL SatCLIP TESSERA AlphaEarth AETHER

L P LP LP PO PRS PRS LR R R PR

Mean over models –

D.4

0.213 0.070 0.200 0.044 0.220 0.054 0.249 0.075 0.230 0.068 0.196 0.052 0.191 0.038 0.122 0.044 0.368 0.075 ::: 0.387 0.068 0.375 0.080

0.191 0.236 0.171 0.209 0.230 0.251 0.262 0.129 0.640 0.617 0.585 :::

0.084 0.098 0.056 0.054 0.080 0.096 0.096 0.079 0.088 0.085 0.091

0.257 0.336 0.306 0.390 0.355 0.321 0.343 0.147 0.685 ::: 0.708 0.713

0.091 0.088 0.093 0.102 0.095 0.090 0.092 0.094 0.105 0.112 0.095

0.036 0.040 0.037 0.033 0.035 ::: 0.040 0.044 0.040 0.037 0.036 0.034

0.020 0.020 0.021 0.021 0.021 0.021 0.021 0.020 0.018 0.017 0.018

0.248 0.273 0.309 0.379 0.354 0.371 0.386 0.016 0.664 0.645 ::: 0.772

0.154 0.158 0.174 0.188 0.142 0.147 0.159 0.090 0.145 0.132 0.136

0.430 0.394 0.500 0.589 ::: 0.560 0.463 0.441 0.205 0.572 0.706 0.610

0.084 0.105 0.110 0.096 0.129 0.078 0.088 0.125 0.136 0.074 0.084

0.617 0.143 0.391 0.523 0.526 0.169 0.134 0.205 0.529 0.533 ::: 0.667

0.252 0.058 0.207 0.210 0.259 0.117 0.081 0.228 0.148 0.137 0.148

0.720 0.273 0.565 0.661 ::: 0.693 0.360 0.315 0.399 0.600 0.612 0.605

0.098 0.105 0.169 0.145 0.132 0.161 0.187 0.106 0.132 0.142 0.167

0.250 0.061 0.320 0.082 0.415 0.096 0.037 0.020 0.402 0.148 0.497 0.101 0.403 0.168 0.527 0.140

6.297 8.359 6.984 4.641 5.156 7.719 8.031 10.031 3.422 ::: 2.859 2.500 –

Exploratory Extension to 26 Cities

To examine whether CityRep can scale beyond the eight benchmark cities, we construct an exploratory 26-city extension using the same task registry, alignment pipeline, spatial split protocol, and evaluation code. The extension includes the eight main benchmark cities and 18 additional cities for which the global raster-based task layers can be prepared. In this exploratory setting, we evaluate two raster 36

AlphaEarth and TESSERA performance across 26 cities TESSERA

0.8

AlphaEarth

Paper 8 cities

RDE (R2 )

R2

0.6 0.4

cair o

ires

san tiag o

ngh ai

nos a

sha

bue

k

ul seo

ai

ban gko

urg

dub

joha n

nes b

nbu l

beij ing

ista

in

drid

berl

ma

toro nto

lo

gele s los an

city

pau sao

me xico

s

tok yo

pari

rta

wn e to

jaka

cap

ai

nair obi

mb mu

re

ney syd

apo sing

york new

lond o

0.0

n

0.2

POP (R2 ) 0.8

R2

0.6 0.4

2

cair o

ires

san tiag o

ngh ai

nos a

sha

bue

k

ul seo

ban gko

ai dub

urg nes b joha n

beij ing

nbu l ista

in

drid

berl

ma

toro nto

lo

gele s los an

pau sao

city me xico

tok yo

s pari

rta

wn e to

jaka

cap

nair obi

ai mb mu

ney syd

re apo sing

york new

0.0

lond on

0.2

GDP (R2 )

R2

0 2

cair o

ires

san tiag o

ngh ai

nos a

sha

bue

k

ul seo

ban gko

ai dub

rg sbu nne joha

beij ing

nbu l ista

in

drid

berl

ma

toro nto

lo

gele s los an

pau sao

city me xico

tok yo

s pari

rta

wn e to

jaka

cap

nair obi

ai mb mu

ney syd

re apo sing

york new

lond on

4

NTL (R2 ) 0.8

R2

0.6 0.4

o

cair o

s aire

tiag san

i ngh a

nos

sha

bue

k

ul seo

gko ban

ai dub

rg sbu joha

nne

ing beij

l ista nbu

in

drid

berl

ma

nto toro

eles ang los

pau lo sao

city me xico

tok yo

s pari

wn e to

jaka rta

cap

nair obi

ai mb mu

ney syd

re apo sing

york new

lond

0.0

on

0.2

o

cair o

o

cair o

s aire

tiag san

i ngh a

nos

sha

bue

k

ul seo

gko ban

ai dub

rg sbu joha

nne

ing beij

l ista nbu ista nbu

in

drid

berl

ma

berl

drid

nto toro nto

eles ang los

pau lo sao

city

tok yo

me xico

tok yo

s pari

s pari

wn

wn e to

jaka rta jaka rta

cap

nair obi nair obi

ai mb mu

ney syd

re apo sing

york

ai

1.0

new

lond

on

R2

PM25 (R2 ) 0.8 0.6 0.4 0.2 0.0 0.2

LST (R2 )

R2

0.8 0.6 0.4

s aire

tiag san

i ngh a

nos

sha

bue

k

ul seo

gko ban

ai dub

rg sbu

l

in

ing beij

nne joha

City

ma

toro

eles ang los

pau lo sao

city me xico

e to cap

mb mu

ney syd

re

on

apo sing

new

lond

0.0

york

0.2

Figure 36: Exploratory 26-city extension using AlphaEarth and TESSERA. Bars report the mean primary metric across five spatial-split seeds for each city-task pair, and error bars report the standard deviation across seeds. The shaded region marks the eight cities used in the main benchmark setting. The extension uses the six globally available raster-based tasks: RDE, POP, GDP, NTL, PM2.5 , and LST. Higher is better for all reported metrics.

representation models, AlphaEarth and TESSERA, on the six tasks available for all 26 cities: road density, population, GDP, NTL, PM2.5 , and land surface temperature. Figure 36 compares AlphaEarth and TESSERA across all available city–task pairs. Overall, the two models exhibit broadly consistent cross-city trends: cities that are difficult for one model often also tend to be difficult for the other, suggesting that part of the performance variation reflects city- and task-specific data characteristics rather than purely model-specific behavior. At the same time, the two models show different strengths across individual tasks. TESSERA performs strongly on several morphology-related and environmental tasks, whereas AlphaEarth is competitive or stronger on some demographic and thermal tasks. This highlights the necessity of multi-task evaluation: results from any single task are insufficient for comprehensively understanding model behavior. Although the multi-city evaluation reveals overall patterns of consistency, observations from an individual city can sometimes contradict the aggregate trend. For example, on the population task, TESSERA performs better in Mexico City and São Paulo, even though AlphaEarth outperforms TESSERA in most cities overall. We therefore treat this extension as evidence for the value of systematic multi-city evaluation. The 26-city extension is not used as the main leaderboard because not all baselines currently provide complete embeddings or reproducible checkpoints for the expanded city set, and tasks such as land use and age distribution require additional city-specific auditing. 37

0.11

0.23

0.16

-0.12

0.28

-0.03

-0.29

0.25

-0.23

0.36

0.58**

0.43*

0.16

0.13

0.06

0.28

0.22

0.18

Population total (log)

-0.11

-0.05

-0.08

0.08

0.09

0.12

-0.26

-0.00

-0.27

0.21

0.30

0.37

0.25

0.22

0.20

0.34

0.27

0.26

Population density (log)

-0.26

-0.32

-0.29

0.13

-0.35

0.05

0.04

-0.38

-0.02

-0.41*

-0.52**

-0.37

-0.04

0.01

0.05

-0.09

-0.04

-0.01

GDP total (log)

0.51**

0.47*

0.50**

-0.13

-0.09

-0.21

0.17

-0.05

0.02

-0.16

0.02

-0.23

-0.19

-0.21

-0.20

-0.18

-0.19

-0.26

GDP per capita (log)

0.48*

0.44*

0.46*

-0.24

-0.19

-0.32

0.20

-0.13

0.10

-0.36

-0.26

-0.48*

-0.31

-0.31

-0.27

-0.43*

-0.39*

-0.46*

GHSL built-up area (log)

-0.07

0.08

0.00

-0.22

0.06

-0.17

-0.15

0.04

-0.12

0.19

0.40*

0.35

0.07

0.09

0.03

0.27

0.17

0.12

GHSL built-up share

-0.39

-0.37

-0.36

-0.13

-0.52**

-0.19

0.11

-0.50**

0.06

-0.41*

-0.51**

-0.33

-0.08

0.03

0.05

-0.22

-0.17

-0.17

OSM road density (log)

-0.55**

-0.69***

-0.60**

0.03

-0.25

0.05

-0.04

-0.33

-0.08

0.18

0.10

0.25

0.37

0.45*

0.45*

0.23

0.25

0.30

OSM POI count (log)

-0.00

-0.07

0.02

-0.37

-0.21

-0.44*

0.32

0.02

0.29

0.11

0.23

0.11

-0.09

-0.14

-0.10

-0.33

-0.39

-0.37

OSM POI density (log)

-0.14

-0.26

-0.17

-0.20

-0.34

-0.29

0.36

-0.17

0.31

-0.19

-0.25

-0.22

-0.09

-0.09

-0.03

-0.38

-0.36

-0.33

1.00 0.75 0.50 0.25 0.00

Spearman rho, performance-oriented

factor

All-city, all-task, all-metric factor correlations Positive rho means higher factor is associated with better performance; stars use raw Spearman p-values Area (log)

0.25 0.50 0.75

E

AE

MS TR LS

2

SE

TR

TM LS

LS

MA E

R2

RM 25 PM

25 PM

E

AE

2

MS

25 PM

LR NT

LM

E

target

NT

LR NT

AE

2

E

MS PR GD

PM GD

PR GD

AE

E

2

MS PR PO

PM PO

PR PO

AE

MS ER RD

EM

ER RD

RD

2

1.00

Figure 37: Exploratory correlation analysis between city-level factors and downstream performance across the 26-city remote-sensing extension. Each cell reports the Spearman correlation between a city factor and the two-model average performance of AlphaEarth and TESSERA for a task-metric pair. Metrics are oriented so that higher values always indicate better performance; KL, MAE, and RMSE are sign-flipped before averaging. Positive correlations therefore mean that larger factor values are associated with better performance. Stars denote raw, unadjusted Spearman p-values. The analysis is descriptive and intended to identify sources of performance heterogeneity rather than establish causal effects.

Nevertheless, the experiment demonstrates that CityRep’s registry-based design can support broader evaluation once new city–task files and model embeddings are registered.

E

Analysis of Performance Heterogeneity Across Tasks and Cities

To better characterize this heterogeneity, we conduct an exploratory city-factor analysis in the 26-city remote-sensing extension. For each city, we construct a common set of interpretable descriptors: total area, population, population density, total GDP, GDP per capita, GHSL built-up area, GHSL built-up share, OSM road density, OSM POI count, and OSM POI density. We then combine AlphaEarth and TESSERA results to obtain a two-model average performance signal for each city-task-metric pair. Metrics are first converted to a common performance orientation, so that larger values always indicate better prediction: R2 and F1 are kept unchanged, while KL divergence, MAE, and RMSE are sign-flipped. Because both models are evaluated on the same task and metric, we average the raw performance-oriented values directly rather than normalizing across cities. Finally, we compute Spearman correlations between each city factor and each task-metric performance signal. Figure 37 reports these correlations. The heatmap shows that city difficulty across all tasks is not explained by a single factor such as city size, population, or economic development. Some associations are task-specific and occasionally counterintuitive. GDP-related factors are positively associated with road-density prediction performance, but they do not uniformly improve performance on socioeconomic or environmental targets. Population density and built-up share are often negatively associated with performance on several metrics, suggesting that compact and highly built-up environments can be more difficult for current representations. OSM road density is strongly negatively associated with road-density prediction performance, indicating that dense street systems remain challenging even when the target itself is derived from road structure. POI-based factors show weaker and less consistent correlations, which may reflect differences in OSM completeness across cities as well as the fact that POI distributions only partially capture the spatial signals needed by the downstream tasks. A useful interpretation is that development-related factors involve two competing effects. On the one hand, more developed cities often have more complex urban structure, stronger functional specialization, denser infrastructure, and more heterogeneous economic activity. These properties can make tasks related to urban function and socioeconomic characteristics harder, because the mapping from observable geospatial patterns to targets such as GDP, nighttime lights, or other functional indicators may be less direct. On the other hand, more developed cities also tend to have richer and 38

higher-quality supporting data, including better coverage of roads, POIs, and other spatial layers, which can improve both model fitting and evaluation reliability. The opposite tension may hold for cities in less developed contexts. Their urban form may in some cases be simpler or more spatially regular, which could make certain prediction problems easier in principle. However, this potential advantage can be offset by weaker data coverage, lower source-data quality, and greater measurement uncertainty. We therefore do not interpret the correlations as showing that either developed or developing regions are uniformly easier. Instead, the observed performance reflects a trade-off between urban complexity and data availability, together with task-specific differences in what each target requires from the representation. This analysis supports the design choice of CityRep. A single aggregate leaderboard score can hide large differences across cities, tasks, and evaluation settings. Robust evaluation of urban foundation models should therefore report not only average performance, but also spatial-split results, random-split results, city-level variation, and coarse geographic stratifications.

39