arXiv CSopen access

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi, Luc Van Gool, and Danda Pani Paudel

(a) OOD gains during multi-task training

(b) Final gain vs. training-data diversity 140

Detection

run A run B

120

Segmentation

80 60 40

Ultra-high-res VQA Multi-modal VQA

20

Multi-view VQA RGB caption Temporal VQA

0

0

1k

2k

3k

4k

training steps

5k

(c) Balanced multi-task training mix 80k unique samples, curated from a 2.3M raw pool

Detection

120

100

OOD gain (%)

OOD gain over base model (%)

arXiv:2607.15942v1 [cs.CV] 17 Jul 2026

INSAIT, Sofia University “St. Kliment Ohridski” [email protected]

16 src

Multi-choice VQA

100 80 peaks at 1k steps, then declines

60

12 src

Caption

Ultra-high-res VQA

40

Multi-view VQA 20

8 src

Free-form VQA

Segmentation

Multi-modal VQA

plateaus

RGB caption Temporal VQA saturated

0 1

2

4

8

6 src + derived from Segmentation

Detection 16

# training labels sources for the domain

Gains averaged over OOD benchmarks per domain. Detection excludes xlrsbench-det, a ×7.6 outlier gain.

13 src

Segmentation

0

1k

2k

3k

4k

samples per 1k training steps generic RGB

high-res

temporal

SAR

multi-view

Fig. 1: Diverse multi-task training drives out-of-distribution generalization in remote sensing. (a) OOD gain over the base model during multi-task RL training, aggregated per task-domain. For robustness test the training is performed a second time with different seed and paraphrased prompts. (b) Final OOD gain vs. number of training sources per domain; for visualization, detection excludes a x7.6 gain outlier, XLRSBench. (c) Training mixture by task and input type; 80k samples balanced from a 2.3M pool.

Abstract. Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks. Most recent progress in this area has been driven by remote-sensing-specific architectural designs, often introducing new encoders, alignment modules, or task-specific fusion mechanisms. In this work, we challenge the necessity of such architectural specialization. We show that a generally capable vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks, provided that it is trained at sufficient scale across diverse data and tasks. Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding. To train this heterogeneous behaviour, we employ a multitask reinforcement learning framework with adaptive task rewards covering multiple-choice VQA, free-form VQA, captioning, detection, and segmentation across a large variety of input types. Our approach achieves

2

S.M. Ailuro et al. competitive results across a broad set of benchmarks, including highresolution, multi-temporal, multi-modal and multi-view tasks. Further, as training data scales, our experiments show consistent improvements across most tasks both in and out of distribution, which correlate with per-task data diversity. These findings suggest that, for remote sensing VLMs, data scale is more important than architectural novelty. Keywords: Remote Sensing · Vision Language Models · Reinforcement Learning

1

Introduction

Remote sensing is becoming central for understanding the Earth at scale. Modern satellite, aerial, and multi-sensor observation systems continuously capture information about land use, urbanization, agriculture, transportation, natural hazards, climate impacts, and environmental change. The resulting imagery is large, heterogeneous, and semantically rich: the same scene may require image-level recognition, fine-grained object localization, pixel-level segmentation, change reasoning, or open-ended question answering. As the volume and diversity of Earth observation data continue to grow, remote sensing increasingly demands models that can move beyond narrow task-specific prediction and support flexible, language-driven interaction with visual and geospatial data. Recent progress in vision-language models has therefore motivated a new generation of remote sensing large multimodal models [18,30,37,39]. These systems aim to connect remote sensing imagery with natural language instructions, enabling unified interfaces for tasks such as captioning, visual question answering, grounding, detection, segmentation, and dialogue. A common trend in this literature is to address the peculiarities of remote sensing by designing new remotesensing-specific architectures, alignment modules, token handling strategies, or modality-fusion mechanisms [3, 20, 23, 24, 35, 37, 52, 55]. Collectively, these works demonstrate the promise of remote sensing VLMs, but they also reflect a dominant assumption: that progress in the domain requires increasingly specialized model designs. In this work, we revisit that assumption. Instead of introducing a new remotesensing-specific architecture, we start from a generally capable vision-language model and scale its remote sensing adaptation through data, tasks, and reinforcement learning (RL). Our central hypothesis is simple: for remote sensing VLMs, broad task coverage and large-scale training data can be more important than architectural novelty. We train a single model across a large and diverse collection of remote sensing datasets and tasks, including free-form visual question answering, captioning, detection, grounding, and segmentation through promptable interaction with SAM3 [4]. To handle the heterogeneity of these tasks, we employ a reinforcement learning strategy with adaptive per-task rewards, allowing the model to optimize across tasks with different output formats and evaluation criteria.

More with Less: a Large Scale RS-VLM with a Simple Recipe

3

Our results show that this simple recipe is sufficient to achieve competitive or state-of-the-art zero-shot performance across a wide range of remote sensing benchmarks, without modifying the underlying VLM architecture. More importantly, we observe consistent gains as the amount and diversity of training data increase (see Fig. 1), suggesting that remote sensing VLM performance remains strongly data-scalable. These findings shift the emphasis from architecture engineering toward data and task scaling: rather than designing a new model for every remote sensing capability or modality, a strong general VLM can be adapted effectively by exposing it to enough diverse remote sensing supervision and optimizing it with task-aware rewards. Our contributions can be summarized as follows: – We present More with Less Remote Senser (MLRS), a remote sensing VLM built without introducing any new model architecture. Despite its architectural simplicity, MLRS achieves competitive or state-of-the-art zero-shot performance across a broad range of remote sensing tasks and input types via large-scale multi-task reinforcement learning, demonstrating that a strong general-purpose VLM can be effectively adapted to compete with or surpass specialized VLMs in the remote sensing domain. – We provide an empirical study of data scaling for remote sensing VLMs. Across training data sizes, model sizes, and optimization variants, we find that performance improves with increased data scale and task diversity, suggesting that data-centric scaling is a promising path toward general and capable remote sensing vision-language models.

2

Related works

Remote-sensing VLMs. Adapting vision–language models to remote sensing has produced a rapidly growing family of RS-VLMs (see Tab. 1). Early instruction-tuned models such as GeoChat [18], SkySenseGPT [23], and LHRSBot [20, 27] established conversational VQA, captioning, and grounding on optical imagery, and subsequent work has expanded coverage along two axes: task and input data types. Along the task axis, models added detection and visual grounding [14, 23], change detection [12, 14, 47], and segmentation [28, 29, 31, 33, 47, 57]. Along the input axis, dedicated architectures were proposed for multi-temporal sequences (TEOChat [14], DVLChat [47], EarthDial [37]), multi-modal inputs (EarthGPT [56], EarthDial [37], EarthMind [35], TerraScope [36]), ultra-high-resolution scenes (GeoLLaVA-8K [39], LRS-VQA [24], GeoVista [61], GeoPixel [33]), and visual prompting (EarthMaker [55], EarthGPTX [54]). Multi-view VLMs that merge information from ground views, street panoramas, and satellite imagery, are being developed separately, taking their roots in geolocalization [44, 50], and only recently multi-view models were applied to urban analysis and broader reasoning [11, 60], bringing them closer to remote sensing VLMs. A common thread in these lines of work is that each new input type is addressed with architectural specialization — temporal encoders, modality-specific branches, geometry encoders for visual prompting, or

4

S.M. Ailuro et al.

Table 1: Capability coverage of existing RS-VLMs. UHR=Ultra-high resolution; MM=Multi-Modal (Optical, NIR, SAR); MT=Multi-Temporal; VP=Visual Prompting; VQA=Visual Question Answering; Cap=Captioning; Det=Detection; Seg=Segmentation; VG=Visual Grounding; CD=Change Detection; TS=Trained Segmentation head; RR=Reinforced Reasoning; V=Versatile, automatically identifies the task and required answer format. ‘✓’ states that segmentation is performed in textual space as a vector polygon rather than a dense mask. Model

Task Type Det VG

VQA

Cap

GeoChat [18] SkySenseGPT [23] LHRS-Bot-nova [20] SkyEyeGPT [52] TEOChat [14] DVLChat [47] VHM [30] LISAt [31] GeoPix [29] GeoPixel [33] SegEarth-R1 [19] SegEarth-R2 [46] UniGeoSeg [28] Think2Seg-RS [57] RemoteReasoner [49] RSThinker [21] GeoZero [38] GeoVLM-R1 [12] GeoEyes [40] GeoVista [61] GeoLLaVA-8K [39] LRS-VQA [24] UrbanLLaVA [11] EarthGPT [56] EarthGPT-X [54] EarthMaker [55] EarthDial [37] EarthMind [35] TerraScope [36] Falcon [16] Earth-OneVision [3]

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓

✗ ✓ ✗ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✗ ✗ ✗ ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✓ ✗ ✗ ✓ ✓

MLRS (Ours)

✓

✓

✓

Method V RR

Seg

CD

TS

✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✓ ✓ ✗ ✓ ✓

✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓

✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✓ ✓

✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✓ ✗ ✓ ✗ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓

✓

✓

✓

✓

✓

Input Type UHR VP

MT

MM

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

✗ ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✓ ✗ ✗ ✓

MV ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

✓

✓

✓

✓

✓

✓

resolution-handling modules. In contrast, we show that a strong general-purpose base model trained on a sufficiently diverse multi-task, multi-domain mixture covers temporal, multi-modal, high-resolution, and multi-view inputs without architectural modification. To our knowledge, our study spans a broader combination of tasks and input types than any prior work. Post-training paradigms. Most RS-VLMs are trained with supervised fine-tuning alone [14, 18, 20, 23, 37, 47, 56]. However, SFT-trained models tend to fit narrowly to their instruction distributions and degrade on held-out benchmarks — a pattern observed directly both in general [7] and remote sensing [1, 10] domains. A recent wave of reinforcement-learning-based RS-VLMs — RSThinker [21], GeoZero [38], GeoVLM-R1 [12], RemoteReasoner [49], Think2SegRS [57], GeoEyes [40], and GeoVista [61] — reports noticeably better generalization, which our evaluation results corroborate. We therefore adopt reinforced reasoning as our post-training strategy. Segmentation interfaces. RS-VLMs realize segmentation through two interfaces: emitting polygons as text (VHM [30], Falcon [16], Earth-OneVision [3]) or attaching a trained segmentation head [6, 17, 32] that decodes dense masks

More with Less: a Large Scale RS-VLM with a Simple Recipe

5

(DVLChat [47], UniGeoSeg [28], SegEarth [19, 46], LISAt [31], GeoPix [29], GeoPixel [33], EarthMind [35], TerraScope [36]). Trained dense-mask heads are better suited to remote sensing, where objects are small, numerous, and irregular [2,5,51]. Yet, trained segmentation heads and reinforced reasoning have been so far mutually exclusive: RL-based models either omit segmentation or rely on frozen mask decoders [49, 57]. Our model is the first remote sensing VLM to combine a trained segmentation head with reinforcement learning, via group relative tool optimization (GRTO) [26], obtaining both dense-mask quality and the generalization benefits of RL post-training.

3

Method

Fig. 2: A single language policy (InternVL [43]) handles all input types — ultra-highresolution, multi-modal, multi-view, multi-temporal, visual prompting. The policy reasons in text, answers directly, or invokes a SAM3 segmentation tool via a textual prompt; components communicate only through text, with no task-specific encoders or fusion modules. Both components are optimized end-to-end with GRTO [26], using multi-task rewards: embedding similarity for captioning and free-form VQA, box IoU for detection, answer-set IoU for MC-VQA, and mask IoU for segmentation.

Data scaling. Our central experiment questions what drives generalization in multi-task RS training. We assemble a mixture spanning all multi-choice and free-form VQA, captioning, detection and segmentation (grounded and general) and input types, where by construction the number and heterogeneity of sources backing each domain varies from a single dataset (multi-view) to more than a dozen (segmentation, generic VQA). We then train at scale over an increasing number of unique datapoints while holding the task composition fixed, and periodically evaluate on ID and OOD benchmarks. This design lets us measure how gains evolve with training exposure, and, crucially, whether the shape of each domain’s trajectory — sustained improvement, plateau, or degradation —

6

S.M. Ailuro et al.

correlates with the diversity of its training sources rather than with the sheer volume of matching data or initial performance of the model. The More with Less Remote Senser (MLRS) model. MLRS is a tool-augmented remote sensing VLM that uses a general vision-language model as the central controller and SAM as an external localization tool. The model can operate on Optical, False-colour, SAR imagery, or mixed inputs. SAR observations are represented as grayscale images, allowing them to be processed through the same visual interface as optical images. MLRS also supports multiimage (multi-temporal, multi-view) inputs: when a task provides several images, all of them are shown to the VLM as context, while spatial outputs are produced with respect to the primary image. Finally, visual prompts are straightforwardly overlaid over the primary image. Given the image or image group and a natural-language instruction, MLRS must decide whether the task requires a direct language response or a spatial prediction. For language-only tasks, such as VQA and captioning, it produces a textual answer. For localization tasks, it instead emits a structured segmentation tool call which describes the target regions using two complementary signals: a noun phrase and one or more bounding boxes, which are passed to SAM3 [4], whose output is filtered through the passed bounding boxes. In this way, the VLM is responsible for semantic grounding and coarse spatial localization, while SAM3 is responsible for producing the final pixel-level mask. This interface is similar in spirit to recent VLM–segmentation-decoder systems for referring segmentation [26, 49, 57], but MLRS uses it as a general remote sensing interface for both textual answering and spatial localization. This design keeps the VLM architecture unchanged. The model communicates with the segmentation tool only through text: it either answers directly or writes a valid localization call that can be parsed into SAM prompts. Thus, MLRS does not require a specialized remote-sensing architecture or a separate task router. Instead, the VLM itself decides whether to respond in language or invoke the localization tool, enabling a single model to handle captioning, VQA, detection, grounding, and segmentation within the same interface. Adaptive-reward RL. To fine-tune MLRS, we employ group relative tool optimization (GRTO) [26], where while the VLM is trained with group relative policy optimization (GRPO) [34], a surrogate loss is applied to the tool, SAM3, enabling joint training between policy and tool. We combine binary cross-entropy and semantic IoU for SAM3’s surrogate objective, following [26]. For the GRPO policy objective, we design an adaptive task-dependent reward of the form: R = \lambda _{\mathrm {fmt}} F + \lambda _{\mathrm {task}} S, \label {eq:reward_general} (1) where F is a binary format-validity term, S is a task-specific score, and we use λfmt = 0.1 and λtask = 0.9 during training. The format term encourages the model to use the correct interface: textual tasks must produce an answer response, while localization tasks must produce a valid segmentation tool call. The score term depends on the task. For multiple-choice VQA, we parse the predicted option set and compute set IoU with the target option set, supporting both

More with Less: a Large Scale RS-VLM with a Simple Recipe

7

single-answer and multi-answer questions. For free-form VQA and captioning, we compare the extracted answer with the reference using Qwen3-Embedding0.6B [58], mapping cosine similarity to [0, 1]. For segmentation, we parse the localization call into phrase-level SAM prompts and score the resulting masks with semantic IoU against the target masks; if the ground truth contains several masks, we give zero reward if the number of predicted masks does not match, otherwise, we take the mean IoU. For detection, we parse the predicted boxes, convert them to image coordinates when necessary, greedily match them to groundtruth boxes, and score them by mean matched box IoU. Thus, the same reward template is used across all tasks, while the correctness metric changes according to the required output type. A detailed definition of each reward component is provided in the appendix. An overview of the framework is shown in Fig. 2.

4

Experiments and results

4.1

Experimental Design

Benchmarks. We separate evaluation into in-distribution (ID) and out-of-distribution (OOD) benchmarks. ID benchmarks provide training corpora and heldout test splits to measure task learning on familiar domains: DisasterM3 [42] (bi-temporal optical/SAR disaster imagery; captioning, VQA, and segmentation), DynamicVL [47] (long-range multi-temporal urban scenes; captioning, VQA, and segmentation over optical/SAR images), SARLANG-1M [45] (SAR captioning and free-form VQA), EarthReason [19] (reasoning segmentation), LaSeRS [46] (referring segmentation of varied granularity, multiplicity, linguistic variability, and reasoning requirements), and a held-out split of GeoSeg-1M – GeoSeg-Bench [28] by Ni et al. (visual grounding and reasoning segmentation over Optical and False-colour images). OOD benchmarks share no images or annotations with training and measure generalization across all input types: XLRSBench [41] probes Ultra-High-Res. understanding (images up to 10,000×10,000 px), reasoning, and visual prompting; RSHR-Bench [8] adds ultra-high resolution (UHR) MC-VQA with adversarially filtered questions including temporal reasoning; VLRS-Bench [25] targets multi-step complex reasoning over UHR images, understanding and forecasting in multi-temporal scenes, and includes textual and visual prompting; UrBench [60] evaluates multi-view reasoning over paired street-level and satellite imagery; GEOBench-VLM [9] spans all five tasks over multi-modal inputs (RGB, SAR, multi-spectral), and its change-detection subset additionally covers multi-temporal reasoning; and GeoSeg-Bench [15] by Jiang et al. evaluates reasoning segmentation under varied difficulty level – to avoid name collision with ID Benchmarks we refer to it as GeoSeg-Bench2 or GeoSeg2. Training mix. To study how per-domain data diversity affects generalization, we assemble a multi-task mixture whose domains are deliberately backed by varying numbers of label sources: generic RGB tasks draw on the GeoZero corpus [38] (11 sources for MC-VQA, 8 for captioning, 6 for detection, 4 for

8

S.M. Ailuro et al.

free-form VQA) plus GeoLLaVA [39] for high-resolution MC-VQA; segmentation combines GeoSeg-1M [28], LaSeRS [46], EarthReason [19], DynamicVL [47], and DisasterM3 [42] (13 sources); SAR tasks use SARLANG-1M [45] and DisasterM3; multi-temporal tasks rely on only two related sources (DynamicVL, DisasterM3); and multi-view tasks on a single source. We construct multi-view instruction data from CVG-Text [50], a cross-view geo-localization corpus of coregistered ground-level panoramas and satellite views: panorama captions are used directly for captioning, and we deterministically generate two MC-VQA item types — city recognition from a panorama–satellite pair, and cross-view matching, where the model must select which of four satellite views (with samecity hard negatives) depicts the panorama’s location. A total raw pool of collected training data consists of 2.3M image-prompt-answer pairs, we curate an 80k training set from it, balancing task domain and question difficulty if it is provided (see Fig. 1c). Training setup. We train on eight NVIDIA H200 GPUs with learning rates of 1 × 10−6 for the VLM and 1 × 10−4 for SAM3. We use a group size of 8, and a global batch size of 16 unique samples for a total of 16 × 8 = 128 rollouts per batch. We set the KL-divergence weight of the GRPO objective to β = 0.01. For all trainings, we use Low-Rank Adaptation (LoRA) [13] with rank equal to 64. The model is prompted to think, and we set the maximum output length to 1024 tokens. To balance multi-task gradients, we preserve the following composition within each batch: 25% MC-VQA, 25% captioning, 25% free-form VQA and detection, 25% segmentation. Experiments. To study how data scale affects remote sensing performance when starting from a general-purpose VLM, we train several MLRS variants using the setup described in Section 3. Our primary run uses InternVL3.5-8B [43] as the VLM backbone and is trained with GRTO for 5K steps, corresponding to 80K unique samples. To check run-to-run robustness, we train a second 8B model under the same setup for 3K steps with a different random seed and paraphrased prompt templates. To test whether the observed trends depend on model size, we also train an MLRS variant with the smaller InternVL3.5-2B backbone. We evaluate every checkpoint at 1K-step intervals, allowing us to track performance as the amount of training data increases. Finally, to disentangle the effects of data scaling between the VLM and SAM3, we train an additional 8B model for 5K steps using plain GRPO instead of GRTO, keeping SAM3 frozen. In all runs, the VLM is prompted to reason, decide whether the task requires a textual response or localization, and produce the corresponding output format. The final MLRS model refers to the last checkpoint of the primary 8B GRTO run. Metrics. We evaluate each task using metrics consistent with the corresponding benchmark whenever possible, introducing alternative metrics only when the original protocol is insufficient for our setting. For tasks with freeform answers, we use dataset-specific LLM-as-judge evaluation [59]. In particular, for DisasterM3 [42], DynamicVL [47], and the VQA split of SARLANG1M [45], we follow the dataset-specific GPT-style scoring rubrics and prompts. For GEOBench-VLM [9] captioning, we report BERTScore [53], following prior

More with Less: a Large Scale RS-VLM with a Simple Recipe

9

work. For captioning tasks in SARLANG-1M [45] and XLRS-Bench [41], where prior work relies on conventional reference-overlap NLP metrics, we instead reevaluate all baselines with G-Eval [22], a structured LLM-as-judge framework that derives scores from next-token probability distributions rather than a single sampled rating and has been shown to align better with human judgments than reference-overlap metrics. We use Qwen2.5-32B-Instruct [48] as the judge model. For multiple-choice VQA, we report accuracy. For segmentation, we report gIoU, computed as the mean per-sample intersection-over-union. For detection, we report [email protected] on GEOBench-VLM [9] and [email protected] on XLRS-Bench [41], following the respective benchmark protocols. 4.2

Results

In-distribution benchmarks

DisasterM3

performance

Multi-choice VQA

SARLANG-1M

EarthReason

0.6

0.30

0.5

0.35

0.28

0.4

0.30

0.26

0.3

0.24

0.2

1k

2k

3k

4k

5k

2k

Out-of-distribution benchmarks Non-temporal VQA

3k

4k

5k

GEOBench-VLM

1k

2k

3k

UrBench

Temporal VQA

4k

5k

0.8

0.475

0.40

0.7

0.450

0.35

0.6

0.425

0.30

0.5

0.400

0.25

2k

3k

training steps

4k

5k

2k

RSHR-Bench

3k

4k

5k

0

1k

2k

3k

4k

5k

4k

5k

GeoSeg-Bench 2

Detection

Segmentation

0.6 0.5

0.6

0.4 0.4

0.3 0.2

0.2

0.1

0.4

0.0

0.20 1k

1k

RGB Caption

0.45

0

0.0 0

VLRS-Bench

0.9

0.50

0.375

0.2

0.1

XLRS-Bench

0.500

base model

0.4 0.2

0

0.525

run B

Segmentation 0.6

0.20

1k

run A

0.3

0.25

0

LaSeRS

SAR Caption 0.4

0.1 0

GeoSeg-Bench

RGB Caption 0.40

0.32

0.22

performance

DynamicVL

Free-form VQA

0

1k

2k

3k

training steps

4k

5k

0

1k

2k

3k

training steps

4k

5k

0

1k

2k

3k

training steps

4k

5k

0

1k

2k

3k

training steps

Fig. 3: Performance vs training steps. Following metrics are used: aggregated accuracy for multi-choice VQA, G-Eval for free-form VQA and caption, aggregated IoU for segmentation, [email protected] for GEOBench-VLM detection, and [email protected] for XLRS-Bench detection.

Effects of scaling training data. Fig. 3 reports per-benchmark performance over the course of training for both runs. On in-distribution benchmarks, multi-task training improves majority of task families, with the largest absolute gains where the base model is weakest: SAR captioning rises from 0.09 to 0.38 G-Eval, and referring segmentation improves from 0.27 to 0.68 IoU on EarthReason and from 0.31 to 0.67 gIoU on LaSeRS. Crucially, these gains transfer out of distribution: on held-out benchmarks, detection precision on GEOBenchVLM more than doubles (0.24 → 0.56 [email protected]), segmentation improves from 0.41 to 0.70 IoU on GeoSeg-Bench2 and from 0.08 to 0.16 IoU on GEOBenchVLM, and VQA accuracy increases from 0.39 to 0.51 on ultra-high-resolution XLRS-Bench and from 0.39 to 0.50 on GEOBench-VLM (+28%). XLRS-Bench detection improves by a factor of 7.6 (0.054 → 0.409 [email protected]). Not all domains benefit equally, however: multi-view VQA on UrBench improves only marginally

10

S.M. Ailuro et al.

Table 2: Caption, free-form VQA, and detection results. For each baseline, the model size closest to 8B is chosen. If multiple versions of the same baseline are reported, we take the one with the highest performance. If a baseline is fine-tuned on the corresponding benchmark, fine-tuned performance is reported and marked with an asterisk (*). Metrics in the [0, 1] range are rescaled to the [0, 100] range. In each column, the best, second-best, and third-best results are highlighted in gold, silver, and bronze, respectively. Zero-shot benchmarks are highlighted in blue. Caption

Free-form VQA

Detection

Method

DM3 DVL SL-1M XLRS GEO DM3 DVL SL-1M GEO XLRS

Qwen GPT GeoChat EarthDial

3.90* 3.03 3.58 39.21 58.95 3.11* 2.34 73.33* 15.18 2.57 4.36 10.40 47.49 64.18 3.19 2.75 0.87 6.13 3.96 43.95 11.51 24.29 1.53 1.215 12.04 11.06 53.78 2.42 1.07

15 46 14 -

Base (InternVL) 1.56 2.49 9.00 44.97 37.89 2.38 0.355 25.40 23.96 5.4 MLRS (InternVL) 1.54 2.80 38.24 45.92 86.88 2.27 0.810 33.20 55.74 40.9

Table 3: MC VQA and segmentation results. For each baseline, the model size closest to 8B is chosen. If multiple versions of the same baseline are reported, we take the one with the highest performance. If a baseline is fine-tuned on the corresponding benchmark, fine-tuned performance is reported and marked with an asterisk (*). Metrics in the [0, 1] range are rescaled to the [0, 100] range. In each column, the best, second-best, and third-best results are highlighted in gold, silver, and bronze, respectively. Zero-shot benchmarks are highlighted in blue. (a) MC VQA Method

(b) Segmentation

DM3 DVL RSHR VLRS UR GEO GEO (chdet) XLRS

Method

EarthReason GeoSeg LaSeRS GeoSeg2 GEO

Qwen 40.40* 23.30 23.06 35.90 40.0 InternVL 41.70* 23.90 26.23 48.80 33.3 LLaVA-OneVision 24.50 19.30 - 41.70 - 30.40 GeoChat 10.70 32.12 33.10 TEOChat 23.00 17.20 EarthDial 22.90 30.30 36.99 - 37.80 DVLChat 33.30* GPT 42.30 34.10 44.04 43.90 61.20 41.10

40.05 26.23 37.03 42.00 44.04

47.40 46.20 42.90 22.9 38.10

LISA PixelLM PSALM SegEarth-R1 UniGeoSeg GlaMM SegEarth-R2

60.88* 60.01* 68.30* 70.75* -

4.93 4.07 64.95* 66.51* 67.75* -

26.6* 29.3* 43.6* 57.2*

39.5 0.0 0.0 -

14.11 -

Base (InternVL) 18.60 22.18 34.49 27.36 49.83 39.41 MLRS (InternVL) 20.65 22.83 29.50 23.00 51.87 50.28

37.89 46.20

39.04 51.34

Base (InternVL) MLRS (InternVL)

27.43 67.78

26.2 47.06

31.06 66.93

41.29 69.84

0.08 15.71

(0.498 → 0.519) and plateaus after 3k steps, while temporal VQA peaks within the first 1k steps and then degrades, ending below the base model on VLRSBench (0.274 → 0.230) and RSHR-Bench (0.345 → 0.295). Captioning metrics on OOD benchmarks remain flat. Finally, the two 8B MLRS GRTO runs track each other closely, suggesting robustness. In our 2B backbone ablation, whose OOD performance is visualized in Fig. 5, the trend remains consistent: segmentation and detection continue increasing as data scales, captioning is saturated and remains relatively flat, temporal VQA, hindered by training data diversity, suffers from overfitting, while non-temporal VQA generally increases. The only exception is XLRSBench in non-temporal VQA. MLRS compared to baselines. We show the performance of MLRS compared to general-purpose and specialized VLMs in Tab. 2 for captioning, free-

More with Less: a Large Scale RS-VLM with a Simple Recipe

(a) Zero-shot performance

11

(b) GRTO vs. GRPO ablation

Fig. 4: Performance comparisons. (a) Zero-shot performance: percentage difference between MLRS and the best-performing existing baseline for each dataset and task. For every comparison, we report the competing model and the evaluation metric used. (b) GRTO vs. GRPO ablation: IoU percentage difference between MLRS, where SAM3 is fine-tuned (GRTO), and keeping SAM3 frozen (GRPO) for segmentation datasets.

Fig. 5: Model size ablation. OOD performance vs training steps of the main data scale training, reproduced for InternVL3.5-2B. The following metrics are used: aggregated accuracy for MC VQA, G-Eval for free-form VQA and captioning, aggregated IoU for segmentation, [email protected] for GEOBench-VLM detection, and [email protected] for XLRS-Bench detection.

form VQA and detection, and Tables 3a and 3b for MC VQA and segmentation. We select the MLRS checkpoint independently of the evaluation curves: the reported model is the final checkpoint of the primary training run, evaluated uniformly across all tasks and benchmarks. In captioning, MLRS ranks first in GEOBench-VLM and second in XLRS-Bench – the two zero-shot benchmarks. In the ID benchmarks, it ranks first in SARLANG-1M, and is surpassed by GPT and Qwen on DisasterM3 and DynamicVL, with a fine-tuned version of Qwen taken for the former. In free-form VQA, MLRS is overshadowed by the much larger GPT and fine-tuned versions of Qwen. In detection – where both datasets are zero-shot – MLRS ranks first in GEOBench-VLM and second in XLRSBench, almost surpassing GPT. In MC VQA, of the six zero-shot benchmarks, MLRS ranks first in three and second in one, failing to reach top-3 only in RSHR and VLRS. Zero-shot, it is once again overshadowed by fine-tuned baselines and GPT. Lastly, in segmentation, MLRS ranks first in both zero-shot benchmarks, first in LaSeRS, third in EarthReason, and fails to reach top-3 only in GeoSeg, where it again loses only to fine-tuned baselines. Overall, MLRS ranks first in 7 OOD benchmarks, second in 3, and fails to reach top-3 only on 2. ID, it tends to

12

S.M. Ailuro et al.

be inferior either to GPT, which is much larger, or baselines fine-tuned solely on the corresponding benchmark. Figure 4a compares the zero-shot performance of MLRS with that of the strongest baseline models. Example outputs of MLRS for various tasks are shown in Figure 6. Effects of data scale on the decoder. Figure 4b compares plain GRPO, where SAM3 is kept frozen, with GRTO, where SAM3 is fine-tuned during reinforcement learning, on the segmentation benchmarks. GRTO improves performance on more benchmarks, indicating that adapting the decoder is generally beneficial. However, the frozen-decoder GRPO variant performs substantially better on the GEOBench-VLM zero-shot benchmark, suggesting that decoder fine-tuning can reduce generalization in some settings. Since GRTO yields gains on the majority of segmentation benchmarks, we use the GRTO-trained model as the final MLRS variant.

5

Discussion

Scaling and diversifying training data as a path towards state-of-theart remote sensing. On tasks with the most-diverse sources we observe that improvement over the base model scales well with data, and reaches competitive or state-of-the-art performance. The shape of the learning trajectory appears tied to the diversity of training data available for a domain (Fig. 1b): multi-view VQA, backed by a single source, plateaus at +4%; temporal VQA, backed by only two closely related sources, peaks at +16% after 1k steps and then declines below the base model, consistent with overfitting to a narrow temporal distribution that no longer generalizes; domains backed by three or more heterogeneous sources continue to improve throughout training. Only RGB captioning plateaus because of saturation by base model – no headroom for improvement. Notably, the trends generally do not change when data scaling is applied to a 2B backbone, with the only exception being degrading performance in XLRSBench VQA. Possibly, the 2B model is inherently not powerful enough to process high-resolution images. Overall, this suggests that the results are not dependent on the size of the VLM. MLRS: all-purpose remote sensing grounded in a general-purpose architecture. The MLRS model achieves top-1 and top-2 results on the majority of benchmarks, except multi-temporal ones, explained by a lack of training data diversity. Baselines outperform MLRS only when trained exclusively on the corresponding benchmark, but never in the OOD setting. The only exception is GPT: a vastly larger general-purpose model. Moreover, GPT is superior on benchmarks for whose annotation it was substantially used itself, i.e. DynamicVL [47], RSHR-Bench [8], XLRS-Bench caption subset [41]; therefore its performance on these benchmarks could be biased. It is also notable that MLRS functions on the largest diversity of input data types and tasks across all baselines. Fine-tuning sparsely used components must be done with care. Although GRTO fine-tuning of SAM3 improves performance on a majority of segmentation benchmarks, it also leads to a substantial drop on the GEOBench-

More with Less: a Large Scale RS-VLM with a Simple Recipe

13

Fig. 6: Example outputs of MLRS for five general tasks: captioning, detection, freeform VQA, segmentation, multiple-choice VQA. For detection, the bounding box is visualized on the input image. For segmentation, SAM3 is prompted with the VLM’s output, and the resulting mask is visualized on the input image.

14

S.M. Ailuro et al.

VLM zero-shot benchmark. This points to a potential overfitting risk when adapting task-specific tools, consistent with the concerns raised in B-GRTO [26]. Unlike the VLM, which is exercised across all tasks and therefore receives broad and diverse supervision, SAM3 is invoked only for localization tasks and is updated using a much narrower subset of the training data. Consequently, whether such components should be fine-tuned depends not only on the amount of available data, but also on the diversity and representativeness of the subset that activates them. Limitations and future work. This study has several limitations. First, our scaling experiments use backbones from the same model family, since repeating large-scale training across many VLM families is computationally expensive. Evaluating whether the same trends hold for other general-purpose VLMs is an important direction for future work. Second, although we compare against representative set of VLMs designed for remote sensing, architecture- and data-centric approaches are yet to be directly compared under single controlled environment. Third, stronger evidence for the role of data composition would require additional controlled training runs with different data mixtures, but such experiments are costly at the scale considered here. Finally, source count is only a coarse proxy for data diversity. Given the limited number of task domains and the possibility that some domains simply have more room for improvement than others, our diversity-related trends should be interpreted as suggestive rather than conclusive. Future work should quantify intra-domain diversity more directly, for example using effective sample diversity in embedding space, and test whether such measures predict the observed peak-and-decline behavior across domains.

6

Conclusion

Our findings demonstrate that, given a powerful general-purpose VLM, architectural ingenuity is not necessary to achieve competitive performance in the remote sensing domain. MLRS follows a deliberately simple recipe: retain the architecture of a strong VLM, expose it to a broad mixture of remote sensing tasks and modalities, and train it to decide when to answer directly and when to invoke a localization tool. Despite this simplicity, it achieves competitive or state-of-theart zero-shot performance across diverse benchmarks, suggesting that strong general VLMs can be effectively adapted to remote sensing through large-scale multi-task training. More broadly, our results indicate that data scale and task diversity are central drivers of remote sensing VLM performance. As the amount and variety of training data increase, performance continues to improve, supporting a datacentric alternative to increasingly specialized architectural design. At the same time, our experiments show that tool-specific components such as SAM3 should be adapted carefully, since fine-tuning them on narrower subsets can trade indomain gains for reduced zero-shot generalization. Overall, MLRS points toward a practical path for building general remote sensing VLMs: start from a strong

More with Less: a Large Scale RS-VLM with a Simple Recipe

15

general-purpose model, scale diverse remote sensing supervision, and use simple tool interfaces to extend the model to spatial tasks.

References 1. Ailuro, S.M., Markov, M., Mahdi, M., Boychev, D., Gool, L.V., Paudel, D.P.: Osmda: Openstreetmap-based domain adaptation for remote sensing vlms (2026), https://arxiv.org/abs/2603.11804 4 2. Blushtein-Livnon, R., Rafaeli, O., Ioffe, D., Boger, A., Esquenazi, K.S., Svoray, T.: On the effectiveness of textual prompting with lightweight fine-tuning for sam3 remote sensing segmentation (2026), https://arxiv.org/abs/2512.15564 5 3. Cai, M., Wang, G., Zhang, W., Zhou, G., Zhuang, Y., Zhang, T., Wang, H., Chen, H., Li, J.: Earth-onevision: Extending remote sensing multimodal large language models to more sensor modalities and tasks (2026), https://arxiv.org/abs/ 2606.10819 2, 4 4. Carion, N., Gustafson, L., Hu, Y.T., Debnath, S., Hu, R., Suris, D., Ryali, C., Alwala, K.V., Khedr, H., Huang, A., Lei, J., Ma, T., Guo, B., Kalla, A., Marks, M., Greer, J., Wang, M., Sun, P., Rädle, R., Afouras, T., Mavroudi, E., Xu, K., Wu, T.H., Zhou, Y., Momeni, L., Hazra, R., Ding, S., Vaze, S., Porcher, F., Li, F., Li, S., Kamath, A., Cheng, H.K., Dollár, P., Ravi, N., Saenko, K., Zhang, P., Feichtenhofer, C.: SAM 3: Segment anything with concepts (2025), https: //arxiv.org/abs/2511.16719 2, 6 5. Chen, T., Cao, R., Yu, X., Zhu, L., Ding, C., Ji, D., Chen, C., Zhu, Q., Xu, C., Mao, P., Zang, Y.: SAM3-Adapter: Efficient adaptation of segment anything 3 for camouflage object segmentation, shadow detection, and medical image segmentation (2025), https://arxiv.org/abs/2511.19425 5 6. Cheng, B., Schwing, A.G., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. In: NeurIPS (2021) 4 7. Chu, T., Zhai, Y., Yang, J., Tong, S., Xie, S., Schuurmans, D., Le, Q.V., Levine, S., Ma, Y.: SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In: Forty-second International Conference on Machine Learning (2025), https://openreview.net/forum?id=dYur3yabMj 4 8. Dang, Y., Zhu, M., Wang, D., Zhang, Y., Yang, J., Fan, Q., Yang, Y., Li, W., Miao, F., Gao, Y.: A benchmark for ultra-high-resolution remote sensing mllms. arXiv preprint arXiv:2512.17319 (2025) 7, 12, 35 9. Danish, M., Munir, M.A., Shah, S.R.A., Kuckreja, K., Khan, F.S., Fraccaro, P., Lacoste, A., Khan, S.: Geobench-vlm: Benchmarking vision-language models for geospatial tasks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 7132–7142 (October 2025) 7, 8, 9 10. Doerksen, K., Kerner, H.: Earthshift: a benchmark for measuring robustness to real-world distribution shifts in earth observation (2026), https://arxiv.org/ abs/2605.29330 4 11. Feng, J., Wang, S., Liu, T., Xi, Y., Li, Y.: Urbanllava: A multi-modal large language model for urban intelligence. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 6209–6219 (October 2025) 3, 4 12. Fiaz, M., Debary, H., Fraccaro, P., Paudel, D., Gool, L.V., Khan, F., Khan, S.: Geovlm-r1: Reinforcement fine-tuning for improved remote sensing reasoning (2025), https://arxiv.org/abs/2509.25026 3, 4

16

S.M. Ailuro et al.

13. Hu, E.J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id= nZeVKeeFYf9 8 14. Irvin, J.A., Liu, E.R., Chen, J.C., Dormoy, I., Kim, J., Khanna, S., Zheng, Z., Ermon, S.: Teochat: A large vision-language assistant for temporal earth observation data. In: International Conference on Learning Representations (2025) 3, 4 15. Jiang, L., Pei, Y., oxi Wu, Zhao, Y., Wu, T., Yu, S., Zhang, L., Cai, D.: Geoseg: Training-free reasoning-driven segmentation in remote sensing imagery (2026), https://arxiv.org/abs/2603.03983 7 16. kelu, Y., Nuo, X., Rong, Y., Yingying, X., Zhuoyan, G., Titinunt, K., yi, R., Pu, Z., Jin, W., Ning, W., Chao, L.: Falcon: A remote sensing vision-language foundation model (2025) 4 17. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. arXiv:2304.02643 (2023) 4 18. Kuckreja, K., Danish, M.S., Naseer, M., Das, A., Khan, S., Khan, F.S.: Geochat: Grounded large vision-language model for remote sensing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 27831–27840 (June 2024) 2, 3, 4 19. Li, K., Xin, Z., Pang, L., Pang, C., Deng, Y., Yao, J., Xia, G., Meng, D., Wang, Z., Cao, X.: SegEarth-R1: Geospatial pixel reasoning via large language model (2025). https://doi.org/10.48550/arXiv.2504.09644, https://arxiv.org/abs/2504. 09644 4, 5, 7, 8, 28 20. Li, Z., Muhtar, D., Gu, F., He, Y., Zhang, X., Xiao, P., He, G., Zhu, X.: Lhrsbot-nova: Improved multimodal large language model for remote sensing visionlanguage interpretation. ISPRS Journal of Photogrammetry and Remote Sensing 227, 539–550 (sep 2025). https://doi.org/10.1016/j.isprsjprs.2025.06.003 2, 3, 4 21. Liu, J., Sun, L., Fu, R., Yang, B.: Towards faithful reasoning in remote sensing: A perceptually-grounded geospatial chain-of-thought for vision-language models (2026), https://arxiv.org/abs/2509.22221 4 22. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C.: G-Eval: NLG evaluation using GPT-4 with better human alignment. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 2511–2522. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.emnlp- main.153, https:// aclanthology.org 9 23. Luo, J., Pang, Z., Zhang, Y., Wang, T., Wang, L., Dang, B., Lao, J., Wang, J., Chen, J., Tan, Y., Li, Y.: Skysensegpt: A fine-grained instruction tuning dataset and model for remote sensing vision-language understanding. arXiv preprint arXiv:2406.10100 (2024). https://doi.org/10.48550/arXiv.2406.10100 2, 3, 4 24. Luo, J., Zhang, Y., Yang, X., Wu, K., Zhu, Q., Liang, L., Chen, J., Li, Y.: When large vision-language model meets large remote sensing imagery: Coarse-to-fine text-guided token pruning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 9206–9217 (October 2025) 2, 3, 4 25. Luo, Z., Wang, D., Wang, H., Guo, H., Zhang, J., Du, B.: Vlrs-bench: A visionlanguage reasoning benchmark for remote sensing. arXiv preprint arXiv:2602.07045 (2026) 7, 36

More with Less: a Large Scale RS-VLM with a Simple Recipe

17

26. Markov, M., Ailuro, S.M., Mahdi, M., Van Gool, L., Paudel, D.P.: B-grto: Bootstrapped group relative tool optimization for referring segmentation. arXiv preprint arXiv:2605.23500 (2026). https://doi.org/10.48550/arXiv.2605.23500 5, 6, 14, 26, 27 27. Muhtar, D., Li, Z., Gu, F., Zhang, X., Xiao, P.: Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds.) Computer Vision – ECCV 2024. pp. 440–457. Springer Nature Switzerland, Cham (2025) 3 28. Ni, S., Wang, D., Chen, H., Guo, H., Zhang, N., Zhang, J.: Unigeoseg: Towards unified open-world segmentation for geospatial scenes (2025), https://arxiv.org/ abs/2511.23332 3, 4, 5, 7, 8, 28, 38 29. Ou, R., Hu, Y., Zhang, F., Chen, J., Liu, Y.: Geopix: A multimodal large language model for pixel-level image understanding in remote sensing. IEEE Geoscience and Remote Sensing Magazine 13(3), 324–337 (2025). https://doi.org/10.1109/ MGRS.2025.3560293 3, 4, 5 30. Pang, C., Weng, X., Wu, J., Li, J., Liu, Y., Sun, J., Li, W., Wang, S., Feng, L., Xia, G.S., He, C.: Vhm: Versatile and honest vision language model for remote sensing image analysis. Proceedings of the AAAI Conference on Artificial Intelligence 39(6), 6381–6388 (Apr 2025). https://doi.org/10.1609/aaai.v39i6.32683, https://ojs.aaai.org/index.php/AAAI/article/view/32683 2, 4 31. Quenum, J., Hsieh, W.H., Wu, T.H., Gupta, R., Darrell, T., Chan, D.M.: LISAt: Language-instructed segmentation assistant for satellite imagery. In: The Thirtyninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2026), https://openreview.net/forum?id=X7CxMmmgkb 3, 4, 5 32. Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K.V., Carion, N., Wu, C.Y., Girshick, R., Dollár, P., Feichtenhofer, C.: SAM 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024), https://arxiv.org/ abs/2408.00714 4 33. Shabbir, A., Zumri, M., Bennamoun, M., Khan, F.S., Khan, S.: Geopixel: Pixel grounding large multimodal model in remote sensing. arXiv preprint arXiv:2501.13925 (2025) 3, 4, 5 34. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., Guo, D.: DeepSeekMath: Pushing the limits of mathematical reasoning in open language models (2024), https://arxiv.org/abs/2402.03300 6 35. Shu, Y., Ren, B., Xiong, Z., Paudel, D.P., Van Gool, L., Demir, B., Sebe, N., Rota, P.: Earthmind: Towards multi-granular and multi-sensor earth observation with large multimodal models. arXiv preprint arXiv:2506.01667 (2025). https: //doi.org/10.48550/arXiv.2506.01667 2, 3, 4, 5 36. Shu, Y., Ren, B., Xiong, Z., Zhu, X.X., Demir, B., Sebe, N., Rota, P.: Terrascope: Pixel-grounded visual reasoning for earth observation. In: IEEE conference on Computer Vision and Pattern Recognition (2026) 3, 4, 5 37. Soni, S., Dudhane, A., Debary, H., Fiaz, M., Munir, M.A., Danish, M.S., Fraccaro, P., Watson, C.D., Klein, L.J., Khan, F.S., Khan, S.: Earthdial: Turning multi-sensory earth observations to interactive dialogues. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14303–14313 (June 2025). https://doi.org/10.1109/CVPR52734.2025.01334 2, 3, 4

18

S.M. Ailuro et al.

38. Wang, D., Liu, S., Jiang, W., Wang, F., Liu, Y., Qin, X., Luo, Z., Zhou, C., Guo, H., Zhang, J., Du, B., Tao, D., Zhang, L.: Geozero: Incentivizing reasoning from scratch on geospatial scenes (2026), https://arxiv.org/abs/2511.22645 4, 7, 28 39. Wang, F., Chen, M., Li, Y., Wang, D., Wang, H., Guo, Z., Wang, Z., Boqi, S., Lan, L., Wang, Y., Wang, H., Yang, W., Du, B., Zhang, J.: GeoLLaVA-8k: Scaling remote-sensing multimodal large language models to 8k resolution. In: The Thirtyninth Annual Conference on Neural Information Processing Systems (2026), https: //openreview.net/forum?id=LTgUInLTbP 2, 3, 4, 8, 28 40. Wang, F., Chen, M., Li, Y., Yang, Y., Zhang, Y., Lan, L., Yang, X., Sun, H., Wang, Y., Wang, D., et al.: Geoeyes: On-demand visual focusing for evidencegrounded understanding of ultra-high-resolution remote sensing imagery. arXiv preprint arXiv:2602.14201 (2026) 4 41. Wang, F., Wang, H., Guo, Z., Wang, D., Wang, Y., Chen, M., Ma, Q., Lan, L., Yang, W., Zhang, J., Liu, Z., Sun, M.: Xlrs-bench: Could your multimodal llms understand extremely large ultra-high-resolution remote sensing imagery? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14325–14336 (June 2025) 7, 9, 12, 34 42. Wang, J., Xuan, W., Qi, H., Liu, Z., Liu, K., Wu, Y., Chen, H., Song, J., Xia, J., Zheng, Z., Yokoya, N.: Disasterm3: A remote sensing vision-language dataset for disaster damage assessment and response. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2026), https://openreview.net/forum?id=sQO1ZEQGqX 7, 8, 28 43. Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025). https://doi. org/10.48550/arXiv.2508.18265, https://arxiv.org/abs/2508.18265 5, 8 44. Wang, Y., Liu, Z., Wang, Z., Hu, H., Liu, P., Rao, Y.: Geovista: Web-augmented agentic visual reasoning for geolocalization (2025), https://arxiv.org/abs/2511. 15705 3 45. Wei, Y., Xiao, A., Ren, Y., Zhu, Y., Chen, H., Xia, J., Yokoya, N.: Sarlang1m: A benchmark for vision–language modeling in sar image understanding. IEEE Transactions on Geoscience and Remote Sensing 64, 1–20 (2026). https: //doi.org/10.1109/TGRS.2026.3652099 7, 8, 9, 28 46. Xin, Z., Li, K., Chen, L., Li, W., Xiao, Y., Qiao, H., Zhang, W., Meng, D., Cao, X.: Segearth-r2: Towards comprehensive language-guided segmentation for remote sensing images (2025), https://arxiv.org/abs/2512.20013 4, 5, 7, 8, 28, 37 47. Xuan, W., Wang, J., Qi, H., Chen, Z., Zheng, Z., Zhong, Y., Xia, J., Yokoya, N.: DynamicVL: Benchmarking multimodal large language models for dynamic city understanding. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2026), https://openreview. net/forum?id=zubCrOvUZ4 3, 4, 5, 7, 8, 12, 28 48. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., Qiu, Z.: Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2024) 9 49. Yao, L., Liu, F., Lu, H., Zhang, C., Min, R., Xu, S., Di, S., Peng, P.: Remotereasoner: Towards unifying geospatial reasoning workflow. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 11883–11891 (2026) 4, 5, 6

More with Less: a Large Scale RS-VLM with a Simple Recipe

19

50. Ye, J., Lin, H., Ou, L., Chen, D., Wang, Z., Zhu, Q., He, C., Li, W.: Where am i? cross-view geo-localization with natural language descriptions (2025), https: //arxiv.org/abs/2412.17007 3, 8, 28 51. Yuan, Z., Mou, L., Hua, Y., Zhu, X.X.: Rrsis: Referring remote sensing image segmentation (2024), https://arxiv.org/abs/2306.08625 5 52. Zhan, Y., Xiong, Z., Yuan, Y.: Skyeyegpt: Unifying remote sensing vision-language tasks via instruction tuning with large language model. ISPRS Journal of Photogrammetry and Remote Sensing 221, 64–77 (2025). https://doi.org/https: //doi.org/10.1016/j.isprsjprs.2025.01.020, https://www.sciencedirect. com/science/article/pii/S0924271625000206 2, 4 53. Zhang*, T., Kishore*, V., Wu*, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. In: International Conference on Learning Representations (2020), https://openreview.net/forum?id=SkeHuCVFDr 8 54. Zhang, W., Cai, M., Ning, Y., Zhang, T., Zhuang, Y., Lu, S., Chen, H., Li, J., Mao, X.: Earthgpt-x: A spatial mllm for multilevel multisource remote sensing imagery understanding with visual prompting. IEEE Transactions on Geoscience and Remote Sensing 63, 1–21 (2025). https://doi.org/10.1109/TGRS.2025. 3626941 3, 4 55. Zhang, W., Cai, M., Zhang, T., Zhuang, Y., Li, J., Mao, X.: Earthmarker: A visual prompting multi-modal large language model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing (2024) 2, 3, 4 56. Zhang, W., Cai, M., Zhang, T., Zhuang, Y., Mao, X.: Earthgpt: A universal multimodal large language model for multisensor image comprehension in remote sensing domain. IEEE Transactions on Geoscience and Remote Sensing 62, 1–20 (2024). https://doi.org/10.1109/TGRS.2024.3409624 3, 4 57. Zhang, X., Ge, J., Zheng, Y., Guo, K., Liang, J.: Bridging semantics and geometry: A decoupled lvlm–sam framework for reasoning segmentation in optical remote sensing. ISPRS Journal of Photogrammetry and Remote Sensing 237, 217–235 (2026). https://doi.org/https://doi.org/10.1016/j. isprsjprs.2026.04.036, https://www.sciencedirect.com/science/article/ pii/S0924271626002091 3, 4, 5, 6 58. Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., Huang, F., Zhou, J.: Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176 (2025). https://doi.org/10.48550/arXiv.2506.05176 7, 25 59. Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E.P., Zhang, H., Gonzalez, J.E., Stoica, I.: Judging LLM-as-a-judge with MT-bench and chatbot arena. In: Advances in Neural Information Processing Systems. vol. 36, pp. 46595–46623 (2023) 8 60. Zhou, B., Yang, H., Chen, D., Ye, J., Bai, T., Yu, J., Zhang, S., Lin, D., He, C., Li, W.: Urbench: A comprehensive benchmark for evaluating large multimodal models in multi-view urban scenarios. AAAI (2025) 3, 7, 37 61. Zhu, J., Fu, R., Hu, J., et al., X.N.: Geovista: Visually grounded active perception for ultra-high-resolution remote sensing understanding (2026), https://arxiv. org/abs/2605.14475 3, 4

20

S.M. Ailuro et al.

A

Trends analysis

A.1

Setup, metrics, and covariates

We analyze three training runs: an 8B model trained to 5k steps, a second 8B run with a different random seed trained to 3k steps, and a 2B model trained to 3k steps; all evaluated every 1k steps on the out-of-distribution (OOD) benchmark suite grouped into seven task domains. Unless stated otherwise, statistics are computed at the checkpoint from step 3k. Gain metric. Benchmarks use heterogeneous metrics and several base-model scores are near zero, our primary measure is the normalized gain \Delta _{\mathrm {norm}} = \frac {s_t - s_0}{1 - s_0},

(2)

i.e., the fraction of the remaining headroom recovered by training; all metrics are mapped to [0, 1] beforehand. Absolute gains and log-ratios (with ε = 0.005) are used as sensitivity checks and never change the sign of any reported effect. Training-mix covariates. Each OOD domain is matched to the training rows of the same task type and modality. Two covariates are derived per domain: nsmp , the summed number of training samples per 1k steps, and nsrc , the number of unique underlying sources of labels (datasets may aggregate several sources). Sources shared across tasks are counted only once; detection additionally inherits the 13 segmentation sources because model produce bounding boxes for both tasks (nsrc = 19); and the multi-modal VQA rows intersect heavily at the source level (DynamicVL, DisasterM3, SARLANG1, SARLANG2 contain SAR, GeoSeg-1M contain False-color images), yielding only 5 unique sources despite 16 nominal entries. The resulting counts are: Detection 19, Segmentation 13, RGB Caption 11, Multi-modal VQA 5, Ultra-high-res. VQA 3, Temporal VQA 2, Multi-view VQA 1. We notice, that nsmp and nsrc are only mildly collinear (ρ = 0.39). Domains are not statistically independent: they share underlying sources, and the two 8B runs share initialization. We therefore use permutation tests for all univariate rank correlations, report leave-one-domain-out (LODO) ranges, cluster standard errors by domain in all regressions, and obtain confidence intervals for domain-level means by bootstrap over benchmark×run observations within each domain. All visual error bars in Figs. 8–10 are bootstrap 95% CIs unless noted. We further anayse effect of random seed, model size and base model performance. The hypothesis are: H2 and H1 – behaviour is independent of seed, model size, and nsmp ; H3 and H4 – source diversity and base model performance affect behavior. A.2

H1: Gains are not driven by sampling volume

We find no evidence that nsmp predicts per-domain gains (Fig. 8, left). At the domain level (n=7), Spearman ρ = 0.14 (permutation p = 0.79; bootstrap 95%

More with Less: a Large Scale RS-VLM with a Simple Recipe

21

Fig. 7: OOD normalized-gain trajectories per task domain (panels ordered by decreasing nsrc ). Lines are means over the benchmarks in each domain, bands are min– max ranges across benchmarks; colors denote the three runs. The three qualitative behaviours are visible: sustained improvement in source-rich domains (Detection, Segmentation, Multi-modal VQA), early peak followed by stagnation or decline in sourcepoor domains (Temporal VQA peaks at step 1k–2k in all three runs; Multi-view VQA gains remain ≲ 0.04), and flat-to-negative movement where the base model is already saturated (RGB Caption, BERTScore base ≈ 0.88).

CI [−0.21, 0.36]); Pearson r = 0.10 (p = 0.82). In the multivariate model of Sec. A.6, which controls for source diversity and base performance, the coefficient on log2 nsmp is small and non-significant (β = −0.015 per SD, p = 0.45), and remains non-positive in all sensitivity specifications. A.3

H2: Seed and model scale do not change the behavior

Seed. Comparing the two 8B seeds pairwise over all 25 benchmarks and tasks (Fig. 10a), score differences are not significant at any shared checkpoint (Wilcoxon signed-rank p = 0.11, 0.43, 0.31 at steps 1k, 2k, 3k), and the mean absolute seed difference (0.014) is 7.3× smaller than the mean absolute training gain (0.10). Per-benchmark normalized gains at 3k are nearly identical across seeds (Spearman ρ = 0.965, p < 10−6 ), i.e., not only the magnitudes but the pattern of gains is reproducible. Model size. The 2B model reproduces the same gain structure as the 8B models on the OOD suite (Fig. 10b): per-benchmark gains correlate at ρ = 0.77 (p = 0.003) with the seed-averaged 8B gains. The 2B gains are marginally smaller in magnitude (median difference −0.048, Wilcoxon p = 0.052), driven mainly by detection and ultra-high-resolution VQA (Fig. 7). Trajectory-shape classifications (sustained vs. peaked, Sec. A.4) agree across seeds and across scales on 57% of domains, with all disagreements occurring in low-amplitude domains where the classification is noise-limited.

22

S.M. Ailuro et al.

Fig. 8: Domain-level normalized gain at step 3000 (mean ± bootstrap 95% CI over benchmarks × 3 runs) against the two training-mix covariates. Left (H1): no dependence on sampling volume. Right (H3): positive dependence on the number of unique sources; the visible off-trend point is RGB Caption, whose negative gain is explained by base-model saturation (Fig. 9) rather than lack of diversity.

We conclude that neither seed nor scale qualitatively alters the mix-dependent behavior, and pool all three runs elsewhere in this analysis. A.4

H3: Gains and trajectory shape increase with source diversity

Overall correlation. The domain-level association between gain at 3k and nsrc is positive and moderate (Fig. 8, right): Spearman ρ = 0.54 (permutation p = 0.24 at n=7; bootstrap 95% CI [0.39, 0.64]; LODO range [0.31, 0.83]). The univariate test is underpowered at seven domains and is additionally confounded by basemodel saturation: RGB Caption has 11 sources yet negative gains because its base score is already ≈ 0.67 on average. Controlling for base performance, the partial rank correlation remains positive (ρ = 0.26), and in the multivariate model (Sec. A.6) log2 nsrc carries the largest positive standardized coefficient (β = +0.054 per SD, p = 0.065; +0.068 excluding detection; +0.050 under Huber robust regression) — consistently positive across every specification we tried, suggesting significance, however base performance competes for the shared variance. Restricting the analysis to the two 8B runs to remove any model-scale effect strengthens the association: domain-level ρ = 0.61 (permutation p = 0.17; LODO [0.37, 0.94]), and the multivariate sources coefficient becomes significant (β = +0.069, p = 0.034). Trajectory behaviour. We classify each domain×run trajectory using min– max-normalized scores: drop from peak (peak minus final value) and monotonicity (Spearman correlation of score with step). The source-poor domains exhibit the failure mode: Temporal VQA (nsrc =2) peaks at step 1/2k and declines in all three runs (mean drop 0.42), and Multi-view VQA (nsrc =1) improves only marginally. Tests over the seven domains are directionally consistent but not yet

More with Less: a Large Scale RS-VLM with a Simple Recipe

23

Fig. 9: Normalized gain at step 3000 against base-model score. Small markers: individual benchmarks per run (circle/square/triangle = 8B-s1/8B-s2/2B); large diamonds: domain means with bootstrap 95% CIs; dashed line: OLS fit on benchmark means. Detection benchmarks (near-zero base) and RGB Caption (base ≈ 0.88 BERTScore) anchor the two extremes of the headroom effect.

significant (peak retention vs. nsrc : ρ = 0.52, permutation p = 0.24; sustained vs. peaked domains, one-sided Mann–Whitney p = 0.57), again limited by n=7 and by RGB Caption, whose decline is saturation-driven rather than diversitydriven. We therefore present the behaviour-type result (Fig. 7) as a qualitative, seed- and scale-reproducible observation rather than a confirmatory finding. The decline in temporal VQA after the peak is yet to be explained by, e.g., effects of forgetting. A.5

H4: Gains decrease with base-model performance

Gains correlate negatively with base performance (Fig. 9): Spearman ρ = −0.71 at the domain level (permutation p = 0.088) and ρ = −0.56 over the 12 OOD benchmarks (p = 0.059); in the multivariate model βbase = −0.070 per SD (p = 0.065; −0.087, p = 0.003 excluding detection; −0.069 under Huber). However, the effect is not linear, it is rather clearest at the extremes: RGB Caption is saturated at base (BERTScore ≈ 0.88; gains ≈ 0 or negative in all runs), while detection starts near zero and improves up to ×7.6 in relative terms. The conservative reading is that base-model headroom is a necessary condition for mix-driven gains. A.6

Multivariate model

To disentangle the three covariates we fit, at the benchmark level (12 OOD benchmarks × 3 runs, n = 36), \Delta _{\mathrm {norm}}^{(3k)} = \beta _0 + \beta _1 \, z\!\left (\log _2 n_{\mathrm {src}}\right ) + \beta _2 \, z\!\left (\log _2 n_{\mathrm {smp}}\right ) + \beta _3 \, z\!\left (s_0\right ) + \epsilon , \label {eq:mv}

(3)

24

S.M. Ailuro et al.

Fig. 10: (a) Seed comparison: 8B seed-1 vs. seed-2 scores over 25 benchmarks at steps 1k–3k (color = step); points lie on the identity line. (b) Scale comparison: 2B vs. 8B (seed-mean) normalized gains at 3k per OOD benchmark (colors = domains as in Fig. 9); the pattern is preserved with slightly attenuated 2B magnitudes. (c) Standardized coefficients of the multivariate model (Eq. 3) with cluster-robust 95% CIs, for the full model, excluding detection benchmarks, and under Huber robust regression: source diversity contributes positively, base performance negatively, and sampling volume contributes nothing, across all specifications.

with standard errors clustered by domain (7 clusters), where z(·) denotes standardization. Results are shown in Fig. 10c. The model explains R2 = 0.31 of benchmark-level gain variance with only these three mix/base covariates. Source diversity and base headroom carry comparable, opposite-signed weights, while sampling volume is null — the quantitative counterpart of H1, H3, and H4. Because nsrc and s0 are themselves anti-correlated in our mix (source-rich domains happen to have low-scoring bases), the two coefficients share variance and each is only marginally significant in the full model; their signs and magnitudes are nevertheless stable across outlier removal and robust estimation.

A.7

Outlier handling and limitations

Outliers. XLRS-Bench detection (relative gain up to ×7.6; 2B base score exactly 0) and GeoSeg-Bench2 (2B base 0.004) would dominate any ratio-based analysis; however, the normalized-gain metric ∆norm bounds their influence, and all results were re-checked excluding detection benchmarks and under Huber robust regression, without sign changes. The main limitations are: statistical power – seven task domains cap the resolution of all domain-level univariate tests, and the behaviour-type association in particular should be treated as observational; the source-sharing dependence between domains, which we mitigate with clustered errors, permutation tests, and LODO; and the collinearity between source diversity and base headroom in this particular mix, which prevents a fully clean attribution between H3 and H4 – a targeted ablation varying nsrc at fixed task type would resolve it. Finally, diversity require stricter measure than number of annotation sources nsrc .

More with Less: a Large Scale RS-VLM with a Simple Recipe

B

25

Detailed Multi-Task Reward

For each sampled completion, we assign a scalar reward that combines a small format bonus with a task-specific correctness score: R = \lambda _{\mathrm {fmt}} F + \lambda _{\mathrm {task}} S, \label {eq:reward_general}

(4)

where F ∈ {0, 1} indicates whether the output follows the required interface, S ∈ [0, 1] is the task-specific score, and we use λfmt = 0.1 and λtask = 0.9 during training. For multiple-choice VQA, the model must answer using the strict <answer>...</answer> format. The reward supports both letter-based options, such as A, B, and C, and text-based options, such as class names or true/false labels. Let  be the parsed set of predicted options, A be the target answer set, and Ω be the set of valid options. The format term is one only if the answer can be parsed and all predicted options are valid: F^{\mathrm {mc}} = \mathbbm {1}\left [ \hat {A} \neq \emptyset \;\wedge \; \hat {A} \subseteq \Omega \right ]. \label {eq:mc_format}

(5)

The task score is set IoU between the predicted and target answer sets: S^{\mathrm {mc}} = \begin {cases} \dfrac {|\hat {A} \cap A|}{|\hat {A} \cup A|}, & \text {if } F^{\mathrm {mc}}=1, \\[6pt] 0, & \text {otherwise.} \end {cases} \label {eq:mc_score}

(6)

This handles both single-answer and multi-answer multiple-choice questions. For free-form VQA and captioning, the model must also produce a non-empty answer inside <answer>...</answer>. Let â be the extracted prediction and a be the reference. The format term is F^{\mathrm {text}} = \mathbbm {1}\left [ \hat {a} \text { is non-empty and correctly enclosed in } \texttt {<answer>...</answer>} \right ]. \label {eq:text_format} (7) When the answer is valid, we score it with Qwen3-Embedding-0.6B [58]. The prediction and reference are embedded, last-token pooled, ℓ2 -normalized, and compared with cosine similarity. We map the cosine score from [−1, 1] to [0, 1]: S^{\mathrm {text}} = \begin {cases} \mathrm {clip}_{[0,1]} \left ( \dfrac { 1 + \cos \left ( \phi (\hat {a}), \phi (a) \right ) }{2} \right ), & \text {if } F^{\mathrm {text}}=1, \\[8pt] 0, & \text {otherwise,} \end {cases} \label {eq:text_score}

(8)

where ϕ(·) denotes the frozen Qwen3-Embedding-0.6B encoder. This gives a dense semantic reward for open-ended language outputs instead of relying on exact string matching. For segmentation, the model must invoke the localization tool by producing a valid <segmentation>...</segmentation> call, containing a noun phrase and a list of bounding boxes for each object type. The language output is parsed into

26

S.M. Ailuro et al.

phrase-level prompts, which are passed to SAM to obtain semantic masks. The format term is the validity of this tool call: F^{\mathrm {seg}} = \mathbbm {1}\left [ \text {the segmentation tool call is valid and yields SAM masks} \right ]. \label {eq:seg_format}

(9)

k T Let M̂ = {m̂k }P k=1 be the predicted phrase-level masks and M = {m }k=1 be the ordered target masks. If the tool call is valid and the number of predicted masks matches the number of target masks, we compute ordered semantic IoU:

S^{\mathrm {seg}} = \begin {cases} \dfrac {1}{T} \displaystyle \sum _{k=1}^{T} \mathrm {IoU}\left (\hat {m}^{k}, m^{k}\right ), & \text {if } F^{\mathrm {seg}}=1 \text { and } P=T, \\[12pt] 0, & \text {otherwise.} \end {cases} \label {eq:seg_score}

(10)

Thus, each predicted semantic mask is compared with the target mask at the same position, and completions that produce the wrong number of masks receive zero segmentation score. If the task contains only one foreground semantic mask, this reduces to standard mask IoU: S^{\mathrm {seg}} = \mathrm {IoU}\left (\hat {m}, m\right ). \label {eq:single_seg_score}

(11)

For detection, the model uses the same localization interface and outputs the same structured JSON object inside <segmentation>...</segmentation>. The format term is one only when the JSON is valid, contains the required noun phrase field, and every object contains a valid box: F^{\mathrm {det}} = \mathbbm {1}\left [ \text {the output is valid localization JSON with valid boxes} \right ]. \label {eq:det_format}

(12)

Predicted boxes are converted to image pixels when they are produced in normalized or Qwen-style grid coordinates. Let B̂ = {b̂p }P p=1 be the predicted boxes t T and B = {b }t=1 be the ground-truth boxes. We greedily match predicted and ground-truth boxes by IoU, using each box at most once, and average over the larger number of predicted and target boxes: S^{\mathrm {det}} = \begin {cases} \dfrac {1}{\max (P,T)} \displaystyle \sum _{(p,t)\in \mathcal {M}} \mathrm {IoU}\left (\hat {b}^{p}, b^{t}\right ), & \text {if } F^{\mathrm {det}}=1, \\[12pt] 0, & \text {otherwise,} \end {cases} \label {eq:det_score}

(13)

where M is the set of matched prediction–target pairs. Across all tasks, the same reward form in Eq. (4) is used, while the task-specific score S changes according to the prompt type. This trains the language policy to either answer directly in text or invoke the localization tool when spatial output is required.

C

Group Relative Tool Optimization

We use Group Relative Tool Optimization (GRTO) [26] to jointly adapt the language policy and the SAM3 localization tool. Standard tool-use reinforcement

More with Less: a Large Scale RS-VLM with a Simple Recipe

27

learning updates only the policy while keeping the tool fixed. In contrast, GRTO treats the VLM and tool as a coupled system: the VLM learns when and how to call the tool, while the tool is updated on the valid tool calls produced by the VLM. Let πθ denote the language policy and ψω denote the parametrized tool. Both are initialized from pretrained reference models πθ0 and ψω0 . The goal is to optimize the KL-regularized objective J_{\mathrm {KL\text {-}RL}}(\theta ,\omega ) = \mathbb {E}_{o\sim \pi _{\theta }} \left [ R(o,\psi _{\omega }) \right ] - \beta \, D_{\mathrm {KL}}\!\left (\pi _{\theta }\,\|\,\pi _{\theta _0}\right ), \label {eq:grto_klrl}

(14)

where o is a generated completion and R is the task reward which depends on both policy and tool. For the policy update, we use the GRPO objective. Given a group of G comG pletions {oi }G i=1 sampled from the old policy πold , we compute rewards {Ri }i=1 , their group mean µ, and standard deviation σ. The relative advantage of completion oi is A_i = \frac {R_i-\mu }{\sigma }. \label {eq:grto_advantage}

(15)

The policy objective is J_{\mathrm {GRPO}}(\theta ,\omega _{\mathrm {old}}) = \frac {1}{G} \sum _{i=1}^{G} \sum _{t=1}^{|o_i|} \min \!\left ( r_{i,t}A_i, \mathrm {clip}(r_{i,t},1-\epsilon ,1+\epsilon )A_i \right ) - \beta \, D_{\mathrm {KL}}\!\left (\pi _{\theta }\,\|\,\pi _{\theta _0}\right ), \label {eq:grto_grpo} (16)

where r_{i,t} = \frac { \pi _{\theta }(o_{i,t}\mid o_{i,<t}) }{ \pi _{\mathrm {old}}(o_{i,t}\mid o_{i,<t}) }. \label {eq:grto_ratio}

(17)

The rewards in Eq. (16) are computed using the current frozen copy of the tool, ωold , so that the policy update follows the usual clipped GRPO form. For the tool update, GRTO reuses the same sampled completions. Since the task reward is generally non-differentiable, the tool is optimized with a differentiable surrogate loss L(ψω ) on valid tool calls. The contribution of each rollout is weighted by the stopped-gradient policy likelihood ratio, yielding the GRTO objective J_{\mathrm {GRTO}}(\theta ,\omega ) = J_{\mathrm {GRPO}}(\theta ,\omega _{\mathrm {old}}) - \frac {1}{G} \sum _{i=1}^{G} \left ( \prod _{t=1}^{|o_i|} r'_{i,t} \right ) \mathcal {L}_i(\psi _{\omega }), \label {eq:grto_objective}

(18)

′ where ri,t = stopgrad(ri,t ). Thus, the language policy is updated by the GRPO reward signal, while the SAM3 tool is updated through the supervised auxiliary loss induced by the policy-generated tool calls. SAM3 auxiliary loss. We fine-tune the semantic head of SAM3 using the same BCE + soft-IoU objective as in B-GRTO [26]. For each valid localization call, the VLM produces a noun phrase and bounding box prompts for SAM3. The SAM3 semantic head predicts a foreground logit mask M̂ ∈ RH×W , which is converted into a soft probability mask

S = \sigma (\hat {M}) \in [0,1]^{H\times W}. \label {eq:sam_soft_mask}

(19)

28

S.M. Ailuro et al.

Let M ∈ {0, 1}H×W be the binary ground-truth foreground mask. The auxiliary segmentation loss is \mathcal {L}_{\mathrm {sem}} = \mathcal {L}_{\mathrm {BCE}} + \mathcal {L}_{\mathrm {sIoU}}. \label {eq:sam_sem_loss} (20) The binary cross-entropy term is \mathcal {L}_{\mathrm {BCE}} = -\frac {1}{HW} \sum _{u=1}^{H} \sum _{v=1}^{W} \left [ M_{uv}\log S_{uv} + (1-M_{uv})\log (1-S_{uv}) \right ], \label {eq:sam_bce}

(21)

and the soft-IoU term is \mathcal {L}_{\mathrm {sIoU}} = 1 - \frac { \sum _{u=1}^{H} \sum _{v=1}^{W} S_{uv}M_{uv} }{ \sum _{u=1}^{H} \sum _{v=1}^{W} \left (S_{uv}+M_{uv}-S_{uv}M_{uv}\right ) }. \label {eq:sam_siou}

(22)

This loss is applied only to valid localization rollouts, i.e., completions that produce a non-empty, parseable SAM3 prompt with valid bounding boxes. Invalid localization calls and language-only completions do not contribute to the SAM3 auxiliary loss. In our implementation, mask logits are filtered by the VLM-provided bounding boxes before the loss is computed, ensuring that the tool update remains spatially tied to the regions selected by the VLM.

D

Training data mix

We gather varied set of tasks and modalities: GeoZero corpus [38], consisting of Hard And Instruct parts, for generic RGB; GeoLLaVA [39] for high-res; GeoSeg1M [28], LaSeRS [46], EarthReason [19] for generic and false-colour segmentation; DynamicVL [47], DisasterM3 [42] for temporal tasks; SARLANG-1M [45] for SAR imagery. SARLANG-1M contains captions and free-form VQA from two sources: text produced from pre-annotated bounding boxes, and text produced from co-registred RGB images – we refer to them as SARLANG1 and SARLANG2. We also construct multi-view instruction data from CVG-Text [50], a cross-view geo-localization corpus of co-registered ground-level panoramas, ground-view captions, and satellite images: panorama captions are used directly for captioning, and we deterministically generate two MC-VQA item types — city recognition from a panorama–satellite pair, and cross-view matching, where the model must select which of four satellite views (with same-city hard negatives) depicts the panorama’s location. A total raw pool of collected training data consists of 2.3M image-prompt-answer pairs. We randomly select a 80k subset to ensure all samples are unique during training. To balance the tasks and sources we utilize the following mix: – Multi-choice VQA: GeoLLaVA, GeoZero Hard, GeoZero Instruct, DynamicVL, DisasterM3, VQA constructed from CVG-text – each 3333 samples; – Free-form VQA: GeoZero Hard, GeoZero Instruct, SARLANG1, SARLANG2, DynamicVL, DisasterM3 – each 2500 samples; – Caption: GeoZero Hard, GeoZero Instruct, SARLANG1, SARLANG2, DynamicVL, DisasterM3, CVG-text – each 2857 samples;

More with Less: a Large Scale RS-VLM with a Simple Recipe

29

– Segmentation: GeoSeg-1M, LaSeRs, EarthReason, DynamicVL, DisasterM3 – each 4000 samples; – Detection: GeoZero Hard, GeoZero Instruct – each 2500 samples.

E

Prompts Used

During training, we replace dataset-specific prompts with generic task templates to avoid artificially inflating the model’s performance. In particular, we do not want the model to exploit benchmark-specific phrasing as a shortcut for deciding how to answer. However, some datasets contain prompts that become ambiguous in a multi-task setting. For example, certain localization datasets provide only a description of the target object, without explicitly stating whether the object should be segmented or detected. In such cases, we minimally disambiguate the instruction by adding a task cue, such as prepending “Segment the ...”. This reflects the expected deployment setting, where users specify the task they want the model to solve rather than providing only an underspecified statement. Main MLRS prompt. In addition to the task-specific instruction, we prepend a default system instruction to every prompt seen by the model during both training and inference. This instruction defines the two allowed output modes: a textual answer for language-only tasks and a segmentation tool call for localization tasks, along with their corresponding output formats. Using the same instruction across all settings provides a stable multi-task interface, making the expected behavior explicit and allowing reinforcement learning to focus on improving task performance rather than discovering the output protocol from scratch. You are solving visual reasoning tasks. You may respond in one of two final formats: 1. If the task requires segmenting/localizing objects in the image, call the segmentation tool by outputting: <segmentation>{"noun phrase": "...", "objects": [{"bbox": [x1, y1, x2, y2]}]}</segmentation> The segmentation JSON must contain: - "noun phrase": exactly one referential noun phrase identifying the target. - "objects": a list of objects, each with one "bbox" field. - Coordinates should use the 0-999 image coordinate grid. If the task includes several images, segmentation tool uses the last rgb image. 2. If the task asks for a textual answer and does not require segmentation, output: <answer>your textual answer</answer>

30

S.M. Ailuro et al.

Use <think>...</think> before the final tag when reasoning is useful. Do not use <answer> for segmentation. Do not use <segmentation> for non-segmentation answers.

G-Eval prompt for captioning. You are an expert judge evaluating satellite image captions. Your task is to compare the Predicted caption against the Ground Truth (GT) and assign a score based on object accuracy, counting, and hallucinations. Evaluation Steps: 1. Analyze the Ground Truth for core objects and counts. 2. Check the Prediction for "Imaginary Objects" (Hallucinations) not present in the GT. 3. Verify if object counts and spatial relationships match the GT. 4. Assign a strict score from 1-5 using the rubric below. Scoring Rubric: 1 (Critical Failure): Major hallucination (imaginary objects) or completely wrong scene classification. 2 (Poor): Correct scene type, but severe errors in object counting or wrong object attributes. 3 (Fair): Captures the main gist, but has minor hallucinations or noticeable counting errors. 4 (Good): Accurate objects and counts, with only very minor semantic differences or missing fine details. 5 (Perfect): Exact match in object types, counts, and spatial layout with no hallucinations. Input: Ground Truth: {gt} Predicted: {pred} Score:

Where we insert the ground truth and provided captions in place of {gt}, and {pred} respectively. G-Eval prompt for free-form VQA. You are an expert judge evaluating Visual Question Answering (VQA) outputs. Your task is to compare the Predicted Answer against the Ground Truth (GT) Answer for the given Question and assign a score based on factual correctness, completeness, and hallucinations. Evaluation Steps: 1. Analyze the Question to understand what information is required. 2. Examine the Ground Truth Answer for key facts, values, and constraints.

More with Less: a Large Scale RS-VLM with a Simple Recipe

31

3. Check the Predicted Answer for hallucinations (information not supported by the GT). 4. Verify correctness, precision, and completeness of the Predicted Answer. 5. Assign a strict score from 1-5 using the rubric below. Scoring Rubric: 1 (Critical Failure): Incorrect answer or major hallucination; does not address the question. 2 (Poor): Partially related but mostly incorrect; major factual errors or missing key elements. 3 (Fair): Captures the general idea but contains minor errors, ambiguity, or incomplete details. 4 (Good): Mostly correct and complete; only very minor inaccuracies or omissions. 5 (Perfect): Exact match with the Ground Truth; fully correct, precise, and no hallucinations. Input: Question: {q} Ground Truth Answer: {gt} Predicted Answer: {pred} Score:

Where we insert the question, ground truth, and provided answer in place of {q}, {gt}, and {pred} respectively.

F

Per-sub-task results

Several benchmarks provide fine split on reasoning and sub-tasks dimensions, therefore we report according performance of our base model and MLRS model, see Tables 4, 5, 6, 7, 8, 9.

G

More example outputs

Additional examples of MLRS outputs are provided in Figures 11 and 12.

32

S.M. Ailuro et al.

Fig. 11: Example outputs of MLRS for five general tasks: captioning, detection, freeform VQA, segmentation, multiple-choice VQA. For detection, the bounding box is visualized on the high-resolution input image and zoomed in. For segmentation, SAM3 is prompted with the VLM’s output, and the resulting mask is visualized on the input image.

More with Less: a Large Scale RS-VLM with a Simple Recipe

33

Fig. 12: Example outputs of MLRS for five general tasks: captioning, detection, freeform VQA, segmentation, multiple-choice VQA. The captioning prompt and model output are shortened. For detection, the bounding box is visualized on the input image. For segmentation, SAM3 is prompted with the VLM’s output, and the resulting mask is visualized on the input image. The MC VQA example prompts visually: the target is zoomed in the high-resolution input image.

34

S.M. Ailuro et al.

Table 4: XLRS-Bench [41] multi-choice VQA. Left-to-right order: OC=Overall counting, RC=Regional counting, OLUC=Overall Land use classification, RLUC=Regional Land use classification, OCC=Object classification, OCL=Object color, OMS=Object motion state, OSR=Object spatial relationship, AD=Anomaly Detection and Interpretation, ECR=Environmental condition reasoning, RP=Route planning, RCCD=Counting with changing detection, CCR=Counting with complex reasoning. ’Avg.’ represents the average accuracy across sub-tasks. Column-wise maxima are shown in bold. Method

Perception

Reasoning

Sub-tasks (L-3)

OC RC OLUC RLUC OCC OCL OMS OSR AD ECR RP RCCD CCR Avg.

Remote Sensing MLLMs GeoChat

16.7 29.0

2.0

23.0

21.1

16.8

35.0

24.2 33.0 43.0 10.0

Closed-source MLLMs GPT-4o GPT-4o-mini Claude 3.7 Sonnet Gemini 2.0 Flash

25.0 32.0 23.3 25.0 27.6 22.7 41.7 45.0

15.0 19.0 17.4 38.0

66.0 59.5 68.4 73.5

9.5 40.9 30.5 34.6

11.3 31.0 29.9 27.6

11.7 65.0 63.6 61.7

24.6 23.6 27.6 32.0

35.0 29.0 34.5 43.0

20.0 6.7 27.8 30.0

25.0 30.0 32.6 51.0

32.4 38.1 40.5 48.7

Open-source MLLMs InternLM-XComposer-2.5 21.7 42.0 LLaVA-Next 26.7 40.0 LLaVA-OneVision-7B 25.0 38.0 InternVL3-8B 40.0 39.0 Qwen2-VL-7B 26.7 40.0 LLaVA-OneVision-72B 33.3 38.0 InternVL2.5-8B 38.3 37.0 33.3 40.0 Qwen2.5-VL-7B InternVL3-78B 23.3 49.0 Qwen2.5-VL-72B 33.3 47.0

7.0 5.0 2.0 10.0 11.0 15.0 10.0 31.0 33.0 39.0

68.0 67.0 69.5 71.5 73.0 72.5 77.0 77.0 74.0 80.0

31.8 28.8 35.9 44.5 35.9 36.3 33.4 40.6 42.5 45.3

27.8 32.8 35.3 30.8 34.6 36.3 35.5 40.5 37.4 42.1

6.7 66.7 65.0 65.0 61.7 66.7 65.0 66.7 66.7 65.0

26.0 72.0 81.0 41.0 30.0 69.0 78.0 27.0 25.2 76.0 83.0 24.0 25.2 77.0 82.0 36.0 31.8 70.0 81.0 35.0 35.6 74.0 83.0 28.0 21.6 73.0 83.0 34.0 36.2 68.0 72.0 27.0 30.0 76.0 81.0 40.0 34.0 71.0 74.0 37.0

36.7 35.0 43.3 21.7 46.7 36.7 50.0 38.3 45.0 43.3

47.0 36.0 36.0 50.0 48.0 43.0 43.0 45.0 42.0 42.0

39.1 41.7 42.9 45.6 45.8 46.0 46.2 47.4 49.2 50.2

21.7 20.0 38.3 39.0

48.0 47.0

50.5 78.0

30.1 35.8

33.1 63.3 23.4 54.0 71.0 38.0 33.5 66.7 31.8 70.0 80.0 52.0

23.3 43.3

31.0 39.0 52.0 51.3

Base MLRS

73.0 71.0 64.8 73.0

73.0 71.0 78.4 82.0

-

21.0 22.9

More with Less: a Large Scale RS-VLM with a Simple Recipe

35

Table 5: RSHR-Bench [8] multi-choice VQA. Left-to-right order: Perception—COL=Color Detection, SHP=Shape Recognition, DET=Detection, OC=Object Classification, REL=Object Spatial Relationship, OGD=Object Grounding, RG=Regional Grounding, OCN=Object Counting, RCN=Regional Counting, Avg.=Perception average; Reasoning—AD=Anomaly (single-turn), FP=Future Prediction (multi-image), MRJC=Multi-region Joint Contrast (multi-image), MRJCS=Multi-region Joint Contrast (single-image, multi-box), OSJ=Object State Judgment (single-turn), Avg.=Reasoning average; Multi-turn—MAD=Anomaly, MTFP=Future Prediction, MOSJ=Object State Judgment, Avg.=Multi-turn average. Column-wise maxima are shown in bold. Perception

Model Remote Sensing VLMs EarthDial 41.0 GeoChat 32.5 GeoLLaVA-8K 25.0 25.5 VHM

22.0 22.0 24.0 25.0

21.0 24.0 25.0 26.0

Open-source VLMs InternVL2.5-8B InternVL3.5-8B MiniCPM2_6 Phi-3.5-Vision Qwen2.5-VL-7B Deepseek-VL VILA-HD

25.5 21.5 21.5 25.0 29.5 22.5 40.0

22.0 28.0 28.0 24.0 25.0 22.0 22.0

26.0 26.0 18.0 21.5 30.0 24.0 25.0 25.0 22.0 28.0 21.0 25.0 22.0 37.0

Closed-source VLMs GPT5 GPT-4o GPT-4o-mini Gemini-2.5-pro

29.0 49.5 41.5 55.0

10.0 23.0 16.0 18.0

Base Best eval (1k step) MLRS (5k step)

30.0 32.5 29.5 40.0 25.0 25.0 26.5 55.0

Reasoning

Avg.

COL SHP DET OC REL OGD RG OCN RCN

Multi-turn

Avg.

AD FP MRJC MRJCS OSJ

Avg.

MAD MTFP MOSJ

30.5 25.0 25.0 25.0

27.1 22.9 21.4 22.9

18.0 22.5 25.0 25.0

31.0 29.0 25.0 25.0

28.1 25.9 24.5 25.7

42.0 30.0 30.0 24.0 24.0 0.0 26.0 24.0

29.5 25.5 0.0 26.5

32.0 30.0 34.0 34.0

52.0 32.0 22.0 28.0

37.1 28.3 16.0 27.7

56.7 48.3 25.0 45.0

60.0 46.0 24.7 53.3

73.5 62.9 47.7 46.2

63.4 52.4 32.5 48.2

22.5 29.0 19.5 23.5 25.0 20.5 35.5

24.5 28.5 29.5 25.0 24.5 26.0 26.0

30.0 30.0 34.3 22.9 24.3 28.6 21.4

22.5 26.5 22.0 25.0 26.5 20.5 24.5

20.0 25.0 29.0 25.0 22.0 22.0 24.0

24.3 25.3 27.4 24.5 25.2 23.1 28.0

26.0 20.0 20.0 16.0 26.0 30.0 24.0 22.0 26.0 28.0 22.0 28.0 58.0 30.0

22.5 29.0 35.0 23.5 25.0 50.0 55.0

34.0 34.0 32.0 30.0 10.0 32.0 32.0

20.0 26.0 30.0 22.0 20.0 20.0 58.0

24.5 25.0 30.6 24.3 21.8 30.4 46.6

25.0 30.0 26.7 28.3 21.7 20.0 65.0

28.7 22.7 23.3 24.7 24.0 23.3 57.3

35.6 40.2 31.1 47.0 10.6 33.3 57.6

29.8 31.0 27.0 33.3 18.8 25.5 60.0

23.0 23.0 37.0 15.0 35.5 30.5 29.0 31.5 31.5 31.0 40.0 41.5

24.5 28.0 32.0 32.5

31.4 27.1 28.6 45.7

20.0 22.5 19.5 25.0

23.0 41.0 32.0 25.0

24.5 74.0 58.0 30.2 68.0 56.0 29.1 54.0 54.0 34.9 66.0 32.0

35.0 30.5 31.5 41.5

34.0 32.0 48.0 38.0

66.0 53.4 78.3 64.0 50.1 70.0 54.0 48.3 78.3 50.0 45.5 56.7

73.3 72.0 68.0 60.0

86.4 84.1 75.0 57.6

79.3 75.4 73.8 58.1

50.5 35.0 27.0 36.0 26.0 22.0 22.9 12.0 43.0 30.5 60.0 14.0 46.5 24.0 27.0 34.0 30.5 37.5 35.7 24.5 41.0 33.5 56.0 24.0 48.0 30.0 9.0 30.5 16.5 24.5 10.0 21.0 38.0 25.3 44.0 10.0

10.0 30.0 30.0

12.0 38.0 30.0

58.0 30.8 48.0 37.0 38.0 30.4

66.7 66.7 55.3

50.0 50.0 36.7

52.7 35.7 40.7

41.3 40.0 30.0

36

S.M. Ailuro et al.

Table 6: VLRS-Bench [25] multi-choice VQA. Left-to-right order: CR=Causal Reasoning, CFR=Counterfactual Reasoning, SIR=Semantic Integration Reasoning, MIR=Mechanistic Interaction Reasoning, ST-CFR=Spatiotemporal Counterfactual Reasoning, ST-CCR=Spatiotemporal Causal-Chain Reasoning, SR-ER=Spatiotemporal Evolution Reasoning, ST-CR=Spatiotemporal Consistency Reasoning, PR=Planning Reasoning, ER=Evaluation Reasoning, ST-CSPR=Spatiotemporal Category-State Prediction Reasoning, ST-M-PR=Spatiotemporal Morphological Prediction Reasoning, ST-SU-PR=Spatiotemporal Scenario Uncertainty Prediction Reasoning, ST-SQ-PR=Spatiotemporal Sequence Prediction Reasoning. Column-wise maxima are shown in bold. Cognition

Models

Decision

CR CFR SIR MIR ST-CFR ST-CCR ST-ER ST-CR PR

Prediction

Avg. Score.

ER ST-CS-PR ST-M-PR ST-SU-PR ST-SQ-PR

General MLLMs GPT-5.4 0.412 0.416 0.516 0.516 GPT-5-chat 0.424 0.400 0.472 0.316 GPT-4o-2024-11-20 0.376 0.432 0.420 0.332 GPT-4o-mini 0.428 0.416 0.428 0.328 Gemini-3.1-Pro-Preview 0.420 0.448 0.496 0.396 Gemini-2.5-flash 0.200 0.188 0.264 0.240 Claude-3.5-haiku 0.308 0.316 0.304 0.360 Claude-Opus-4.6 0.272 0.264 0.372 0.408 Grok-2-vision 0.188 0.216 0.288 0.368

0.416 0.276 0.360 0.400 0.476 0.188 0.192 0.452 0.232

0.384 0.352 0.352 0.356 0.436 0.160 0.208 0.392 0.240

0.436 0.380 0.400 0.352 0.460 0.116 0.232 0.336 0.240

0.416 0.368 0.364 0.336 0.456 0.160 0.200 0.300 0.280

0.456 0.484 0.388 0.388 0.286 0.334 0.248 0.304 0.428 0.458 0.232 0.240 0.372 0.370 0.398 0.348 0.252 0.300

0.424 0.388 0.416 0.424 0.400 0.164 0.208 0.264 0.220

0.384 0.276 0.352 0.372 0.360 0.168 0.224 0.320 0.196

0.400 0.280 0.284 0.280 0.428 0.116 0.248 0.396 0.172

0.416 0.292 0.340 0.304 0.432 0.152 0.168 0.340 0.148

0.439 0.356 0.361 0.355 0.436 0.190 0.270 0.350 0.240

Deepseek-vl2 GLM-4.5v LLama-3.2-11B LLama-3.2-90B Qwen2.5-VL-7B Qwen2.5-VL-32B Qwen2.5-VL-72B Qwen3-VL-2B Qwen3-VL-8B Qwen3-VL-32B

0.372 0.392 0.452 0.344 0.268 0.136 0.248 0.312 0.232 0.228 0.244 0.264 0.368 0.364 0.356 0.308 0.256 0.172 0.384 0.176 0.292 0.312 0.368 0.296 0.216 0.300 0.296 0.392 0.350 0.361 0.468 0.185 0.341 0.363 0.472 0.211 0.372 0.416 0.476 0.316

0.144 0.152 0.236 0.224 0.244 0.316 0.204 0.388 0.408

0.216 0.084 0.268 0.324 0.300 0.240 0.428 0.405 0.428

0.200 0.084 0.300 0.248 0.256 0.216 0.412 0.398 0.388

0.148 0.132 0.268 0.216 0.236 0.204 0.336 0.371 0.392

0.416 0.446 0.312 0.340 0.292 0.286 0.408 0.430 0.198 0.238 0.370 0.308 0.402 0.370 0.287 0.260 0.233 0.275 0.416 0.364

0.096 0.132 0.308 0.328 0.276 0.172 0.380 0.381 0.388

0.128 0.172 0.352 0.280 0.284 0.180 0.312 0.401 0.456

0.064 0.180 0.268 0.232 0.236 0.204 0.304 0.401 0.336

0.080 0.112 0.212 0.212 0.156 0.188 0.312 0.384 0.372

0.250 0.190 0.110 0.318 0.249 0.281 0.264 0.330 0.359 0.395

Remote Sensing MLLMs GeoChat 0.280 0.332 0.360 0.308 VHM 0.302 0.297 0.308 0.210 ScoreRS w/ SFT 0.403 0.367 0.421 0.345 ScoreRS w/ RL 0.313 0.338 0.382 0.295

0.294 0.399

0.310 0.335

0.288 0.367

0.284 0.392

0.352 0.356 0.324 0.332 0.382 0.419 0.409 0.371

0.341 0.338

0.320 0.313

0.313 0.382

0.295 0.342

0.331 0.296 0.347 0.355

0.300 0.300 0.300 0.316 0.296 0.268 0.272 0.300 0.248 0.180 0.224 0.240

0.188 0.196 0.176

0.268 0.328 0.232

0.196 0.272 0.236

0.224 0.312 0.244

0.306 0.300 0.276 0.294 0.250 0.250

0.348 0.336 0.236

0.280 0.320 0.264

0.268 0.296 0.252

0.236 0.260 0.188

0.274 0.288 0.230

Base Best eval (2k step) MLRS (5k step)

More with Less: a Large Scale RS-VLM with a Simple Recipe

37

Table 7: UrBench [60] VQA. Left-to-right order: CR=City Retrieval, IR=Image Retrieval, CL-Camera Localization, OR=Orientation, SR=Scene Recognition, RU=Road Understanding, CO=Counting, SC=Scene Comparison, RBR=Role-based Reasoning, TSR=Traffic Sign Reasoning, VPR=Visual Prompt Reasoning, OM=Object Matching, OG=Object Grounding, OAR=Object Attribute Recognition. Column-wise maxima are shown in bold. GeoLocalization

Model CR

IR

Scene Understanding

CL OR SR RU CO

Object Understanding

Scene Reasoning

Overall

SC RBR TSR VPR OM OG OAR

Human Random

30.0 92.6 82.9 85.7 59.2 87.2 94.1 85.1 24.8 23.9 25.1 23.2 17.7 25.7 21.4 25.3

61.6 21.5

69.9 23.5

GPT-4o Gemini-1.5-Flash Claude-3.5-Sonnet

79.2 85.9 35.3 30.7 65.0 66.3 40.1 79.0 79.6 68.2 77.9 28.0 46.5 50.1 69.7 25.9 25.9 24.0 57.9 71.0 29.1 67.7 77.8 75.8 69.8 22.0 39.1 40.9 72.3 55.8 30.8 33.3 52.4 59.0 48.0 81.0 73.7 37.7 66.7 22.0 61.5 45.4

61.2 50.9 55.0

TinyLLaVA 51.9 InternVL2-2B 50.3 InternVL2-4B 55.0 XComposer2-4KHD 61.9 51.6 LLaVA-NeXT-7B-Mistral LLaVA-NeXT-7B-Vicuna 51.2 InstructBLIP-Vicuna-7B 40.4 LLaVA-NeXT-Interleave-7B 57.9 Mantis-LLaMA3-SigLIP 67.0 69.0 Mantis-Idefics2 LLaVA-NeXT-8B 54.4 InternVL2-8B 50.8 65.5 Idefics-2-8B LLaVA-NeXT-13B 52.0 VILA-1.5-13B 62.7 InternVL2-26b 61.3 58.4 LLaVA-NeXT-34B VILA-1.5-40B 70.1 Base MLRS

23.2 24.7 23.8 31.9 24.2 27.4 26.0 27.5 25.9 24.2 24.6 27.2 25.7 25.4 41.6 27.6 32.4 27.0 29.9 27.0 27.0 27.8 26.6 31.8 23.8 26.0 24.5 27.7 33.7 28.6 23.0 32.3 26.0 28.5 62.5 36.8

27.9 8.6 29.0 47.9 23.1 52.3 25.9 55.6 24.0 55.6 23.3 56.1 27.1 33.0 25.5 52.6 27.2 59.2 25.7 50.3 26.0 55.7 25.2 53.0 24.1 52.3 26.7 53.9 24.1 47.7 24.7 65.0 27.8 58.3 27.9 53.6

9.3 47.9 53.1 65.8 47.3 20.2 14.8 50.7 44.5 49.3 44.5 52.6 47.9 50.7 43.9 41.1 49.0 51.7

9.5 28.6 22.1 35.8 34.1 34.3 20.4 37.3 27.4 22.9 34.1 43.0 25.9 33.8 23.9 30.1 21.6 32.1

27.6 30.1 39.4 30.3 28.4 25.9 28.0 48.4 52.4 56.0 24.0 51.4 27.8 25.1 48.6 52.6 53.9 66.7

54.2 34.7 29.9 23.6 95.8 53.1 31.1 60.0 54.8 42.6 26.9 22.9 95.4 52.1 34.8 59.4

87.4 85.7 88.2 95.2 95.5 23.9 24.2 30.6 21.8 22.1

40.3 64.8 73.0 75.5 59.2 49.6 30.7 65.8 67.6 68.9 55.2 74.9 64.7 54.0 66.3 77.9 65.6 76.4

32.7 45.9 56.6 62.6 42.7 51.5 25.7 55.9 41.6 50.6 52.8 54.8 60.4 52.1 43.8 63.8 59.3 55.5

48.6 54.5 62.6 60.8 45.5 54.5 29.3 63.1 57.7 56.3 58.6 62.6 42.8 52.3 46.4 71.2 56.8 61.3

22.9 41.1 25.5 30.3 30.6 30.2 16.2 42.6 43.9 33.1 31.8 27.2 22.9 22.7 37.3 37.6 25.2 34.2 29.6 37.8 39.8 41.7 30.3 32.0 24.8 21.1 31.8 34.8 25.8 32.3 26.1 37.3 28.7 40.5 34.1 48.3

18.8 40.5 40.2 46.9 30.0 31.9 17.3 41.5 38.6 35.6 28.4 41.3 27.4 26.3 38.2 48.4 24.1 39.5

29.9 41.2 43.5 47.8 39.2 37.1 27.6 40.4 45.3 40.7 38.8 48.8 42.7 46.5 45.8 46.0 43.7 53.1

75.3 64.2 72.5 29.2 31.4 78.6 66.9 76.1 32.9 35.2

42.6 47.4

49.8 51.9

Table 8: LaSeRS [46] segmentation. Metric is IoU/cIoU Column-wise maxima are shown in bold. Model

Hierarchical Segmentation Granularity Target Multiplicity Reasoning Requirements Linguistic Variability Semantic Instance Part Single Multiple Explicit Implicit Short Long

Avg.

LISA-7B 26.4/23.2 20.5/25.0 27.0/24.5 22.3/25.6 LISA-13B PixelLM-7B 32.0/32.8 26.6/30.0 PixelLM-13B 31.6/34.0 27.5/30.2 GLaMM-ft-7B 44.8/47.9 41.2/48.3 M 2 A-7B 30.1/33.0 23.0/24.8 GeoPixel-8B 51.4/57.2 44.1/49.3 SegEarth-R2-3B 60.2/71.8 65.4/70.3

16.1/11.6 17.7/13.1 13.2/16.5 15.8/17.6 32.6/42.7 18.6/17.2 43.9/52.4 64.8/68.3

37.3/32.2 18.2/22.4 27.1/24.3 38.4/34.2 19.9/23.5 27.1/25.5 44.3/40.4 20.2/23.5 25.0/23.1 42.2/40.5 20.9/22.4 26.3/24.4 47.3/50.3 32.2/41.0 59.1/60.3 45.4/37.6 20.9/24.8 35.8/30.4 55.0/45.8 49.2/49.7 66.5/61.3 55.1/69.2 38.3/56.2 78.4/80.4

21.5/25.6 22.6/25.8 23.9/21.9 25.9/22.1 42.6/44.8 23.3/26.7 41.1/58.3 42.8/59.7

34.1/27.8 35.2/28.0 41.6/38.9 42.0/39.1 50.4/54.8 35.8/32.8 51.1/59.3 60.2/69.9

40.2/39.4 41.3/32.6 71.1/70.7 72.2/70.0

15.4/19.7 51.3/63.1

37.9/35.1 1.3/1.3 33.2/27.4 67.3/59.2 59.5/58.7 67.4/65.6

30.7/23.7 68.1/67.0

36.4/45.8 42.9/35.6 31.1/29.0 70.1/74.5 75.2/77.9 66.9/67.4

Base MLRS

38.4/33.9 26.6/25.1 38.4/34.3 27.6/26.1 37.1/34.5 29.3/29.1 37.1/34.5 29.9/29.4 42.6/44.8 43.6/48.3 41.5/36.7 30.5/29.3 51.4/63.2 50.4/55.2 50.1/65.7 57.2/67.9

38

S.M. Ailuro et al.

Table 9: GeoSeg-Bench [28] segmentation. The best results are shown in bold. Method LISA PixelLM PSALM HIPIE SegLLM Geopixel Geopix RemoteSAM Earthmind LISAT Segearth-R1

Interactive

Referring

Reasoning

cIoU gIoU cIoU gIoU cIoU gIoU 2.52 3.12 3.53 4.56 7.09 5.77 0.08 0.11 6.04 5.80 6.36 6.30 6.35 10.83 31.77 18.91 11.88 9.27 6.01 12.06 29.25 39.14 7.90 11.76 8.97 17.72 15.90 35.27 11.06 16.41 17.21 18.71 37.34 40.14 27.36 26.71 15.28 17.63 28.81 28.52 20.80 18.25 4.85 6.49 12.06 29.31 8.09 9.01 16.38 16.57 44.53 46.48 31.01 28.80 4.27 5.52 37.87 40.14 22.06 20.09 4.88 5.28 14.65 9.39 12.72 11.97

PSALM-ft 70.78 74.10 68.70 71.15 47.53 49.59 Geopixel-ft 42.62 46.48 42.70 45.50 30.25 28.51 Earthmnind-ft 67.74 70.89 48.09 49.24 36.84 25.71 LISAT-ft 68.43 73.00 59.82 62.46 41.53 31.25 Segearth-R1-ft 72.09 75.00 70.76 72.98 53.31 51.56 UniGeoSeg-ft 74.44 75.56 72.93 74.58 58.35 53.12 Base MLRS

43.6 42.3

43.5 46.5

19.7 53.0

18.2 57.0

21.2 35.0

16.9 37.7