SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

artificialintelligence knowledgerepresentation reasoning

artificial intelligence, reasoning, knowledge representation

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

arXiv:2606.09669v1 [cs.AI] 8 Jun 2026

Hongcheng Gao∗†1,4 , Hailong Qu∗2 , Jingyi Tang3 , Jiahao Wang5 , Zihao Huang6 Hengkang Qiao2 , Shihong Huang3 , Junming Yang7 , Yi Li1 , Hongyixuan Yuan2 Wenjie Li8 , Bohan Zeng3 , Wenbo Li9 , Bo Wang6 , Jianhui Liu10 , Olive Huang3 Haoyang Huang9 , Wentao Zhang3 , Guoqing Huang2 , Nan Duan9 , Yinpeng Dong†1 1

Tsinghua University 2 Chongqing University 3 Peking University 4 ZenoMind AI Xi’an Jiaotong University 6 Beijing Institute of Technology 7 Southeast University 8 Shanghai Jiao Tong University 9 Joy Future Academy 10 The University of Hong Kong 5

Project Page: https://spatial-world.github.io

Abstract Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However, existing benchmarks predominantly rely on passive evaluation (e.g., static VQA) or simulator-specific pipelines, failing to assess general interactive spatial understanding. We introduce S PATIALW ORLD, a unified benchmark designed specifically for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks. Integrating eight heterogeneous simulation backends under a shared, simulator-agnostic protocol, S PATIALW ORLD features 760 humanannotated tasks across diverse domains (e.g., household routines, travel, social collaboration). Agents must solve tasks under vision-only partial observability, actively gathering egocentric visual evidence and expressing decisions via a unified, text-based action interface native to MLLMs. For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier. Evaluating 15 advanced agents reveals that robust spatial task solving remains challenging: the strongest model, GPT-5, achieves an average task success rate (TSR) of only 17.4%, while the leading open-source model, Qwen-3.5, reaches 14.1%. Further analysis exposes a clear mismatch between task success and execution efficiency, alongside substantial domain-specific performance variations. These bottlenecks in active exploration and long-horizon planning position S PATIALW ORLD as a rigorous testbed for future spatial agents.

Introduction

Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive, understand, and operate within the physical world [48, 1, 12]. Existing benchmarks for spatial reasoning predominantly adopt a passive evaluation paradigm, such as static Visual Question Answering (VQA) [30, 28, 16, 71, 47] or the understanding of pre-recorded videos [79, 41, 69]. Although these tasks assess the basic understanding of models regarding spatial relations, object layouts, and scene structures, they struggle to capture the interactive and dynamic nature of spatial understanding in real-world environments. Since physical spaces are partially observable, agents cannot acquire complete information from a single view. Instead, they must actively navigate to gather progressive visual evidence, update spatial beliefs [89, 6, 9], and plan subsequent actions. ∗ Equal contribution. † Corresponding author.

Preprint.

Therefore, evaluating MLLM spatial reasoning must move beyond static scene recognition to assess their capacity for dynamic exploration and interactive task completion. Existing embodied benchmarks [54, 80, 11, 35] provide important interactive testbeds for navigation, manipulation, and task execution, yet many of them are designed around simulator-specific embodiments [54, 85, 80], sensor assumptions [53, 34, 35], action interfaces [85, 11, 35, 44], or execution pipelines [54, 85, 80]. This makes it difficult to determine whether task success reflects general interactive spatial reasoning or adaptation to a particular simulator or action space [56]. The rapid progress of general MLLMs therefore raises a different evaluation question: can an off-theshelf multimodal model, without being trained for a specific simulator, solve spatial tasks through egocentric visual observation, language-grounded high-level decisions, and closed-loop interaction across heterogeneous 3D environments? Answering this question requires an evaluation regime with three key properties. First, agents should operate under vision-only partial observability, without relying on additional sensor inputs or privileged state information [53, 34]. Second, the action interface should be native to MLLMs: expressing high-level navigation, viewpoint, interaction, and task-control decisions through a text-based action space enables the model to decompose and solve complex real-world tasks via explicit chain-of-thought reasoning [45]. Third, the protocol should be simulator-agnostic, using a unified interaction interface across environments rather than action designs deeply coupled with a single backend [44, 36, 22]. Such a highly decoupled paradigm eliminates the interference of low-level simulator characteristics, thereby providing a rigorous and faithful evaluation of the model’s capacity for active exploration and decision-making based solely on visual observations and instructions. To realize this objective, we introduce S PATIALW ORLD, a unified benchmark designed specifically for evaluating the interactive spatial understanding of multimodal agents in 3D environments. As summarized in Table 2, S PATIALW ORLD contains 760 human-annotated tasks that span household routines, work and study, entertainment, travel, social collaboration, and digital spatial games. These tasks are instantiated across eight simulation backends, including AI2-THOR [32], ProcTHOR [13], VirtualHome [50], CARLA [14], EmbodiedCity [21], their multi-agent variants, and lightweight environments for 3D games. S PATIALW ORLD wraps these heterogeneous platforms into a unified end-to-end evaluation framework with shared interfaces for observation, action, and verification. This design allows S PATIALW ORLD to diagnose failures across complementary forms of 3D reasoning, rather than reducing the performance of agents to a single score specific to a simulator. By abstracting away the underlying complexities of disparate simulators, this unified architecture enables us to rigorously assess spatial reasoning under constraints that closely mimic real-world interactions. In contrast to prior benchmarks [75, 8, 80], our framework ensures that the performance of agents is evaluated reliably based on perceptual constraints that match tasks in the real world. To guarantee this reliability, each task is paired with an initial-state configuration validated by human annotators, a reference trajectory, and a task-specific terminal-state verifier. These artifacts allow us to measure not only whether an agent reaches the goal state, but also whether this achievement is realized through an efficient and interpretable sequence of actions. We conduct extensive experiments on S PATIALW ORLD with fifteen advanced multimodal agents from both open-source and proprietary model families. Our systematic evaluation reveals three major findings. First, current agents remain far from reliable in solving 3D tasks: across the full benchmark, the strongest model, GPT-5, achieves an average TSR of only 17.4%, while the best open-source model, Qwen-3.5-397B-A17B, reaches 14.1%. Second, there is a clear mismatch between task success and execution efficiency: models with a higher TSR do not necessarily achieve higher efficiency, which suggests that success is often accompanied by redundant exploration or shortcuts dependent on the task. Third, the rankings of models vary substantially across domains: GPT-5 leads daily household, travel, and social collaboration tasks, Qwen-3.5-397B-A17B ties GPT-5 in Work & Study and leads physical entertainment, and Gemini-3.1-Pro achieves the highest scores in digital games. These findings demonstrate that S PATIALW ORLD exposes multiple, separable bottlenecks in spatial reasoning, long-horizon planning, and action execution, rather than reducing the evaluation of 3D tasks to a single score on a leaderboard.

S PATIALW ORLD Benchmark

We formalize the task as a vision-only POMDP (§2.1) and present S PATIALW ORLD’s distinguishing design principles from existing benchmarks (§2.2). We then describe the system architecture—a 2

Environments Outdoor Navigation

Indoor Simulation

AI2-THOR

ProcTHOR

VirtualHome

EmbodiedCity

CARLA

Observation Interface

Digital Game Environments

Snake 3D

Maze 3D

Evaluation

Action Interface

Efficiency

Control:{ type: “move”, length: 0.4, pos: [45, 90, 90] }

Agent Localization

View Tranformation

Execution

Control Translation

Block 3D

Predicted Step

MLLM-based Agent Dialogue History

Model Response

User (t=1): please help me …

Plan ＆ Reasoning

move forward

Observation (t=1): <image>

The current view is a bedroom with a ceiling light on, a bed with a blue blanket and green pillows, a book …

turn right pick up

Plan (t=1): Let me think … Action (t=1): “move forward” …

Golden Step

Action

Action Prediction Prediction Action

done

Action: pickup(pot)

…

Ratio of golden steps to actual agent steps.

Accuracy Light (Bedroom): Off Key chain location: Bedroom Bed State: Clean

Final Result: Fail Satisfaction of target state requirements.

Figure 1: SpatialWorld is a scalable, general-purpose evaluation framework for multimodal agents, supporting end-to-end task solving and structured plan generation. It unifies diverse 3D backends under a standardized observation-action interface, enabling rigorous assessment of interactive spatial reasoning via reproducible benchmarks and automated efficiency metrics. unified observation-action interface across heterogeneous simulators and execution-based evaluation (§2.3)—followed by the construction pipeline covering task taxonomy and data annotation (§2.4). 2.1 Task Formulation Each task in S PATIALW ORLD is formulated as a partially observable Markov decision process (POMDP) defined by the tuple ⟨S, O, A, T , Ω, R⟩. At step t, the agent receives a natural-language goal g and a raw egocentric RGB observation ot ∈ O, strictly without privileged state signals (e.g., depth or global maps). Given the trajectory history Ht = (o1 , a1 , . . . , ot ), the MLLM-based policy πθ predicts the next high-level action at ∼ πθ (at | Ht , g) ∈ A. The simulator then executes at , transitions the hidden environment state st+1 , and renders the next visual observation ot+1 : st+1 ∼ T (st+1 | st , at ),

ot+1 ∼ Ω(ot+1 | st+1 ).

(1)

This dynamic interaction continues until the agent executes EndTask or exhausts the step budget. By enforcing this strict vision-only, multi-turn formulation, S PATIALW ORLD ensures agents must reason under the exact same perceptual conditions as a human operator, distinguishing it from benchmarks that rely on privileged state inputs. 2.2

Benchmark Protocol

To rigorously evaluate the active spatial reasoning capabilities of general MLLMs, we introduce S PATIALW ORLD. As shown in Table 1, existing spatial benchmarks typically fall into two categories: static ImageQA/VideoQA datasets that fail to capture dynamic environmental interactions, and simulator-coupled embodied frameworks that rely heavily on non-visual metadata and low-level action parameters. Addressing these limitations, S PATIALW ORLD introduces a unified, interactive evaluation paradigm guided by four core design principles: (1) Pure egocentric vision: As highlighted in Table 1, agents receive strictly first-person, vision-only observations without any privileged state information (e.g., ground-truth object coordinates or semantic metadata). This guarantees a genuine evaluation of the coupling between visual perception and spatial reasoning. (2) Cross-platform unification: We abstract simulator-specific complexities into a unified interface 3

：

System prompt 1). where we can go? 2).

collect environments

You are an intelligent driver in the CARLA autonomous driving simulation environment. Your goal is to complete navigation tasks through visual observation and step-by-step actions.......

how to move?

3). how to interact? 4). Linking to real-world

learn tutorials

Instruction

validate on environments

learn tutorials

validation checklist

Task Instrution

check file completeness

write instructions

：

I'm a bit hungry now. Can you see the table with the sun umbrella in front of you? Walk up to the first table in front of you. I'd like to have a good meal here.

write instructions

Carla_001/ images/

"success_conditions": { "type": ... , "object_type": ..., "receptacle_type": ... }

completeness

step0.png

......

write success conditions

success state

absence

log.json

max step object state

task.json

Validate pass/fail against success conditions pass

validate on environments succeed?

success_conditions

fail

Re-verification by other annotators pass

fail?

cross-validate on environments

mechanics

location

task.json "Tasks": Carla_001, "instructions": ......, "scene": ......, "step_count": ......, "success_conditions": xxx

"success_conditions": { "max step": ... , "object_state": ..., "location": ..., "mechanics":... }

write success conditions

fail

Figure 2: Data construction pipeline of SpatialWorld. We first collect a series of environments, have annotators learn tutorials and write instructions, define success conditions, and then calibrate the data through automated execution validation in virtual environments and human cross-validation. Table 1: Spatial benchmark comparison. Representative spatial ImageQA, VideoQA, and embodiedagent benchmarks differ in key properties motivating S PATIALW ORLD: unified cross-platform interface, interactivity, first-person observations, vision-only inputs, and language-form outputs. ✓ and ✘ indicate presence and absence. Further comparison in Appendix A.1. Instances

Unified crossplatform interface

Interactive env.

First-person observation

Vision-only input

Language-form output

SpatialEval [66] 3DSRBench [46] EmbSpatial-Bench [16] SpatialScore [71]

4635 2772 3640 5025

✘ ✘ ✘ ✘

✘ ✘ ✓ ✘

✓ ✓ ✓ ✓

VideoQA

SpatialBench [76] SITE [69] VSI-Bench [79]

3193 8068 5130

✘ ✘ ✘

✓ ✘ ✓

✓ ✓ ✓

Embodied Bench

ALFRED [54] VLABench [85]

25.7k 100

✘ ✘

✓ ✓

✓ ✘

✘ ✘

Ours

S PATIALW ORLD

760

✓

Type

Benchmark

ImageQA

driven by standardized language-form outputs. This enables the direct evaluation of general MLLMs across distinct domains while fully preserving the native physical challenges of each underlying environment. (3) Factored complexity: We systematically decouple photorealistic visual semantics from pure geometric reasoning by incorporating abstract 3D games alongside daily embodied setups. This factorization allows us to pinpoint specific bottlenecks in a model’s spatial cognition without confounding variables. (4) Execution-based verification: Success is objectively verified via terminal environment states rather than strict adherence to predefined action trajectories. This approach accommodates the open-ended exploration and diverse reasoning paths characteristic of autonomous MLLM agents. 2.3 SpatialWorld Architecture To systematically decouple environment execution from agent decision-making, S PATIALW ORLD introduces a modular architecture comprising five standardized components, as illustrated in Fig. 1. In this closed-loop pipeline, the Environment and Verification interfaces rigorously manage task initialization and deterministic success checking, while the Agent Module acts as the central MLLMbased reasoning engine. Crucially, to bridge the gap between diverse simulator backends and a 4

(a) Environment Initialization Mode A:Direct State Loading

Load pre-configured scene

Mode B:Action-List Exec

Load base environment

Execute init_actions[] sequentially

Location

Posture

State

Environment Ready

init_actions: [ move(forward, 2.0), rotate(left, 90), open(Fridge) ]

Category sequentia Action lly Move(direction, [granularity]) Navigation

Scene

Egocentric View

Move MLLM agents

Viewpoint & Ready Posture

Rotate(direction, [angle]), Tilt(direction, [angle]), ChangePosture(pose)

Adjust view, heading, or stance

Interaction

Pick/Place(obj, [target]), ChangeState(obj, state), Manipulate(obj, action)

Change object states

Task-Control & EndTask(status), Coordination Communicate(msg)

Signal execution status and communicate

Environment Ready

(d) Unified Action-to-Code Mapping

(b) Unified Observation Interface

3D environment

Description

RGB Observation

MLLM agent

Unified Action: Move (forward, medium)

VirtualHome: WalkForward(Medium) → <char0> [walkforward]

Action Interface

CARLA: FollowLane(distance=10)→ vehicle.apply_control(control)

ProcTHOR: controller.step( action= "MoveAhead", moveMagnitude= 0.5 )

Figure 3: The Observation and Action Interfaces. (a) Flexible environment initialization via direct state loading or action-list execution. (b) A unified interface providing standardized egocentric RGB observations. (c) A structured, unified action space A. (d) Action-to-code mapping that translates unified actions into environment-specific commands, enabling cross-simulator deployment. unified agent policy, the Observation and Action interfaces are designed as a strictly defined I/O bottleneck: they encapsulate all heterogeneous sensory rendering and physics-engine executions into a standardized interaction protocol. As illustrated in Fig. 3, these interfaces serve as a crucial bridge by normalizing complex 3D environments into egocentric visual observations and translating high-level decisions into simulator-specific execution codes. Guided by this architectural abstraction, we next formally detail the unified observation and action spaces exposed to the agent. Observation Space. At each step, the agent receives a single egocentric RGB screenshot at the simulator’s native resolution. No auxiliary modality (e.g., depth map, optical flow, semantic segmentation, or global occupancy map) is available. This vision-only constraint is the primary departure from VLA-style benchmarks, which typically inject privileged sensor states (joint angles, object lists, navigation graphs) alongside visual input, and from offline 3D benchmarks, which supply pre-captured multi-view scans or videos rather than requiring active information gathering. Action Space. Rather than requiring low-level continuous motor commands (e.g., joint torques or velocity vectors), we expose a unified high-level action space A that abstracts heterogeneous simulator backends behind a common symbolic interface. This design choice serves two purposes: (i) it enables direct evaluation of off-the-shelf MLLMs without task-specific fine-tuning, and (ii) it produces interpretable, language-grounded reasoning traces as a natural byproduct of decision-making. Concretely, as shown in Fig. 3, A encompasses four high-level functional categories: (i) Navigation (e.g., Move), (ii) Viewpoint & Posture (e.g., Rotate), (iii) Interaction (e.g., Pick/Place), and (iv) Task-Control & Coordination (e.g., EndTask). The Action Interface translates these unified primitives into simulator-specific execution calls, ensuring that a single agent policy generalizes across diverse environments without modification. Detailed definitions of all available actions and parameters are deferred to Appendix F. The Action Interface (component iv) maps these unified primitives to simulator-specific execution calls, ensuring that a single agent policy generalizes across environments without modification (see Fig. 3 for a complete overview). Environment Suite. We integrate eight backends under a unified agent-side abstraction to ensure cross-environment comparisons evaluate genuine real-world interactive spatial understanding rather than interface bias. The suite is organized into three families: Indoor Simulation (AI2-THOR [32], ProcTHOR [13], VirtualHome [50]) provides explicit physical affordances to test fine-grained object grounding, temporally ordered routines, and multi-agent coordination. Outdoor Navigation (CARLA [14], EmbodiedCity) extends to macroscopic scales, evaluating long-range route planning and progress estimation across dynamic urban and aerial topologies. Finally, while realistic simulators are indispensable, their difficulty is often entangled with photorealistic semantics and natural scene priors. To address this, we specifically implemented Custom Digital Games (e.g., Block3D, Snake3D, 5

Environment

Daily

Work

Entertain.

Travel

Social

Total

AI2-THOR ProcTHOR VirtualHome CARLA EmbodiedCity Multi-AI2THOR Multi-ProcTHOR 3D Games

219 92 27 0 12 0 0 0

41 10 8 0 0 0 0 0

40 23 3 0 2 0 0 105

11 2 0 80 39 0 0 0

0 0 0 0 0 29 17 0

311 127 38 80 53 29 17 105

Total

350

173

132

760

Task count

Table 2: Scenario distribution. Task distribution across environments and scenario categories. “Social” denotes Social Collaboration; “Entertain.” denotes Entertainment.

300 200 100 0

Dail

k Wor

r Ente

l l . tain Trave Socia

Scenario category Figure 4: Task-category counts. Task distribution across different categories.

Rubik’s Cube) as controlled closed-loop probes. By stripping away visual shortcuts, these lightweight environments isolate the abstract spatial logic and topological reasoning that fundamentally underpin real-world interactive spatial understanding. Detailed descriptions are provided in Appendix D. Execution-Based Evaluation. Following OSWorld [75] and Spider2-V [8], S PATIALW ORLD adopts terminal-state verification rather than static trajectory matching. A custom verifier Vi queries the final state to assess performance via two complementary metrics. The primary metric is TSR, which PN (i) measures the fraction of tasks where the terminal goal is fully satisfied: TSR = N1 i=1 1[Vi (sT ) = (i) 1], where sT is the terminal state for task i (task-specific adaptations, e.g., for Snake3D, are detailed in Appendix A.2). To evaluate efficiency beyond success, we measure Step Efficiency (SE): P n∗ 1 ∗ i SE = |S| i∈S ni , where S is the set of successful tasks, ni is the agent’s step count, and ni is the human-annotated reference length. Jointly reporting TSR and SE distinguishes efficient agents from exhaustive trial-and-error. 2.4 Benchmark Construction To systematically standardize the evaluation of spatial intelligence, S PATIALW ORLD establishes a comprehensive benchmark construction protocol for its 760 tasks. Specifically, we first define a rigorous task taxonomy structured along two core dimensions: Scenario Categories and Complexity Levels, ensuring a diverse and hierarchical coverage of spatial capabilities. Guided by this taxonomy, we then employ a unified Data Construction pipeline to guarantee the high quality, consistency, and reproducibility of the entire dataset. Scenario Categories. S PATIALW ORLD separates everyday embodied operation from abstract spatial reasoning while keeping both under the same closed-loop evaluation protocol. The physical portion covers household routines, study and work activities, entertainment scenarios, travel-oriented navigation, and social collaboration, so that the benchmark spans object-centric manipulation, room-scale exploration, large-scale movement, and multi-agent coordination rather than a single simulatorspecific skill. The digital portion introduces 3D games as a complementary family: these tasks remove photorealistic semantics and instead emphasize geometric counting, maze planning, state tracking, and spatial transformation. Table 2 summarizes how these scenario categories are distributed across the eight environments. Complexity Levels. Each task carries one of three complexity labels that reflect the cognitive demands placed on the agent. Navigation tasks require the agent to explore the 3D environment and reach a target location or object, without manipulating environment state. Interaction tasks require object-level state changes—picking, placing, opening, or toggling objects—but do not demand extensive spatial exploration. Hybrid tasks combine long-horizon navigation with multi-step manipulation, demanding both spatial exploration and fine-grained physical interaction. Data Construction. As illustrated in Fig. 2, we adopt a unified data construction pipeline across all environments in S PATIALW ORLD. For each task, the construction process consists of three stages: (i) task design, where annotators define the natural-language instruction and configure the initial environment state; (ii) human execution, where trained annotators independently solve each task in the simulator, recording the ground-truth terminal state and reference action sequence; and (iii) verification, where separate expert reviewers cross-check the task feasibility, instruction clarity, and evaluation script correctness. All verifier logic and success conditions are validated through rigorous inter-annotator cross-checking, further ensuring the consistency, accuracy, reproducibility, and unambiguity of the evaluation signal. Representative examples of the annotated evaluation scripts and human-validated cases are provided in Appendix E. 6

Table 3: Performance Evaluation. Main-benchmark TSR (%) across task categories for 15 evaluated models. Bold and underlined entries denote the best and second-best per column. Physical categories follow the benchmark scenario taxonomy; digital corresponds to the 3D game suite. Physical Overall is the weighted average of Daily, Work, Entertain., Travel, and Social categories. Physical

Model Daily

Work

Entertain.

Digital

Travel

Social

Overall

Entertain.

0.8 1.5 4.5 3.8 4.5 1.5 1.5 0.0 3.8

2.2 4.3 10.9 10.9 19.6 13.0 0.0 0.0 17.4

3.4 4.9 6.9 6.1 12.2 4.0 2.7 0.9 9.2

7.6 7.9 5.0 28.3 26.0 14.5 8.1 3.3 31.0

3.8 6.1 4.5 6.8 3.8 3.0

10.9 4.3 8.7 34.8 6.5 13.0

6.7 7.2 9.2 14.4 6.6 5.8

32.6 38.1 39.0 36.4 11.9 24.8

(A) Open-Source Models Qwen2.5-VL-72B [3] Qwen3-VL-30B-A3B [78] Qwen3-VL-235B-Instruct [78] Qwen3-VL-235B-Thinking [78] Qwen-3.5-397B-A17B [64] GLM-4.5V [26] GLM-4.6V [61] Kimi-VL-A3B [62] Kimi-K2.5 [63]

3.7 6.3 6.9 5.7 13.1 3.7 2.9 1.1 11.1

8.5 5.1 8.5 8.5 16.9 3.4 5.1 3.4 8.5

2.9 4.4 7.4 7.4 13.2 4.4 4.4 0.0 4.4

(B) Closed-Source Models Gemini-2.5-Pro [60] Gemini-3-Flash [58] Gemini-3.1-Pro [57] GPT-5 [55] GPT-5.4 [49] Doubao-2.0-Lite [5]

Experiment

3.1

Experimental Setup

7.4 8.0 11.4 14.9 8.0 5.7

11.9 10.2 10.2 16.9 5.1 6.8

1.5 4.4 5.9 10.3 5.9 5.9

Models and Tasks. We benchmark 15 state-of-the-art MLLMs spanning open-source and proprietary families: Qwen series [3, 78, 64]; GLM series [26, 61]; Kimi series [62, 63]; Gemini series [60, 57, 58]; GPT series [55, 49]; Seed Series [5]. All models are evaluated using their official APIs or open-weight checkpoints without task-specific fine-tuning. Each model is prompted with the egocentric RGB screenshot and a natural-language task description at every step; no privileged state information is provided. All models are evaluated on the full S PATIALW ORLD benchmark comprising 760 tasks across 8 simulation environments. Evaluation Details. We use temperature τ = 1.0 and retain the latest w = 30 turns of interaction as context for all main experiments. The step budget for each task is dynamically determined as 2g + 10, where g denotes the golden action count annotated by human annotators. Unless otherwise specified, we report TSR and solution efficiency (SE) aggregated over evaluated trajectories. 3.2

Main Results

Table 3 and Table 4 benchmark the TSR and SE performance of state-of-the-art MLLMs, revealing the following key insights: A significant gap remains between MLLM agents and real-world 3D environments. Current models struggle significantly with physical tasks, where the best-performing GPT-5 reaches only 14.4% Physical Overall TSR, followed by Qwen-3.5-397B-A17B at 12.2%. Furthermore, these successes are predominantly restricted to short-horizon, fundamental operations (e.g., turning on a device). Despite moderately better performance in digital domains, the universally low success rates underscore a persistent shortfall in human-level spatial intelligence. Success Efficiency (SE) reveals reliance on trial-and-error among similarly capable models. SE is informative for differentiating models with comparable TSR; large TSR gaps render it incomparable due to divergent completed task counts and difficulty distributions. For instance, Kimi-K2.5 and GPT-5.4 exhibit comparable Physical Overall TSRs (9.2% vs. 6.6%), yet GPT-5.4 achieves a higher SE (0.569 vs. 0.486). This contrast indicates that Kimi-K2.5 relies heavily on extensive trial-and-error, executing considerably more redundant or invalid actions to reach the same objectives. Real-world spatial complexity demands comprehensive evaluation. Since no single model universally dominates—exemplified by GPT-5 and Qwen-3.5-397B-A17B tying in Work tasks 7

GPT-5.4

Q2.5-72B

GPT-5

Q3-235B-I

Kimi-K2.5

Q3-235B-T Doubao2

Gemini3.1-Pro Gemini2.5-Pro GLM-4.6V Indoor

GPT-5 Qwen3.5 Kimi-K2.5

Q3-30B

Gemini3.1-P Gemini2.5 25

Gemini3-F GPT-5 Gemini3.1-P

GPT-5.4 Qwen3-I

50 20

TSR (%)

Kimi-VL

TSR (%)

Q3.5-397B

30 20

15 10 5

Gemini3-F GLM-4.5V Outdoor

AI2

Proc

Indoor Top-5

VHome

CARLA

E.City

Outdoor Top-5

Figure 5: Indoor and outdoor physical domains. Overall TSR across indoor and outdoor physical environments, with environment-level bars for the top-five models in each domain. Table 4: Efficiency Evaluation. Main-benchmark SE across task categories. Bold and underlined denote the best and second-best per column. Physical Overall is the weighted mean over successful valid physical trajectories; – indicates no successful trajectory. Physical

Model Daily

Work

Entertain.

Digital

Travel

Social

Overall

Entertain.

0.889 0.875 0.449 0.524 0.633 0.450 0.417 – 0.553

0.143 0.174 0.243 0.218 0.290 0.270 – – 0.226

0.526 0.658 0.587 0.471 0.508 0.529 0.576 0.535 0.486

0.688 0.765 0.397 0.747 0.737 0.809 0.920 0.948 0.626

0.483 0.612 0.732 0.544 0.513 0.708

0.399 0.183 0.281 0.248 0.305 0.522

0.569 0.536 0.649 0.511 0.569 0.704

0.518 0.657 0.717 0.583 0.720 0.599

0.545 0.702 0.708 0.536 0.552 0.663 0.705 0.636 0.519

0.510 0.667 0.574 0.453 0.477 0.583 0.381 0.333 0.556

0.458 0.500 0.529 0.424 0.453 0.482 0.444 – 0.517

(B) Closed-Source Models Gemini-2.5-Pro [60] Gemini-3-Flash [58] Gemini-3.1-Pro [57] GPT-5 [55] GPT-5.4 [49] Doubao-2.0-Lite [5]

0.615 0.575 0.708 0.597 0.617 0.776

0.667 0.504 0.466 0.387 0.427 0.604

0.567 0.390 0.544 0.540 0.667 0.708

(16.9%), GPT-5 leading Travel (6.8%), and Gemini-3.1-Pro leading Digital domains (39.0%)—our multifaceted benchmark is essential to accurately capture the diverse spatial capabilities required in real-world scenarios. 3.3

Analysis

Beyond aggregate metrics, we dissect the benchmark along complementary axes—indoor–outdoor split, task complexity, multi-agent coordination, and game-family breakdown—to expose capabilityspecific bottlenecks, and probe perceptual factors (resolution, field of view) and inference-time hyperparameters (temperature, history window, action parameterization). Indoor vs. Outdoor Physical Environments. We partition the single-agent physical benchmark into indoor and outdoor domains, excluding games and multi-agent setups to isolate scene comprehension. Fig. 5 and Table 6 (Appendix A.3) reveal a pronounced domain shift: GPT-5 (14.1%) and Qwen-3.5397B-A17B (13.7%) lead indoors, while Gemini-3-Flash (9.0%) and GPT-5 (8.3%) lead outdoors This divergence exposes distinct algorithmic biases. GPT-5’s indoor superiority suggests robust fine-grained object grounding and low-level control, whereas the Gemini series’ outdoor success highlights strengths in long-horizon spatial reasoning and macro-level navigation. 8

300 200 100

Mean SE

400

0.8

Mean TSR (%)

Task count

500

30 20 10 0

Nav. Inter. Both (a) Task distribution.

Nav.

0.6 0.4 0.2 0.0

Inter. Both

(b) Performance.

Nav. Inter. Both (c) Efficiency.

0.1 0.3

0.7

Temperature

(a) Temperature

1.0

10.0 7.5 5.0 2.5 0.0

GLM-4.6V Gemini3.1-P

Window size

(b) History window

5 0 5 -VLQwe A3B n Qwe 3-30B n2.5 -7 Gem 2B ini3GLM F -4. Gem 6V ini2. 5 Gem ini3. 1-P

Qwen3-30B Gemini3-F

Kimi

10.0 7.5 5.0 2.5 0.0

GLM-4.6V Gemini3.1-P

TSR (%)

Qwen3-30B Gemini3-F

TSR (pp)

Figure 6: Complexity profile. Task counts, mean TSR, and mean SE across the three parallel complexity modes in the physical benchmark.

Model

Figure 7: Ablation trends. TSR under temperature and history-window settings, together with the signed TSR gap between continuous and discrete action parameterizations. Complexity Modes. Categorizing tasks by the action signatures from Section 2.4 reveals distinct complexity modes derived from golden action primitives: Navigation (movement and viewpoint), Interaction (object-state), and Navigation–Interaction (both). Fig. 6 illustrates a compound bottleneck: executing precise manipulations alongside long-term spatial progress (Navigation–Interaction, 4.2% mean TSR) is demonstrably harder than Interaction (50.2%). The variation in leading models across modes—with Gemini-series leading Navigation (8.6%) and GPT-5 dominating Interaction (69.4%) and the combined mode (12.1%)—validates the taxonomy. These modes successfully evaluate orthogonal capabilities rather than a singular difficulty scale. Multi-Agent Social Environments. The Social Collaboration column in Table 3 pools MultiAI2THOR and Multi-ProcTHOR, but these environments stress different coordination patterns. Fig. 8a reports them separately. GPT-5 achieves the best pooled social TSR at 34.8%, followed by Qwen-3.5-397B-A17B at 19.6% and Kimi-K2.5 at 17.4%. Most of this signal comes from MultiAI2THOR, where object-centric cooperative routines are more frequently solved; Multi-ProcTHOR remains substantially harder, with the best models reaching only 5.9%. This suggests that current agents can coordinate in familiar, hand-authored indoor layouts, but procedural multi-agent layouts sharply reduce the reliability of shared progress tracking and role assignment. Game-Level Breakdown. We further analyze performance across five digital game families (see Table 7 in Appendix A.4). The results reveal that while top models achieve strong reactive control in navigation and Snake tasks, they systematically struggle with games requiring explicit geometric reasoning and multi-step state transformations (e.g., Rubik and Block3D), indicating that spatial manipulation remains a fundamental bottleneck for multimodal agents. Perceptual Factors. Figs. 8b and 8c show that perceptual configuration affects the visual evidence available to the agent. The resolution curve remains comparatively flat and locally non-monotonic, suggesting that image resolution does not materially affect interactive spatial understanding (see Appendix C for the visualization). For field of view, higher-FOV settings outperform narrow views overall, but gains plateau and vary by setting. We nevertheless keep the default field of view at 60 to more closely approximate a human-like first-person viewing condition. Inference-time Hyperparameters. We also ablate temperature, history window size, and action parameterization on a subset in Fig. 7: temperature sensitivity is marginal, and optimal choices for 9

Multi-AI2THOR

Multi-ProcTHOR

Overall

GPT-5 Qwen3.5 Kimi-K2.5 GLM-4.5V Doubao Gemini2.5 Qwen3-I Qwen3-T Gemini3.1-P GPT-5.4 Gemini3-F Qwen3-30B Qwen2.5 Kimi-VL-A3B GLM-4.6V

Qwen3-30B

Gemini3-F

GLM-4.6V

TSR (%)

6 4 2 0

0.1 0.2

0.4

0.6

0.8

Resolution scale

1.0

(b) Resolution-scale probe on AI2THOR.

TSR (%)

Qwen3-30B

GLM-4.6V

6 4 2 0

(a) Social collaboration TSR on Multi-AI2THOR and MultiProcTHOR; the dark marker denotes the pooled social score.

Gemini3-F

Field of view

100

120

Figure 8: Social and perceptual profiles. Three complementary additional observations: multi-agent social performance, image-resolution sensitivity, and field-of-view sensitivity. window size and motion type are completely model-dependent. As no single setting proves universally optimal, we default to standard configurations. Detailed analysis is provided in Appendix A.

Related Work

4.1

Multimodal Agents

Multimodal agents are characterized by unified perception and state representations over multimodal inputs (e.g., text, images, and videos), together with multi-step planning and decision-making to complete complex tasks via tool use or direct actions [74, 84, 38, 3, 59]. Early research followed two main directions: one focused on improving multimodal foundation models for stronger representation and understanding, establishing generalizable perceptual features and vision-language alignment [62, 43, 23]. The other developed agentic interaction and long-horizon task execution frameworks, enabling iterative planning and sequential decision-making in interactive environments [25, 24, 68]. The multimodal agents have been applied across a wide range of settings, including image understanding and editing [87, 86, 20], computer use via screen-based interaction [65, 67, 70], and physical embodied environments [15, 39, 18, 10]. 4.2

3D Environment Simulators

A wide range of 3D simulation platforms have been developed to support spatial reasoning, navigation, and autonomous decision-making across diverse domains [34, 72, 51, 73, 83]. For indoor embodied interaction, AI2-THOR [32] provides interactive, near-photorealistic scenes with rich object affordances for studying task-oriented manipulation. Habitat [53] offers an efficient modular framework with configurable sensors and agent embodiments, and is widely used for navigation, instruction following, and embodied question answering. For autonomous driving, CARLA [14] and MetaDrive [37] simulate urban traffic with flexible sensor suites and dynamic actors, serving as standard testbeds for learning-based perception and control. Recent efforts further expand simulator coverage and realism [19, 4, 82, 33], broadening the space of scenarios and world dynamics available for evaluation. Despite these advances, existing platforms and their associated scenarios remain domain-specific (e.g., indoor manipulation vs. urban driving) and adopt heterogeneous task definitions and interfaces, making it difficult to compare general, open-ended task-solving ability across settings. 4.3

Spatial Reasoning of Multimodal Agents

Spatial reasoning in multimodal agents refers to the ability to ground goals in real-world perceptual observations and maintain an evolving spatial belief about the environment under observability [89, 6, 77, 9], enabling agents to localize objects, infer relative motion relationships, and support reliable planning and action in physical space [44, 88, 45]. Spatial reasoning has been primarily evaluated 10

through visual question answering benchmarks [30, 28, 42, 2, 47, 16, 71, 17] under fixed 2D observations, with recent evaluations extending to 3D and video settings [81, 41, 27, 79] to test whether models can build and recall spatial structure from sequential observations. Multi-turn interaction is essential for spatial reasoning, as agents must make sequential decisions to gather information and update spatial beliefs over time. However, most existing multi-step benchmarks are either grounded in 2D screen [75, 52, 31] or adopt heterogeneous embodied interfaces and observation assumptions [44, 36, 22]. For instance, EmbodiedBench [80] evaluates agents that process visual and sensor data to predict low-level atomic navigation or manipulation actions. In contrast, S PATIALW ORLD evaluates foundation multimodal agents under a unified closed-loop protocol across heterogeneous 3D environments. Agents receive only egocentric RGB observations and high-level task instructions, and their success is determined by task-specific terminal-state verifiers rather than static answer matching or low-level action prediction accuracy.

Conclusion

We introduced S PATIALW ORLD, a unified benchmark designed to evaluate the interactive spatial reasoning of MLLMs. By abstracting simulator-specific complexities into a shared text-based interface, our benchmark rigorously assesses an agent’s capacity for active egocentric exploration and decision-making under partial observability. Extensive evaluations of 15 leading MLLMs reveal a critical vulnerability: while current models excel at static scene perception, they struggle profoundly with dynamic physical environments—exhibiting low task success rates, severe execution inefficiencies, and high domain variance. These bottlenecks underscore a fundamental gap in robust interactive spatial reasoning and long-horizon planning. We envision S PATIALW ORLD as a foundational testbed to shift the paradigm of MLLM research from passive observation to the realization of general-purpose spatial agents capable of seamless interaction in the real world.

References [1] Anthropic. Introducing claude opus 4.5. https://www.anthropic.com/news/claude-opus-4-5, 2025. [2] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19129–19139, 2022. [3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https://arxiv.org/abs/2502.13923. [4] Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503.06669, 2025. [5] ByteDance. Seed2.0, 2026. URL https://seed.bytedance.com/en/seed2. [6] Zhongang Cai, Ruisi Wang, Chenyang Gu, Fanyi Pu, Junxiang Xu, Yubo Wang, Wanqi Yin, Zhitao Yang, Chen Wei, Qingping Sun, et al. Scaling spatial intelligence with multimodal foundation models. arXiv preprint arXiv:2511.13719, 2025. [7] Zhongang Cai, Yubo Wang, Qingping Sun, Ruisi Wang, Chenyang Gu, Wanqi Yin, Zhiqian Lin, Zhitao Yang, Chen Wei, Xuanke Shi, Kewang Deng, Xiaoyang Han, Zukai Chen, Jiaqi Li, Xiangyu Fan, Hanming Deng, Lewei Lu, Bo Li, Ziwei Liu, Quan Wang, Dahua Lin, and Lei Yang. Holistic evaluation of multimodal llms on spatial intelligence. arXiv preprint arXiv:2508.13142, 2025. [8] Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37:107703–107744, 2024. [9] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14455–14465, 2024. [10] Yaran Chen, Wenbo Cui, Yuanwen Chen, Mining Tan, Xinyao Zhang, Jinrui Liu, Haoran Li, Dongbin Zhao, and He Wang. Robogpt: an llm-based long-term decision-making embodied agent for instruction following tasks. IEEE Transactions on Cognitive and Developmental Systems, 2025. [11] Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, Lei Shi, and Maosong Sun. EmbodiedEval: Evaluate multimodal LLMs as embodied agents. arXiv preprint arXiv:2501.11858, 2025. [12] Google Deepmind. Gemini 3 pro best for complex tasks and bringing creative concepts to life. https://deepmind.google/models/gemini/pro/, 2025. [13] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied AI using procedural generation. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 27c546ab1e4f1d7d638e6a8dfbad9a07-Abstract-Conference.html. [14] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1–16. PMLR, 2017. 12

[15] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: an embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, pages 8469–8488, 2023. [16] Mengfei Du, Binhao Wu, Zejun Li, Xuan-Jing Huang, and Zhongyu Wei. Embspatial-bench: Benchmarking spatial understanding for embodied tasks with large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 346–355, 2024. [17] Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pages 11198–11201, 2024. [18] Mohamed Elnoor, Kasun Weerakoon, Gershom Seneviratne, Ruiqi Xian, Tianrui Guan, Mohamed Khalid M Jaffar, Vignesh Rajagopal, and Dinesh Manocha. Vlm-gronav: Robot navigation using physically grounded vision-language models in outdoor environments. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 2391–2398. IEEE, 2025. [19] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35:18343–18362, 2022. [20] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision, pages 75–92. Springer, 2024. [21] Chen Gao, Baining Zhao, Weichen Zhang, Jinzhu Mao, Jun Zhang, Zhiheng Zheng, Fanhang Man, Jianjie Fang, Zile Zhou, Jinqiang Cui, Xinlei Chen, and Yong Li. EmbodiedCity: A benchmark platform for embodied agent in real-world city environment. arXiv preprint arXiv:2410.09604, 2024. [22] Mohsen Gholami, Ahmad Rezaei, Zhou Weimin, Sitong Mao, Shunbo Zhou, Yong Zhang, and Mohammad Akbari. Spatial reasoning with vision-language models in ego-centric multi-view scenes. arXiv preprint arXiv:2509.06266, 2025. [23] Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report. arXiv preprint arXiv:2505.07062, 2025. [24] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024. [25] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024. [26] Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.5 v and glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006, 2025. [27] Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9202–9212, 2023. [28] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019. 13

[29] Mengdi Jia, Zekun Qi, Shaochen Zhang, Wenyao Zhang, Xinqiang Yu, Jiawei He, He Wang, and Li Yi. OmniSpatial: Towards comprehensive spatial reasoning benchmark for vision language models. In International Conference on Learning Representations, 2026. [30] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017. [31] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 881–905, 2024. [32] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017. [33] Zhiqian Lan, Yuxuan Jiang, Ruiqi Wang, Xuanbing Xie, Rongkui Zhang, Yicheng Zhu, Peihang Li, Tianshuo Yang, Tianxing Chen, Haoyu Gao, et al. Autobio: A simulation and benchmark for robotic automation in digital biology laboratory. arXiv preprint arXiv:2505.14030, 2025. [34] Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.03272, 2021. [35] Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking LLMs for embodied decision making. arXiv preprint arXiv:2410.07166, 2024. [36] Mingsheng Li, Xin Chen, Chi Zhang, Sijin Chen, Hongyuan Zhu, Fukun Yin, Gang Yu, and Tao Chen. M3dbench: Let’s instruct large models with multi-modal 3d prompts. arXiv preprint arXiv:2312.10763, 2023. [37] Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3):3461–3475, 2022. [38] Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system 1 to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.17419, 2025. [39] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26689–26699, 2024. [40] Jingli Lin, Runsen Xu, Shaohao Zhu, Sihan Yang, Peizhou Cao, Yunlong Ran, Miao Hu, Chenming Zhu, Yiman Xie, Yilin Long, Wenbo Hu, Dahua Lin, Tai Wang, and Jiangmiao Pang. MMSI-Video-Bench: A holistic benchmark for video-based spatial intelligence. arXiv preprint arXiv:2512.10863, 2025. [41] JingLi Lin, Chenming Zhu, Runsen Xu, Xiaohan Mao, Xihui Liu, Tai Wang, and Jiangmiao Pang. Ost-bench: Evaluating the capabilities of mllms in online spatio-temporal scene understanding. arXiv preprint arXiv:2507.07984, 2025. [42] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635–651, 2023. [43] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. In European conference on computer vision, pages 126–142. Springer, 2024. 14

[44] Weichen Liu, Qiyao Xue, Haoming Wang, Xiangyu Yin, Boyuan Yang, and Wei Gao. Spatial reasoning in multimodal large language models: A survey of tasks, benchmarks and methods. arXiv preprint arXiv:2511.15722, 2025. [45] Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang, Yaochen Hu, Lingfeng Zhang, Yingxue Zhang, Shuang Wu, Tongtong Cao, Guowei Huang, et al. Spatialcot: Advancing spatial reasoning through coordinate alignment and chain-of-thought for embodied task planning. arXiv preprint arXiv:2501.10074, 2025. [46] Wufei Ma, Haoyu Chen, Guofeng Zhang, Yu-Cheng Chou, Jieneng Chen, Celso de Melo, and Alan Yuille. 3DSRBench: A comprehensive 3D spatial reasoning benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6924–6934, 2025. [47] Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. [48] OpenAI. Introducing gpt-5.2. https://openai.com/index/introducing-gpt-5-2/, 2025. [49] OpenAI. Gpt-5.4 thinking system card, 2026. gpt-5-4-thinking-system-card/.

URL https://openai.com/index/

[50] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 8494–8502. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR.2018.00886. URL http://openaccess.thecvf.com/content_cvpr_ 2018/html/Puig_VirtualHome_Simulating_Household_CVPR_2018_paper.html. [51] Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2109.08238, 2021. [52] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024. [53] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9339–9347, 2019. [54] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10740–10749, 2020. [55] Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267, 2025. [56] Tin Stribor Sohn, Maximilian Dillitzer, Jason J. Corso, and Eric Sax. Embodied4c: Measuring what matters for embodied vision-language navigation, 2025. URL https://arxiv.org/ abs/2512.18028. [57] Gemini Team. Gemini 3 pro: the frontier of vision ai, 2025b. URL https://blog.google/ technology/developers/gemini-3-pro-vision. [58] Gemini Team. Gemini 3 flash, 2025b. URL https://deepmind.google/models/gemini/ flash/. 15

[59] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [60] Gemini 2.5 Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https://arxiv.org/abs/ 2507.06261. [61] GLM-V Team. Glm-4.6v: Open source multimodal models with native tool use, 2025a. URL https://z.ai/blog/glm-4.6v. [62] Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025. [63] Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2. 5: Visual agentic intelligence. arXiv preprint arXiv:2602.02276, 2026. [64] Qwen Team. Qwen3.5: Accelerating productivity with native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5. [65] Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint arXiv:2509.02544, 2025. [66] Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thousand words? delving into spatial reasoning for vision language models. In Advances in Neural Information Processing Systems, volume 37, 2024. [67] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024. [68] Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245, 2023. [69] Wenqi Wang, Reuben Tan, Pengyue Zhu, Jianwei Yang, Zhengyuan Yang, Lijuan Wang, Andrey Kolobov, Jianfeng Gao, and Boqing Gong. SITE: Towards spatial intelligence thorough evaluation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9058–9069, 2025. [70] Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, et al. Opencua: Open foundations for computer-use agents. arXiv preprint arXiv:2508.09123, 2025. [71] Haoning Wu, Xiao Huang, Yaohui Chen, Ya Zhang, Yanfeng Wang, and Weidi Xie. SpatialScore: Towards unified evaluation for multimodal spatial understanding. arXiv preprint arXiv:2505.17012, 2025. [72] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9068–9079, 2018. [73] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11097–11107, 2020. [74] Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. Large multimodal agents: A survey. arXiv preprint arXiv:2402.15116, 2024. 16

[75] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024. [76] Peiran Xu, Sudong Wang, Yao Zhu, Jianing Li, Gege Qi, and Yunjian Zhang. SpatialBench: Benchmarking multimodal large language models for spatial cognition. arXiv preprint arXiv:2511.21471, 2025. [77] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In European Conference on Computer Vision, pages 131–147. Springer, 2024. [78] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. [79] Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 10632–10643, 2025. [80] Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09560, 2025. [81] Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, Dahua Lin, Tai Wang, and Jiangmiao Pang. MMSI-Bench: A benchmark for multi-image spatial intelligence. arXiv preprint arXiv:2505.23764, 2025. [82] Xuemeng Yang, Licheng Wen, Tiantian Wei, Yukai Ma, Jianbiao Mei, Xin Li, Wenjie Lei, Daocheng Fu, Pinlong Cai, Min Dou, et al. Drivearena: A closed-loop generative simulation platform for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 26933–26943, 2025. [83] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020. [84] Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mmllms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601, 2024. [85] Shiduo Zhang, Zhe Xu, Peiju Liu, Xiaopeng Yu, Yuan Li, Qinghui Gao, Zhaoye Fei, Zhangyue Yin, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. VLABench: A large-scale benchmark for language-conditioned robotics manipulation with long-horizon reasoning tasks. arXiv preprint arXiv:2412.18194, 2024. [86] Yi-Fan Zhang, Xingyu Lu, Shukang Yin, Chaoyou Fu, Wei Chen, Xiao Hu, Bin Wen, Kaiyu Jiang, Changyi Liu, Tianke Zhang, et al. Thyme: Think beyond images. arXiv preprint arXiv:2508.11630, 2025. [87] Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505.14362, 2025. 17

[88] Enshen Zhou, Cheng Chi, Yibo Li, Jingkun An, Jiayuan Zhang, Shanyu Rong, Yi Han, Yuheng Ji, Mengzhen Liu, Pengwei Wang, et al. Robotracer: Mastering spatial trace with reasoning in vision-language models for robotics. arXiv preprint arXiv:2512.13660, 2025. [89] Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness. arXiv preprint arXiv:2409.18125, 2024.

Additional Benchmark Details

This section provides supplementary details for the benchmark construction and evaluation described in Sections 3.1 and the main benchmark protocol. We include a full benchmark comparison table, task-specific evaluation criteria, and fine-grained performance breakdowns across indoor/outdoor environments and digital game families. A.1

Detailed Benchmark Comparison

Table 5 presents an extended comparison of S PATIALW ORLD against existing spatial reasoning benchmarks spanning three major categories: ImageQA, VideoQA, and embodied-agent evaluation. We compare along five critical dimensions: (1) whether the benchmark provides a unified crossplatform interface that abstracts away environment-specific APIs, (2) whether agents interact with a dynamic, interactive environment rather than answering questions over static inputs, (3) whether observations are captured from a first-person (egocentric) perspective, (4) whether the input modality is purely visual without auxiliary structured data such as depth maps or object coordinates, and (5) whether the output is expressed in natural language form. As shown in the table, existing ImageQA and VideoQA benchmarks [66, 46, 71, 76, 79] predominantly evaluate passive spatial understanding through static question answering, lacking interactive environments and unified interfaces. Embodied benchmarks [54, 85, 11, 80] introduce interactivity but typically sacrifice one or more desirable properties—either requiring privileged non-visual inputs, lacking language-form outputs, or being restricted to a single simulation platform. In contrast, S PATIALW ORLD is the only benchmark that simultaneously satisfies all five criteria, enabling a holistic evaluation of active spatial reasoning under realistic embodied constraints across diverse environments. Table 5: Detailed spatial benchmark comparison. Extended version of Table 1, including the full set of representative ImageQA, VideoQA, and embodied-agent benchmarks used to motivate the benchmark construction. Instances

Unified crossplatform interface

Interactive env.

First-person observation

Vision-only input

Language-form output

4635 2772 3640 5025 8400 24k 1000

✘ ✘ ✘ ✘ ✘ ✘ ✘

✘ ✘

ImageQA

SpatialEval [66] 3DSRBench [46] EmbSpatial-Bench [16] SpatialScore [71] OmniSpatial [29] EASI [7] MMSI-Bench [81]

✓ ✓ ✓ ✓ ✓ ✓ ✓

VideoQA

SpatialBench [76] SITE [69] MMSI-Video-Bench [40] VSI-Bench [79] OST-Bench [41]

3193 8068 1106 5130 10k

✘ ✘ ✘ ✘ ✘

✓ ✓ ✓ ✓ ✓

ALFRED [54] VLABench [85] EAI [35] EmbodiedEval [11] EmbodiedBench [80] EmbodiedCity [21]

25.7k 100 438 328 1128 87.1k

✘ ✘

✓

✘ ✘

✓ ✓ ✓

✓

✘

✓ ✓ ✓ ✓ ✓ ✓

✓

✓ ✓

S PATIALW ORLD

760

✓

Type

Benchmark

Embodied Bench

Ours

A.2

✓ ✘

✓

✓ ✘ ✘ ✘ ✘

✓ ✘ ✘

✓ ✓ ✓ ✓ ✘

✘ ✘

✓

Task-Specific Evaluation Details

While S PATIALW ORLD primarily evaluates tasks using the binary TSR based on exact goal satisfaction, certain environments require task-specific adaptations. For instance, in the Snake3D environment, exact completion is too sparse to separate weak partial progress from complete failure. Therefore, instead of using a binary success indicator, we evaluate performance by reporting a scale-normalized discrete score. This is calculated by dividing the achieved snake score by the spatial edge length of the game environment, providing a more granular measure of the agent’s progress. 19

A.3

Indoor vs. Outdoor Performance Breakdown

Table 6 reports the per-environment TSR for all 15 evaluated models, partitioned into indoor (AI2THOR, ProcTHOR, VirtualHome) and outdoor (CARLA, EmbodiedCity) domains. Multiagent environments are excluded here and analyzed separately in Fig. 8a. The overall columns pool the environments within each domain. This fine-grained breakdown reveals that GPT-5 and Qwen-3.5-397B-A17B dominate in indoor scenarios requiring precise object grounding, whereas GPT-5 and Gemini-3-Flash lead in outdoor scenarios that demand long-range navigation and spatial planning. Table 6: Indoor-outdoor. The TSR (%) of the single-agent physical benchmark across indoor and outdoor environments. Bold and underlined entries denote the best and second-best values in each column, respectively. Multi-agent environments are excluded here and analyzed separately in Figure 8a. The overall columns pool the environments within each domain and are located at the right edge of each domain group. Indoor

Model AI2THOR

ProcTHOR

Outdoor VHome

Overall

CARLA

E.City

Overall

4.2 5.0 6.9 6.3 13.7 3.8 3.6 1.3 9.7

1.2 1.2 1.2 1.2 2.5 1.2 0.0 0.0 3.8

0.0 9.4 11.3 7.5 7.5 1.9 1.9 0.0 5.7

0.8 4.5 5.3 3.8 4.5 1.5 0.8 0.0 4.5

7.4 6.9 9.7 14.1 6.7 6.3

1.2 3.8 2.5 6.2 0.0 1.2

5.7 17.0 15.1 11.3 15.1 1.9

3.0 9.0 7.5 8.3 6.0 1.5

5.1 6.4 7.7 9.0 16.7 4.5 4.8 1.6 10.9

Gemini-2.5-Pro [60] Gemini-3-Flash [58] Gemini-3.1-Pro [57] GPT-5 [55] GPT-5.4 [49] Doubao-2.0-Lite [5]

8.7 7.7 6.4 16.1 8.4 6.1

0.0 0.0 0.8 0.0 0.0 0.0 0.0 0.0 1.6

10.5 10.5 21.1 5.3 34.2 10.5 5.3 2.6 26.3

(B) Closed-Source Models

A.4

0.0 0.8 5.5 0.0 0.0 0.0

21.1 21.1 50.0 44.7 15.8 28.9

Game-Level Performance Breakdown

Table 7 presents the per-game-family TSR for all evaluated models across five digital game environments: Block3D (B3D), Maze, Maze3D (M3D), Rubik’s Cube, and Snake. Each column pools the available levels for the corresponding game. The Snake environment normalizes scores by the spatial edge length and caps each level contribution at 100%. Gemini-3.1-Pro demonstrates the highest overall efficacy (39.0%), driven by strong results on Block3D (40.0%) and Snake (90.0%), while Gemini-3-Flash leads Rubik’s Cube (50.0%). In contrast, topological traversal tasks expose different architectural strengths: Qwen3-VL-235B-Thinking excels in both 2D pathfinding (Maze, 70.0%) and 3D perspective navigation (Maze3D, 32.0%), whereas GPT-5 is strongest on Snake (91.2%). This performance divergence reveals that while top-tier architectures demonstrate robust proficiency in reactive visual-motor alignment and topological routing, they systematically falter on tasks demanding explicit geometric reasoning and complex structural state transformations. The generally low success rates on Rubik and Block3D emphasize that multi-step spatial manipulation remains a fundamental bottleneck for embodied intelligence.

Ablation Studies

This section provides detailed ablation results complementing the analysis in Section 3.1. As summarized in Fig. 7 of the main text, we study three inference-time factors—temperature, history window size, and action parameterization—and find that their optimal settings are model-dependent rather than universal. Below we discuss each factor in detail. 20

Table 7: Performance of Game. TSR (%) by game families. Bold and underlined entries denote the best/second-best values in each column. B3D denotes Block3D, and M3D denotes Maze3D. Model

B3D

Maze

M3D

Rubik

Snake

Overall

0.0 5.0 5.0 10.0 5.0 0.0 5.0 5.0 10.0

5.0 6.2 1.2 23.8 36.2 36.2 2.5 2.5 72.5

7.6 7.9 5.0 28.3 26.0 14.5 8.1 3.3 31.0

5.0 50.0 45.0 10.0 0.0 10.0

81.2 85.0 90.0 91.2 12.5 65.0

32.6 38.1 39.0 36.4 11.9 24.8

0.0 5.0 10.0 5.0 5.0 0.0 0.0 0.0 5.0

30.0 25.0 5.0 70.0 65.0 25.0 30.0 0.0 40.0

4.0 0.0 4.0 32.0 20.0 12.0 4.0 8.0 28.0

(B) Closed-Source Models Gemini-2.5-Pro [60] Gemini-3-Flash [58] Gemini-3.1-Pro [57] GPT-5 [55] GPT-5.4 [49] Doubao-2.0-Lite [5]

B.1

5.0 35.0 40.0 0.0 5.0 0.0

60.0 10.0 5.0 65.0 15.0 35.0

16.0 16.0 20.0 20.0 24.0 16.0

Temperature

Fig. 7 illustrates the impact of sampling temperature on performance. We observe that nearly all models, with the sole exception of Gemini-3-Flash, achieve their optimal performance at τ = 1.0. Consequently, following the evaluation protocol of OSWorld [75], we set τ = 1.0 for all models. This configuration ensures protocol uniformity across model families while preserving moderate sampling diversity throughout long-horizon interactions. B.2

History Window Size

Fig. 7 illustrates that w = 30 serves as an optimal sliding window size across most evaluated models. We observe that while performance improves with initial increases in window size, it tends to plateau or slightly diminish beyond w = 30. This suggests that a context window of 30 frames provides sufficient temporal information, and further extending the visual history does not yield universal gains. Consequently, we adopt w = 30 as the default context window for the main benchmark. In our ablation analysis, the corresponding main-run result is treated as the w = 30 baseline to ensure consistency. B.3

Continuous versus Discrete Motion

Fig. 7 reveals no universal preference for continuous over discrete motion. In the actionparameterization panel, each bar reports ∆TSR = TSRcontinuous − TSRdiscrete in percentage points; positive values favor continuous action parameters, whereas negative values favor the discrete interface. Because the optimal action parameterization is model-dependent, we utilize discrete actions in the main benchmark. This decision maintains interface consistency across environments and avoids biasing the leaderboard toward a control granularity that favors a specific model family.

Observation Sensitivity Analysis

Fig. 9 shows visualizations at various resolutions. Decreasing resolution does not impair the model’s spatial reasoning because spatial perception relies on physical, projective, and ray relationships, which are unaffected by lower resolutions. The visual results confirm that the model maintains accurate spatial awareness.

0.1

0.2

0.4

0.6

0.8

1.0

Figure 9: Observation Sensitivity Analysis under the Same Viewpoint with Varying Resolutions. We progressively increase the resolution ratio along the x-axis, reaching the highest clarity at 1.0.

Environment Suite

S PATIALW ORLD uses its environment suite as the main source of domain diversity rather than as a passive collection of scenes. We wrap eight 3D backends with a shared agent-side API, so agents interact through the same observation and action abstractions while the environments retain their native differences in scale, dynamics, object affordances, and scene generation. This design exposes a broad spectrum of spatial demands under one evaluation surface: hand-authored indoor worlds test fine-grained object grounding, procedural houses test layout generalization, urban and aerial environments test long-range progress estimation, multi-agent variants test coordination, and digital games isolate abstract geometric reasoning. As a result, cross-environment comparisons reflect genuine differences in 3D task-solving requirements rather than changes in the agent interface. The suite is organized into three families. Indoor Simulation. AI2-THOR [32] provides hand-designed, near-photorealistic indoor scenes with explicit object affordances, physical interactions, and state changes; it therefore anchors the benchmark in fine-grained object grounding and manipulation. ProcTHOR [13] extends this setting through procedurally generated houses, increasing layout diversity and reducing reliance on a fixed set of manually authored rooms. VirtualHome [50] complements both environments by representing household activities as executable programs, making it suitable for scripted daily routines that require temporally ordered actions. Building on the same indoor affordance space, the multi-agent variants of AI2-THOR and ProcTHOR introduce cooperative tasks in which success depends not only on locating or manipulating objects, but also on coordinating role-specific progress across agents. Outdoor Navigation. CARLA [14] shifts the benchmark from indoor object interaction to urban driving, where agents must reason over road topology, long-range route progress, and dynamic traffic context. EmbodiedCity further broadens the outdoor setting to aerial city navigation, emphasizing landmark-based localization, altitude-aware movement, and macroscopic spatial planning over dense urban layouts. Together, the outdoor environments test whether a model can maintain progress estimates and termination decisions when the relevant spatial evidence is distributed across a much larger field than a household room. Digital Game Environments. Indoor and outdoor simulators are indispensable for realistic embodied tasks, but they do not exhaust the space of 3D reasoning problems: their difficulty is often entangled with photorealistic semantics, simulator affordances, and natural scene priors. We therefore add lightweight 3D games as controlled probes that isolate abstract spatial abilities under closed-loop interaction. Block3D requires three-view geometric counting from orthographic projections; Maze and Maze3D test topological planning in two- and three-dimensional layouts; Snake3D stresses incremental state tracking under self-occlusion and limited free space; and Rubik’s Cube evaluates spatial transformation and configuration reasoning. These games broaden the benchmark beyond 22

household and urban navigation by exposing spatial structures that are rare in realistic simulators but central to general 3D reasoning.

Human Annotation

To ensure the quality and reproducibility of S PATIALW ORLD, all 760 tasks undergo a rigorous three-stage human annotation process (see Fig. 2 for an overview of the pipeline). In the first stage, annotators design each task by specifying the natural-language instruction and configuring the initial environment state. In the second stage, annotators independently solve each task inside the simulator, recording the ground-truth terminal state and a reference action trajectory. In the third stage, annotators cross-check each other’s work to verify task feasibility, instruction clarity, and evaluation-script correctness. Table 8 presents representative examples of the resulting annotated evaluation scripts, which retrieve dynamic state data (e.g., spatial coordinates, object containment, vehicle kinematics) from the 3D simulators to reliably assess functional correctness in open-ended physical environments. Table 8: Examples of our annotated execution-based evaluation scripts in SpatialWorld. The scripts retrieve dynamic state data (e.g., spatial coordinates, object containment, vehicle kinematics) from the 3D simulators to reliably assess functional correctness in open-ended physical environments. Overview State

Initial State

Task Instruction

Success Condition

I found the lettuce was rotten; please help me throw it in the trash.

object_in_receptacle object_type: Lettuce receptacle_type: GarbageCan

Walk to the position marked by the red line in the screenshot. You can turn and move in any direction.

distance_to_waypoint target_location: [41.9, 32.9, 1.2] threshold_label: medium

I need to tidy up the kitchen. Please open the refrigerator door and put the salmon inside, but do not close the refrigerator door.

1. object_state obj: fridge, state: isOpen Value:True 2. object_in_receptacle obj: salmon, rec: fridge Value:True

Action Space Definition

This section details the S PATIALW ORLD unified action interface. As introduced in Section 2.3, this interface abstracts raw backend commands into high-level text primitives to form a unified MLLM-native action space. Table 9 categorizes these primitives into four canonical groups. In this specification, 0 denotes an explicit no-motion/wait decision, used when the agent should hold its position without changing pose (e.g., waiting at a red light or pausing for coordination). This specification maps diverse environment behaviors to a standard set of expected primitives. For instance, Move seamlessly handles everything from a 0.25 m indoor step to a 10 m driving advance, or simply waiting in place (0). Similarly, object interactions are distinctly separated at the interface level: ChangeState targets verifiable persistent state transitions (e.g., opening, cooking), while Manipulate handles local force-based or relational interventions (e.g., pushing, grabbing).

GPT-5 vs. GPT-5.4 Case Study

To explain why GPT-5 outperforms GPT-5.4 in the current benchmark, we compare the two models on the 609 shared single-agent physical tasks spanning AI2THOR, ProcTHOR, VirtualHome, CARLA, 23

GPT-5

Premature DONE Explicit FAIL Early stop

GPT-5.4 100

Failure-mode share (%)

TSR on shared tasks (%)

20 10 0

Max-step Parse error Other

AI2THO

OR ProcTH

ome

VirtualH

iedCity

CARLA

80 60 40 20 0

Embod

(a) Shared-task TSR by environment.

GPT-5

GPT-5.4

(b) Failure-type composition.

25 120 100

Failure steps

Success steps

20 15 10

60 40

0 GPT-5

GPT-5.4

GPT-5

GPT-5.4

(d) Step counts on failed tasks.

Figure 10: Why GPT-5 currently outperforms GPT-5.4. GPT-5 achieves higher shared-task TSR in most physical environments, while GPT-5.4 exhibits a stronger tendency toward premature termination. The step-count plots further show that GPT-5 typically spends more actions both when it succeeds and when it fails, consistent with a slower but more persistent search strategy. and EmbodiedCity. GPT-5 succeeds on 78 of these tasks (12.8%), whereas GPT-5.4 succeeds on 40 (6.6%). The largest shared-task gaps appear in VirtualHome (+28.9 points), AI2THOR (+7.7), and CARLA (+6.2), while EmbodiedCity slightly favors GPT-5.4 (-3.8) and ProcTHOR remains unsolved by both. The disagreement is also asymmetric: GPT-5 solves 52 tasks that GPT-5.4 misses, whereas GPT-5.4 recovers only 14 in the reverse direction. Most of GPT-5’s advantage comes from Daily Household, Work & Study, and Travel tasks, rather than from a uniform lead across every category. The error profile suggests that the main difference is termination policy rather than raw action speed. Fig. 10 shows that GPT-5.4 fails primarily by stopping too early: across the environments with comparable failure logs, 32.4% of its failures are premature DONE decisions and 48.5% are explicit FAIL decisions. GPT-5 instead fails more often by persistence without completion, with 63.6% of failures ending at the step limit and another 15.1% ending after repeated action failures. This behavioral gap is mirrored in action counts. On successful tasks, GPT-5 uses a median of 7 steps, compared with 3.5 for GPT-5.4; on failed tasks, the medians are 22 and 11, respectively. GPT-5 is therefore slower and less efficient, but it is also more willing to keep exploring until the verifier conditions are actually met, whereas GPT-5.4 often commits to an early terminal decision before the state is sufficiently verified.

Qualitative Analysis

To complement the quantitative evaluation in Section 3.1, we provide a qualitative analysis of agent failure modes and their relationship to spatial reasoning capabilities. Failure Mode Breakdown. We manually inspect 100 failed trajectories and categorize failures into: (i) Spatial Disorientation—the agent loses track of its position and cannot return to a target location; (ii) Object Hallucination—the agent issues Interact actions on objects not present in the current view; (iii) Premature Termination—the agent issues EndTask(status=DONE or FAIL) before the 24

Table 9: Detailed unified action-space specification. The benchmark action space is defined by four canonical categories of high-level text primitives, mapping diverse repository-specific commands to a single standardized interface. Category

Expected unified action(s)

Detailed parameterization

Navigation Move MLLM agents

Move(direction, [granularity])

Egocentric translation. Move(..., 0) is a no-op wait. Step sizes vary by environment: AI2-THOR / Procdirection ∈ {forward, backward, THOR / VirtualHome use Small = left, right, up, down}. 0.25 m, Medium = 0.5 m, Large = granularity ∈ {0, Small, Medium, 1 m. EmbodiedCity uses 0.5 / 2.0 / Large} or numeric distance. 0 means 5.0 m. CARLA uses coarse routestay in place / wait. progress steps (4 / 10 m for vehicles, 3 / 10 m for pedestrians). Continuous ProcTHOR supports exact numeric meters.

Viewpoint & Posture Adjust view/stance

Changes viewpoint or body stance without altering object states. Granon the backend: deRotate(direction, [angle]) For Rotate, direction ∈ {left, ularity depends right}. For Tilt, direction ∈ {up, faults are 90◦ (rotate) and 30◦ (tilt) Tilt(direction, [angle]) down}. in AI2-THOR/ProcTHOR; 30◦ /90◦ ChangePosture(pose) in VirtualHome; and 5◦ /15◦ /45◦ in pose ∈ {crouch, stand, stand_up}. EmbodiedCity. Angles are freely tunable in continuous settings.

Interaction Change object states

obj and optional target are exact class tokens. state ∈ {open, close, on, off, clean, dirty, sliced, broken, cooked, filled, empty, used_up}. action ∈ {push, pull, throw, touch, look_at, drink, etc.}.

Task-Control Status & communicate

Pick/Place(obj, [target]) ChangeState(obj, state) Manipulate(obj, action)

EndTask(status) Communicate(msg)

Meaning and backend realization

Subsumes grasping, placement, and manipulation under an objectcentric interface. AI2-THOR and ProcTHOR support the broadest set of persistent state changes (e.g., cook, slice, fill). VirtualHome realizes this via a smaller subset (e.g., Grab, SwitchOn, Drink). Backend specific names are purely wrappers.

EndTask triggers evaluator verification of the terminal goal (CARLA only exposes successful complestatus ∈ {DONE, FAIL}. msg is a short free-form natural- tion). Communicate is active exclulanguage report or request. sively in collaborative multi-agent tasks, typically via structured output tags.

goal is satisfied; and (iv) Action Loop—the agent cycles through the same sequence of ineffective actions until the step budget is exhausted. We select four representative bad cases for detailed qualitative analysis in the following section, covering spatial disorientation, object hallucination, premature termination, and action-loop behaviors. Bad cases analysis. In Fig. 11 - 14, we present several bad cases for various state-of-the-art models across four distinct environments. These examples cover the full spectrum of failure modes. We present the performance of two mainstream closed-source models, GPT-5 and Gemini-3.1-Pro, in terms of two failure modes, Spatial Disorientation and Premature Termination, under different environments in Fig. 11 and 13. In Fig. 11, GPT-5 exhibits spatial disorientation at Step 6, failing to accurately perceive the surrounding obstacles. This deficiency makes it difficult for the agent to reach the LightSwitch while moving forward, thereby hindering interaction and ultimately triggering a premature termination before the task is completed. Similarly, in Fig. 13, Gemini-3.1-Pro suffers from spatial disorientation during a simple localization and navigation task. Unable to determine the correct path to the street lamp, the model performs multiple ineffective turns and prematurely invokes the Done action before actually reaching the target storefront. These two cases demonstrate that simple spatial localization and object interaction tasks, while trivial for humans, still pose significant challenges for current MLLMs. Fig. 12 illustrates Object Hallucination and Action Loop as two additional types of failure modes. At Step 7, Gemini-3.1-Pro mistakenly assumes it has already grasped the phone, proceeding directly to execute the second task of picking up the mouse. This fundamentally stems from the model’s lack of complex spatial understanding capabilities in real-world scenarios. Consequently, despite colliding with the wall after Step 9, the model continues attempting to move forward, resulting in an action loop. Even the formidable open-source model, Qwen-3.5-397B-A17B, exhibits corresponding issues when executing a simple daily routine task, as shown in Fig. 14. 25

Table 10: Representative GPT-5 vs. GPT-5.4 disagreement cases. These examples illustrate the recurring pattern that GPT-5.4 often terminates after a short or partial trajectory, while GPT-5 spends more actions and eventually satisfies the verifier. Env.

Task ID

Task summary

GPT-5

GPT-5.4

AI2THOR

ai2thor05010

Turn on a laptop and verify that it is actually powered on.

18 steps, success

2 steps, fail

VirtualHome

virtualhome00013

Turn off the ceiling light whose switch is by the door.

17 steps, success

2 steps, fail

CARLA

carla00418

Walk to the house marked by the red frame.

14 steps, success

5 steps, fail

ProcTHOR

procthor107

Return a bowl to the kitchen and bring a pen back to the living room.

97 steps, success

37 steps, fail

Observation GPT-5.4 stops immediately after a single switch action, whereas GPT-5 keeps probing until the verifier confirms the powered-on state. GPT-5.4 terminates after a direct switch attempt, while GPT-5 spends extra navigation and orientation steps to reach a verifierconsistent state. GPT-5.4 under-travels to an approximate region and stops early; GPT-5 continues the route and reaches the target waypoint. GPT-5.4 makes partial progress but exits before finishing the second subgoal; GPT-5 is slower but eventually completes both required state changes.

Human Validation. Human annotators solve tasks during benchmark construction to verify feasibility and provide reference action counts. We use these trajectories as validation artifacts for task correctness rather than as a separate leaderboard.

Limitations and Broader Impact

Limitations. Like most embodied AI benchmarks, S PATIALW ORLD operates in simulated environments rather than on physical robotic platforms. While the selected simulators provide nearphotorealistic rendering and physically plausible dynamics, extending evaluation to real-world settings remains a promising direction for future work. Furthermore, to ensure annotation quality, the current 760 tasks are carefully handcrafted, making the scale more modest compared to automatically generated datasets. These tasks cover six scenario categories and eight backends, and the rich diversity of real-world spatial reasoning scenarios offers ample room for future expansion. Broader Impact. S PATIALW ORLD serves primarily as a diagnostic and scientific tool for understanding the spatial reasoning capabilities of multimodal agents. By systematically characterizing agent failure modes, this work contributes to the development of more reliable and trustworthy spatial agent systems, while the emphasis on open and reproducible evaluation fosters transparency in the research community. On the other hand, improvements in spatial reasoning could potentially be misused to enhance autonomous surveillance or enable unintended physical-world manipulation by embodied agents; we encourage the community to develop appropriate safety guidelines as these capabilities advance.

Compute Resources

For proprietary models (GPT-5, Gemini-3.1-Pro-Preview, Claude-Sonnet-4.6, etc.), we access them exclusively through their official APIs. For open-source models (Qwen2.5-VL-72B-Instruct, InternVL378B, etc.), we deploy them on a GUP-server equipped with 8× NVIDIA H200 GPUs. The full evaluation campaign across all models equipped on GPU server consumed approximately 5,000 GPU hours in total.

LLM Usage

We used an OpenAI LLM (GPT-5) as a writing and formatting assistant. In particular, it helped refine grammar and phrasing, improve clarity, and suggest edits to figure/table captions and layout (e.g., column alignment, caption length, placement). The LLM did not contribute to research ideation, experimental design, implementation, data analysis, or technical content beyond surface-level edits. All outputs were reviewed and edited by the authors, who take full responsibility for the final text and 26

Instruction: Please clean the pan and bowl for me, then turn off the light; I'm getting ready to leave.

Failure Mode: Spatial Disorientation Premature Termination

GPT-5 Step 0

Step 1

Thinking: I am in the kitchen facing a counter with several objects. A dirty pan is clearly visible in front of me, and the task requires cleaning the pan and bowl before turning off the light. I should clean the pan first because it is close and easy to reach. Action: CleanObject(Pan)

Step 5

area, which is a likely place for a light switch. There appears to be a vertical object near the door that could be the switch. I should try toggling the light switch off. Action: ToggleObjectOff(LightSwitch)

Step 6

Thinking: The pan has been cleaned. I can see a bowl-like silver object on the counter near the other items, so the next required object is likely the bowl. I should clean the bowl before searching for the light switch. Action: CleanObject(Bowl)

Step 2

Thinking: I am now facing a door and wall

Thinking: The attempt to turn off the switch failed because the switch was not close enough or not properly in view. Since the light switch appears to be near the door, I should move closer to get within interaction range. Action: MoveAhead(Small)

Step 7 Thinking: The bowl may still be farther

Thinking: I am still facing the door and

back on the counter, and the previous cleaning action might not have reached the correct object or range. To make sure I can interact with it, I should move closer to the counter and bowl.

wall where the light switch should be. I may still be slightly outside the required range, so I should continue moving forward a little before trying to toggle it again.

Action: MoveAhead(Medium)

Action: MoveAhead(Small)

Step 8

Step 3

Thinking: I have moved closer to the

Thinking: My view is still focused on the kitchen counter, and the bowl is visible among the objects near the vegetables. Since the task still requires the bowl to be clean, I should attempt to clean the bowl again from this position. Action: CleanObject(Bowl)

door and the light switch area. The switch should now be easier to interact with, and turning off the light is the last remaining goal. I should try toggling the light switch off again. Action: ToggleObjectOff(LightSwitch)

Step 9

Step 4 Thinking: The pan has been cleaned, and the bowl has now been handled. The remaining task is to turn off the light. I do not see a switch from the counter view, so I need to rotate and inspect the walls or doorway area. Action: RotateRight

Thinking: I have already cleaned the

pan, attempted and confirmed the bowl cleaning, and made multiple attempts to turn off the light switch. I believe all parts of the task have been addressed, so I should finish with DONE. Action: Done

Figure 11: Failure case of GPT-5 in the AI2-THOR environment. The failure modes include Spatial Disorientation and Premature Termination. visuals. LLMs are not incorporated as any core, original, or non-standard component of our proposed methodology. We only employ 15 multimodal LLMs as external test agents to evaluate the proposed benchmark, which does not constitute a part of our core method design.

Instruction: Please place a mobile phone on the table, and then pick up a mouse. Failure Mode: Object Hallucination Action Loop

Gemini-3.1-Pro Step 5

Step 0

Thinking: The cellphone grab failed because the phone was not in the current view or not reachable. I assume I am still slightly too far away, so I should move forward a small step to get closer to the cellphone. Action: WalkForward(Small)

Thinking: I am in a room facing a desk with a mouse on it, and there is a chair in front of the desk. The task is to place a mobile phone on the table and then pick up the mouse. Since I am too far from the desk to interact, I should first move closer. Action: WalkForward(Medium)

Step 1

Step 6 Thinking: I moved closer, but the camera is looking too high and the ceiling blocks much of the view. The desk and mouse are only partly visible. I need to look down to clearly see the table surface and nearby objects. Action: LookDown

Step 2

Step 7 Thinking: I incorrectly assume that I have grabbed the cellphone and now need to return to the desk with the mouse. Since I am facing away from the desk area, I should turn right to face back toward the desk. Action: TurnRight(normal)

Thinking: I can see the desk and mouse, but the mobile phone is not on the desk or in my current view. Since I need the phone before placing it on the table, I should turn left and check the adjacent room. Action: TurnLeft(normal)

Step 3

Step 4

Thinking: I try to grab the cellphone again after moving closer, but this step also has no valid thinking block. The intended plan is to obtain the phone before returning to the desk. Action: Grab(cellphone)

Step 8 Thinking: I now see the mobile phone on a coffee table in front of me. It is visible but still too far away to grab. I need to walk forward toward the coffee table so I can reach the phone. Action: WalkForward(Medium)

Thinking: I try to grab the cellphone from the coffee table, but the action has no valid thinking block. The intended goal is still to pick up the phone first so it can later be placed on the desk. Action: Grab(cellphone)

Step 9

Thinking: I need to approach the desk area so I can place the cellphone on it. After turning back toward the desk, I should walk forward a small step to get closer to the target table. Action: WalkForward(Small)

Thinking: I need to move back toward the desk so I can place the cellphone on the target table. After reorienting myself toward the desk area, I should take a small step forward to get within interaction range. Action: WalkForward(Small)

...... Figure 12: Failure case of Gemini-3.1-Pro in the VirtualHome environment. The failure modes include Object Hallucination and Action Loop.

Instruction: Can you see the storefront marked by the red box? Please walk forward to the front of this shop.

Failure Mode: Spatial Disorientation Premature Termination

Gemini-3.1-Pro Step 0

Step 1

Step 2

Step 3

Step 4

Thinking: I am facing a white building, road, and guardrail, but the target storefront is not visible. Since the storefront should be near a gas station, I need to turn left and search for the gas station area. Action: TurnLeft(degrees=large)

Thinking: I can now see the gas station and car wash area. The storefront seems to be ahead, so I should walk forward toward it. Because it is still far away, I will take a large forward step. Action: WalkForward(distance=large)

Thinking: Walking forward did not appear to change my position, so I may be blocked. The gas station is still in the target direction, but I need another path. I should turn right to look for a clearer route. Action: TurnRight(degrees=large)

Thinking: The view still did not change after turning right, which suggests I may be stuck or the action failed. I should try turning left instead, hoping it changes my orientation and helps me escape the stuck state. Action: TurnLeft(degrees=large)

Thinking: I still seem stuck because both movement and turning have failed to change the scene. Since repeating them may not help, I should try walking backward to leave the blocked or unstable position. Action: WalkBackward(distance=large)

Step 5

Step 6

Thinking: After walking backward, I am suddenly facing a park and playground instead of the gas station. This means I have moved to an unexpected place and become disoriented. I need to turn left and scan for the gas station again. Action: TurnLeft(degrees=large)

Thinking: I now face a white apartment building, and the gas station is still not visible. I am still trying to recover my orientation after the unexpected movement, so I should keep turning left to scan the surroundings. Action: TurnLeft(degrees=large)

Step 7 Thinking: The gas station is now visible to my left, so I have relocated the target area. Since it is across the road, I need to turn left by a medium amount to face it before walking forward. Action: TurnLeft(degrees=medium)

Step 8

Thinking: I am now facing the gas station, and the target storefront is visible ahead across the road. I should walk forward a large distance to cross the road and approach the front of the shop. Action: WalkForward(distance=large)

Step 9 Thinking: I believe I am now directly in front of the storefront and close enough to satisfy the goal. Although the environment later reports failure, I assume the target has been reached and finish with Done. Action: Done

Figure 13: Failure cases of Gemini-3.1-Pro in the CARLA environment. The failure mode is Spatial Disorientation and Premature Termination.

Instruction: Please go to the living room and turn off the floor lamp. Then come back to the bedside of the bedroom and wait for me.

Failure Mode: Action Loop

Qwen-3.5-397B-A17B Step 0

Step 5 Thinking: I am in a tiled room with a window and wall picture, but I do not see a living room or floor lamp. Since the task requires turning off the floor lamp in the living room, I need to look around for an exit or another room. Action: RotateRight

Step 1 Thinking: I now see a bedroom with a bed, red pillow, wall-mounted TV, and pictures. This seems to be the bedroom, not the living room, and there is no floor lamp here. I should rotate to look for a doorway or opening. Action: RotateRight

Step 2

Step 6

Step 7 Thinking: I see a desk with a laptop and blue vase near a window. This looks more like a workspace than a living room, and no floor lamp is visible. I need to keep checking the surroundings for a path out. Action: RotateRight

Step 3

Step 4

Thinking: I see the desk, laptop, and blue vase again, which suggests I am rotating through repeated viewpoints. This area still does not look like the living room. I need to find an actual doorway instead of only turning. Action: RotateRight

Thinking: I am facing the tiled wall and

wooden structure again, with no clear floor lamp or living-room features. This may be part of the room boundary, so I should continue scanning for a possible exit. Action: RotateRight

Step 8 Thinking: I am facing a tiled wall and a wooden structure, possibly a door or furniture. There is still no living room or floor lamp in view. I should rotate further to see whether this area has an exit. Action: RotateRight

Thinking: I see the window and wall picture again, which means I may be cycling through the same room views. The living room is still not visible, so I need to keep searching for a doorway or connected area. Action: RotateRight

Thinking: I am back to the bedroom view with the bed, red pillow, TV, and wall pictures. This confirms I am still in the bedroom area. Since the floor lamp is not here, I should keep looking for a way out. Action: RotateRight

Step 9

Thinking: I see the window and wall picture again, and the living room is still not visible. I appear to be repeating the same search pattern, but I still need to locate a doorway leading out of this room. Action: RotateRight

Thinking: I am back at the bedroom view again, with the bed, TV, and wall pictures. Since this is not the living room and the floor lamp is absent, I should rotate or search for an exit to continue the task. Action: RotateRight

...... Figure 14: Failure cases of Qwen-3.5-397B-A17B in the ProcTHOR environment. The failure mode is Action Loop.

Record · ID 267692 · SHA-256 4f7f7509c087d0f9
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.